[PUP-4653] filebucket fails when environment is specified Created: 2015/05/26  Updated: 2019/04/04  Resolved: 2015/07/09

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: PUP 2.7.23
Fix Version/s: PUP 4.3.0

Type: Bug Priority: Normal
Reporter: Felix Frank Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: community, indirector
Remaining Estimate: 0 minutes
Time Spent: 1 hour
Original Estimate: 0 minutes

Issue Links:
Blocks
blocks PUP-1072 support HTTP(S) URL as the file 'source' Closed
Duplicate
is duplicated by PUP-4954 filebucket fails when environment is ... Closed
is duplicated by ENTERPRISE-841 filebucket ignoring settings and look... Closed
is duplicated by PUP-5466 Agent Filebucket operations fail afte... Closed
Template:
Story Points: 2
Sprint: Client 2015-07-22
Release Notes: Bug Fix
Release Notes Summary: This ticket was supposed to cleanup the client-side filebucket code so that in the future we could support HTTP(S) based file sources. After this change was made, we realized it fixed another bug where filebucket requests failed when an environment was specified (PUP-4954). So for release notes, puppet will correctly filebucket files when an environment is specified, such as when using agent specified environments.
QA Contact: Eric Thompson

 Description   

Since the 0.25 days, the filebucket client in the form of Puppet::FileBucket::Dipper has interfaces with the indirector by constructing URLs in the following fashion:

@rest_path = "https://#{server}:#{port}/#{environment}/file_bucket_file/"
# then
dest_path = "#{@rest_path}#{file_bucket_file.name}/#{files_original_path}"

The indirector breaks this down into its fields in Puppet::Indirector::Request#set_uri_key.

env, indirector, @key = URI.unescape(uri.path.sub(/^\//, '')).split('/',3)
@key ||= ''
self.environment = env unless env == ''

All of this is quite unecessary because

1. the indirection implicitly follows from the indirected class
2. the environment is not significant for filebucket interactions

(As an aside, the API does not construct URLs like this anymore, either - the environment is a parameter and not a part of the URI path. This is not important here, though, because these URLs never hit the wire. They are only for the benefit of the Indirector::Request class.)

A consequence of this construct is that new indirector uses with URLs in keys will be misinterpreted by the Request class.

It would be beneficial to transform these internally used URLs into a form that

1. confers only the necessary information to the indirector
2. does not clash with other possible URLs

Summary

Although this sounds like a network change, it's not. It only changes how the client internally manages the filebucket request, but doesn't affect how the client sends the request on the wire. It also doesn't change file manifests, e.g. filebucket resources, etc.



 Comments   
Comment by Felix Frank [ 2015/05/26 ]

There is already a PR to convert these URLs to a custom filebucket:// scheme.

Comment by Eric Thompson [ 2015/06/30 ]

sounds like it needs acceptance.
QA: see: https://testrail.ops.puppetlabs.net/index.php?/cases/view/63196
and: https://testrail.ops.puppetlabs.net/index.php?/cases/view/4840

Comment by Shaigy Nixon [X] (Inactive) [ 2015/07/09 ]

Validated basic filebucket functionalities with 'puppet agent -t' and 'puppet apply ' on SHA=9b28aee3e99572f4d98973d683592dc11239f8d6 on redhat-6-x86_64:
On master:

[root@p6wkgpdvls4w7xi manifests]# cat /etc/puppetlabs/code/environments/production/manifests/site.pp
 
filebucket { "main":
     server => "p6wkgpdvls4w7xi.delivery.puppetlabs.net",
     path => false  }
 
File { backup => "main" }
 
node 'default' {
  include test
}
 
class test {
  file { "/root/testfile.txt":
    source => "puppet:///modules/test/testfile.txt",
    owner => "root",
    group => "root",
    mode => "0660",
    ensure => present,
  }
}
 
[root@p6wkgpdvls4w7xi manifests]# cat /etc/puppetlabs/code/environments/production/modules/test/files/testfile.txt
Hi There
[root@p6wkgpdvls4w7xi manifests]# puppet apply site.pp
Notice: Compiled catalog for p6wkgpdvls4w7xi.delivery.puppetlabs.net in environment production in 0.33 seconds
Notice: /Stage[main]/Test/File[/root/testfile.txt]/ensure: defined content as '{md5}dedc1f20145899a3f253a03ffc752d6b'
Notice: Applied catalog in 0.05 seconds
[root@p6wkgpdvls4w7xi manifests]# cat /root/testfile.txt
Hi There
 
[root@p6wkgpdvls4w7xi manifests]# echo "Adding text on master" >> /root/testfile.txt
[root@p6wkgpdvls4w7xi manifests]# cat /root/testfile.txt
Hi There
Adding text on master
[root@p6wkgpdvls4w7xi manifests]# puppet apply site.pp
Notice: Compiled catalog for p6wkgpdvls4w7xi.delivery.puppetlabs.net in environment production in 0.32 seconds
Notice: /Stage[main]/Test/File[/root/testfile.txt]/content: content changed '{md5}6ea354b2f032c76ca592082f6f154dbe' to '{md5}dedc1f20145899a3f253a03ffc752d6b'
Notice: Applied catalog in 0.18 seconds
[root@p6wkgpdvls4w7xi manifests]# find / -name 6ea354b2f032c76ca592082f6f154dbe
/opt/puppetlabs/server/data/puppetserver/bucket/6/e/a/3/5/4/b/2/6ea354b2f032c76ca592082f6f154dbe
[root@p6wkgpdvls4w7xi manifests]# cat /opt/puppetlabs/server/data/puppetserver/bucket/6/e/a/3/5/4/b/2/6ea354b2f032c76ca592082f6f154dbe/contents
Hi There
Adding text on master
 
root@p6wkgpdvls4w7xi manifests]# puppet filebucket get 6ea354b2f032c76ca592082f6f154dbe --local --bucket /opt/puppetlabs/server/data/puppetserver/bucket
Hi There
Adding text on master
 
 
[root@p6wkgpdvls4w7xi manifests]# puppet filebucket restore /root/restorefile.txt 6ea354b2f032c76ca592082f6f154dbe --local --bucket /opt/puppetlabs/server/data/puppetserver/bucket
[root@p6wkgpdvls4w7xi manifests]# cat /root/restorefile.txt
Hi There
Adding text on master

On Agent:

[root@slc5k671tnrjg1f ~]# puppet agent --test --server p6wkgpdvls4w7xi.delivery.puppetlabs.net
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for slc5k671tnrjg1f.delivery.puppetlabs.net
Info: Applying configuration version '1436477616'
Notice: /Stage[main]/Test/File[/root/testfile.txt]/ensure: defined content as '{md5}dedc1f20145899a3f253a03ffc752d6b'
Notice: Applied catalog in 0.06 seconds
 
[root@slc5k671tnrjg1f ~]# cat /root/testfile.txt
Hi There
 
[root@slc5k671tnrjg1f ~]# echo "Adding text on agent" >> /root/testfile.txt
[root@slc5k671tnrjg1f ~]# cat /root/testfile.txt
Hi There
Adding text on agent
 
[root@slc5k671tnrjg1f ~]# puppet agent --test --server p6wkgpdvls4w7xi.delivery.puppetlabs.net
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for slc5k671tnrjg1f.delivery.puppetlabs.net
Info: Applying configuration version '1436477663'
Notice: /Stage[main]/Test/File[/root/testfile.txt]/content:
--- /root/testfile.txt	2015-07-09 14:34:09.632879332 -0700
+++ /tmp/puppet-file20150709-5657-3usk2d	2015-07-09 14:34:23.766881072 -0700
@@ -1,2 +1 @@
 Hi There
-Adding text on agent
 
Info: Computing checksum on file /root/testfile.txt
Info: /Stage[main]/Test/File[/root/testfile.txt]: Filebucketed /root/testfile.txt to main with sum 2612bfea1806457482a2e5bc2a2eedae
Notice: /Stage[main]/Test/File[/root/testfile.txt]/content: content changed '{md5}2612bfea1806457482a2e5bc2a2eedae' to '{md5}dedc1f20145899a3f253a03ffc752d6b'
Notice: Applied catalog in 0.14 seconds
 
[root@slc5k671tnrjg1f ~]# puppet filebucket get 2612bfea1806457482a2e5bc2a2eedae --remote --server p6wkgpdvls4w7xi.delivery.puppetlabs.net
Hi There
Adding text on agent
 
[root@slc5k671tnrjg1f ~]# puppet filebucket restore /root/rstore.txt  2612bfea1806457482a2e5bc2a2eedae --remote --server p6wkgpdvls4w7xi.delivery.puppetlabs.net
[root@slc5k671tnrjg1f ~]# cat /root/rstore.txt
Hi There
Adding text on agent

On master:

[root@p6wkgpdvls4w7xi manifests]# find / -name 2612bfea1806457482a2e5bc2a2eedae
/opt/puppetlabs/server/data/puppetserver/bucket/2/6/1/2/b/f/e/a/2612bfea1806457482a2e5bc2a2eedae
[root@p6wkgpdvls4w7xi manifests]# cat /opt/puppetlabs/server/data/puppetserver/bucket/2/6/1/2/b/f/e/a/2612bfea1806457482a2e5bc2a2eedae/contents
Hi There
Adding text on agent

After modifying the manifest on master to use clientbucket:

node 'default' {
  include test
}
 
class test {
  file { "/root/testfile.txt":
    source => "puppet:///modules/test/testfile.txt",
    owner => "root",
    group => "root",
    mode => "0660",
    ensure => present,
#    backup => 'main',
  }

On Agent:

[root@slc5k671tnrjg1f ~]# puppet agent --test --server p6wkgpdvls4w7xi.delivery.puppetlabs.netInfo: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for slc5k671tnrjg1f.delivery.puppetlabs.net
Info: Applying configuration version '1436478254'
Notice: Applied catalog in 0.05 seconds
[root@slc5k671tnrjg1f ~]# cat /root/testfile.txt
Hi There
 
[root@slc5k671tnrjg1f ~]# echo "blahblahblah" >>/root/testfile.txt
[root@slc5k671tnrjg1f ~]# cat /root/testfile.txt
Hi There
blahblahblah
 
[root@slc5k671tnrjg1f ~]# puppet agent --test --server p6wkgpdvls4w7xi.delivery.puppetlabs.net
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for slc5k671tnrjg1f.delivery.puppetlabs.net
Info: Applying configuration version '1436478299'
Notice: /Stage[main]/Test/File[/root/testfile.txt]/content:
--- /root/testfile.txt	2015-07-09 14:44:45.017879920 -0700
+++ /tmp/puppet-file20150709-5759-9ddjk0	2015-07-09 14:44:58.180880266 -0700
@@ -1,2 +1 @@
 Hi There
-blahblahblah
 
Info: Computing checksum on file /root/testfile.txt
Info: /Stage[main]/Test/File[/root/testfile.txt]: Filebucketed /root/testfile.txt to puppet with sum 830690be6b97a88d2547c42c7290bde6
Notice: /Stage[main]/Test/File[/root/testfile.txt]/content: content changed '{md5}830690be6b97a88d2547c42c7290bde6' to '{md5}dedc1f20145899a3f253a03ffc752d6b'
Notice: Applied catalog in 0.11 seconds
 
[root@slc5k671tnrjg1f ~]# find / -name 830690be6b97a88d2547c42c7290bde6
/opt/puppetlabs/puppet/cache/clientbucket/8/3/0/6/9/0/b/e/830690be6b97a88d2547c42c7290bde6
[root@slc5k671tnrjg1f ~]# cat /opt/puppetlabs/puppet/cache/clientbucket/8/3/0/6/9/0/b/e/830690be6b97a88d2547c42c7290bde6/contents
Hi There
blahblahblah
 
[root@slc5k671tnrjg1f ~]# puppet filebucket get 830690be6b97a88d2547c42c7290bde6 --local --bucket /opt/puppetlabs/puppet/cache/clientbucket
Hi There
blahblahblah
 
[root@slc5k671tnrjg1f ~]# puppet filebucket restore /root/restore.txt  830690be6b97a88d2547c42c7290bde6 --local --bucket /opt/puppetlabs/puppet/cache/clientbucket
[root@slc5k671tnrjg1f ~]# cat /root/restore.txt
Hi There
blahblahblah

Comment by Shaigy Nixon [X] (Inactive) [ 2015/07/09 ]

Validated with agent running on win2012r2-rubyx64:
Manifest on master:

[root@xvexth0tmgidygv manifests]# cat site.pp
filebucket { "main":
     server => "xvexth0tmgidygv.delivery.puppetlabs.net",
     path => false  }
 
File { backup => "main" }
 
node 'default' {
  include test
}
 
class test {
  file { "C:\\Users\\Administrator\\test\\testfile.txt":
    source => "puppet:///modules/test/testfile.txt",
    ensure => present,
  }
}
 
[root@xvexth0tmgidygv manifests]# cat /etc/puppetlabs/code/environments/production/modules/test/files/testfile.txt
Hi Windows

On agent node:

$ cat "c:\Users\Administrator\test\testfile.txt"
cat: c:\Users\Administrator\test\testfile.txt: No such file or directory
 
Administrator@o3q0lzxo00zfxlp ~
$ cmd /c puppet agent --test
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for o3q0lzxo00zfxlp.delivery.puppetlabs.net
Info: Applying configuration version '1436482388'
Notice: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]/ensure: defined content as '{md5}37a449d502c9d57556bb815cf4c079e9'
Notice: Applied catalog in 0.09 seconds
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\testfile.txt"
Hi Windows
 
Administrator@o3q0lzxo00zfxlp ~
$ echo Adding text from windows >> "c:\Users\Administrator\test\testfile.txt"
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\testfile.txt"
Hi Windows
Adding text from windows
 
Administrator@o3q0lzxo00zfxlp ~
$ cmd /c puppet agent --test
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for o3q0lzxo00zfxlp.delivery.puppetlabs.net
Info: Applying configuration version '1436482455'
Notice: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]/content:
 
Info: Computing checksum on file C:/Users/Administrator/test/testfile.txt
Info: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]: Filebucketed C:/Users/Administrator/test/testfile.txt to main with sum 1936e8b993904f6a99a9ecd1e0ea2445
Notice: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]/content: content changed '{md5}1936e8b993904f6a99a9ecd1e0ea2445' to '{md5}37a449d502c9d57556bb815cf4c079e9'
Notice: Applied catalog in 0.19 seconds
 
$ cmd /c puppet filebucket get 1936e8b993904f6a99a9ecd1e0ea2445 --remote
Hi Windows
Adding text from windows
 
Administrator@o3q0lzxo00zfxlp ~
$ cmd /c puppet filebucket restore "c:\Users\Administrator\test\restore.txt"  205ea76a461e3ba51358cac96c510aec --remote
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\restore.txt"
Hi Windows
Adding more text on windows

After modifying manifest to use clientbucket:

node 'default' {
  include test
}
 
class test {
  file { "C:\\Users\\Administrator\\test\\testfile.txt":
    source => "puppet:///modules/test/testfile.txt",
    ensure => present,
  }
}

On agent node:

$ cmd /c puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for o3q0lzxo00zfxlp.delivery.puppetlabs.net
Info: Applying configuration version '1436482674'
Notice: Applied catalog in 0.08 seconds
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\testfile.txt"
Hi Windows
 
Administrator@o3q0lzxo00zfxlp ~
$ echo BlahBlahBlah >> "c:\Users\Administrator\test\testfile.txt"
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\testfile.txt"
Hi Windows
BlahBlahBlah
 
Administrator@o3q0lzxo00zfxlp ~
$ cmd /c puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for o3q0lzxo00zfxlp.delivery.puppetlabs.net
Info: Applying configuration version '1436482736'
Notice: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]/content:
 
Info: Computing checksum on file C:/Users/Administrator/test/testfile.txt
Info: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]: Filebucketed C:/Users/Administrator/test/testfile.txt to puppet with sum cad5cf02472371fb78ddb4627eafd0aa
Notice: /Stage[main]/Test/File[C:\Users\Administrator\test\testfile.txt]/content: content changed '{md5}cad5cf02472371fb78ddb4627eafd0aa' to '{md5}37a449d502c9d57556bb815cf4c079e9'
Notice: Applied catalog in 0.14 seconds
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "C:/ProgramData/PuppetLabs/puppet/cache/clientbucket/c/a/d/5/c/f/0/2/cad5cf02472371fb78ddb4627eafd0aa/contents"
Hi Windows
BlahBlahBlah
 
$ cmd /c puppet filebucket get cad5cf02472371fb78ddb4627eafd0aa --local --bucket "C:/ProgramData/PuppetLabs/puppet/cache/clientbucket"
Hi Windows
BlahBlahBlah
 
$ cmd /c puppet filebucket restore "c:\Users\Administrator\test\restore.txt" cad5cf02472371fb78ddb4627eafd0aa --local --bucket "C:/ProgramData/PuppetLabs/puppet/cache/clientbucket"
 
Administrator@o3q0lzxo00zfxlp ~
$ cat "c:\Users\Administrator\test\restore.txt"
Hi Windows
BlahBlahBlah

Generated at Sat Dec 07 21:10:47 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.