[PUP-1515] When compiling a catalog, providers should be loaded from specified environment Created: 2014/01/24  Updated: 2019/04/04  Resolved: 2015/03/26

Status: Closed
Project: Puppet
Component/s: Docs
Affects Version/s: PUP 3.6.2
Fix Version/s: PUP 3.7.5, PUP 4.0.0

Type: Bug Priority: Critical
Reporter: redmine.exporter Assignee: Lindsey Smith
Resolution: Done Votes: 26
Labels: redmine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Cloners
is cloned by ENTERPRISE-551 Invalid parameter provider for custom... In Progress
Relates
relates to MODULES-1690 Invalid parameter provider Closed
relates to MODULES-4357 puppetlabs-f5 virtualserver can't pro... Closed
relates to PUP-8696 Puppet loses track of the current env... Closed
Template:
Epic Link: 4.0 Important Fixes
Story Points: 3
Sprint: Language 2015-01-21, Language 2015-02-04
QA Contact: Kurt Wall

 Description   

When compiling a catalog, types and providers are validated, but while Puppet will load types from the agent's environment, it only loads providers from the compiling master's default environment (production).

This is a problem because the method that creates the provider parameter is sometimes called due to code in the provider, not the type. If a custom resource type only exists in a non-default environment, and you declare a resource that specifies a value for the provider attribute, this can cause an invalid parameter provider error.

The expected behavior is that Puppet will autoload types and providers from the agent's environment.


original description:
In Puppet 3 I am getting an error on all definitions for custom types. It says "Error 400 on SERVER: Invalid parameter provider...". Provider should be a given parameter for custom types because otherwise there is no way to specify which provider should be used with it. This is potentially a very major bug. Please let me know how I can help so you're able to reproduce and fix the issue.


Final resolution: When validating resources during catalog compilation, Puppet will load types and providers from the environment that the catalog is being compiled for. (Modulo that other bug that causes cross-environment contamination.)



 Comments   
Comment by Joshua Chaitin-Pollak [ 2014/01/24 ]

I am experiencing a similar issue with puppet 3.4.2. I've posted here as well (https://github.com/puppetlabs/puppetlabs-vcsrepo/issues/118), I apologize in advance for crossposting, I'm not sure if this is a Puppet issue or a vcsrepo one.

I have the following configuration:

vcsrepo { "$directory" :
    ensure => latest,
    provider => git,
    source => "git@github.com:example/project.git",
    revision => "$revision",
    user => "$user",
    require => Class['::nginx'],
}

And when I run puppet agent on my node, I get this error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter provider at /var/lib/puppet/modules/project/manifests/init.pp:21 on node mynode.example.com

This error is repeated in the puppetmaster's log as well.

I restarted the puppet master, but that didn't solve the problem. I did verify that the git.rb file is in the puppet repository.

Both the master and slave are running Puppet 3.4.2

Any help would be appreciated.

Comment by Alejandro Figueroa [ 2014/02/03 ]

I ran into this issue as well with multiple environments.

The master server is running in the 'production' environment and the client is running in the 'staging' environment.
I was able to get it running after doing a puppet run on the master while forcing the staging environment as mentioned here:
http://projects.puppetlabs.com/issues/17814

I also restarted the puppetmaster apache process as well, though I'm not sure if this had any effect.

All puppet versions at 3.4.2.

Comment by Net Kgk [ 2014/02/12 ]

This issue makes it impossible to use any native type with environments (which is officially recommended approach), so raising priority.

Comment by Eric Sorenson [ 2014/02/19 ]

Net Kgk Alejandro Figueroa i am having trouble reproducing this issue. is it true in your case that, as Andy Shinn said on the redmine bug "my default environment for the puppetmaster doesn’t have some new types I am testing in another environment and when testing that environment it is failing." ??

Comment by Joshua Chaitin-Pollak [ 2014/02/19 ]

You didn't tag me, but I can confirm our puppetmaster has a default configuration of the puppet environment, but is configured to serve one of two different puppet libraries based on parameters of the connected client.

In other words, I believe out puppet master executes with a different set of puppet modules than it serves to clients, which I think is what people say causes this situation.

Comment by Net Kgk [ 2014/02/20 ]

I've got this issue, with next settings:

...
[master]
...
environment = production
...
  manifest = $confdir/environments/$environment/manifests/site.pp
  modulepath = $confdir/environments/$environment/modules
...

Then, when I've updated different environment (let's say test) with new module (puppet-labs/mysql), I've got this exception on master while running puppet agent with --environment='test' on different host, because puppet-labs/mysql was absent in production environment.

Comment by Alejandro Figueroa [ 2014/02/20 ]

I can confirm Net Kgk's diagnosis. The module was different on my servers (puppetlabs/vcsrepo) but the results were the same.

Comment by Matt Wise [ 2014/03/04 ]

We're seeing this as well on our Puppet 3.4.1 servers. We had to copy the library into our 'master' branch of code in order for it to work anywhere. This is a huge bug because it makes doing development work on other branches difficult.

Comment by Nicholas Hinds [ 2014/03/14 ]

In case people are interested, it's possible to work around this bug for a small number of agents by running a plugin sync on the puppet master before applying the configuration on the puppet agent; of course, this doesn't scale nor does it allow you to setup agents to poll the puppet master unless they're in the default production environment:

  1. Run puppet plugin download --environment ENVIRONMENT-NAME-HERE && service puppetmaster restart on the puppet master to make the plugins available in the correct place
  2. Apply the puppet configuration on the puppet agent (puppet agent -t)
  3. Run puppet plugin download && service puppetmaster restart on the puppet master to restore the original plugins
Comment by Eric Sorenson [ 2014/04/30 ]

Simple steps to repro on one host, running Puppet 3.5.1 agent and master as non-root (user vagrant).

1. Set up a puppet.conf as follows, to enable directory environments:

[main]
environmentpath = $confdir/environments

2. Create a directory structure under ~/.puppet/environments like:

[vagrant@victim environments]$ tree -L 2 .
.
├── fail
│   ├── manifests
│   └── modules
└── production
    ├── manifests
    └── modules

3. Install puppetlabs-vcsrepo into the 'fail' environment's modulepath:

[vagrant@victim manifests]$ puppet module install --environment fail puppetlabs-vcsrepo
Notice: Preparing to install into /home/vagrant/.puppet/environments/fail/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/home/vagrant/.puppet/environments/fail/modules
└── puppetlabs-vcsrepo (v0.2.0)

4. Create a site.pp for each environment; the 'fail' environment should have a vcsrepo resource:

[vagrant@victim environments]$ cat production/manifests/site.pp
node default {
        notify {'environment': message => "current environment is ${environment}" }
}
[vagrant@victim environments]$ cat fail/manifests/site.pp
node default {
  notify {'environment': message => "current environment is ${environment}" }
 
  vcsrepo { 'dotfiles':
    ensure => present,
    provider => git,
    revision => master,
    path => '/home/vagrant/dotfiles',
    source => 'https://github.com/ahpook/dotfiles.git',
  }
 
}

5. Start the master

[vagrant@victim .puppet]$ puppet master --no-daemonize --debug

6. Point the agent at the master using a separate vardir (as if it were a separate host), using the fail environment:

[vagrant@victim manifests]$ puppet agent -tv --server victim.local --vardir /home/vagrant/.puppet/var.agent --environment fail
Info: Retrieving pluginfacts
Info: Retrieving plugin
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter provider at /home/vagrant/.puppet/environments/fail/manifests/site.pp:10
on node victim.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Master logs the following error:

Error: Invalid parameter provider at /home/vagrant/.puppet/environments/fail/manifests/site.pp:10 on node victim.local
Wrapped exception:
Invalid parameter provider
Error: Invalid parameter provider at /home/vagrant/.puppet/environments/fail/manifests/site.pp:10 on node victim.local
Error: Invalid parameter provider at /home/vagrant/.puppet/environments/fail/manifests/site.pp:10 on node victim.local

Comment by Andreas Ntaflos [ 2014/05/01 ]

"Me, too."

Our Puppet master currently runs in the 'production' environment while a growing number of nodes run in a different environment (called 'production_redux', as we are overhauling the Puppet infrastructure on that site). Using the puppetlabs-mysql module in the 'production_redux' environment (but not in 'production') we get the "invalid parameter provider" exception trying to manage MySQL users and databases.

Do I understand correctly that switching the Puppet master's own environment to (in our case) 'production_redux' would work around this problem?

FWIW, running puppet plugin download --environment production_redux on the Puppet master results in this:

# puppet plugin download --environment production_redux 
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': undefined method `intern' for nil:NilClass
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://puppet01.example.com/plugins: undefined method `intern' for nil:NilClass
Wrapped exception:
undefined method `intern' for nil:NilClass
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': undefined method `intern' for nil:NilClass
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://puppet01.example.com/pluginfacts: undefined method `intern' for nil:NilClass
Wrapped exception:
undefined method `intern' for nil:NilClass
No plugins downloaded.

I have no idea what to make of this.

Comment by vinod kumar [ 2014/05/05 ]

I also observed the similar behavior with custom environment path.

Following is the alternate to work with:

  • Install puppet module without any environment i.e. puppet module install puppetlabs-vcsrepo

This will install the vcsrepo module under <$puppet-conf-dir>/modules directory.

  • Now, add this path to modulepath in puppet.conf directory as follows under the custom environment:

[fail]
modulepath = /Users/vinodp/Vinod/DevOps/modules:/Users/vinodp/.puppet/modules

  • Also enable 'pluginsync = true' in puppet.conf file
  • Now, you can access puppet agent with the custom environment like

puppet agent --test --environment fail --debug

Hope this will work for every one as it is working for me.

Comment by Martin Mörner [ 2014/07/03 ]

Please fix this bug as any currently advocated development method with environments and usage of r10k builds works against this. Especially when using official modules like puppetlabs/mysql:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter provider on Mysql_database[foo] at /etc/puppet/environments/dev_mysql/modules/mysql/manifests/db.pp:27 on node...

/etc/puppet/puppet.conf

[main]
environment  =  production
confdir      =  /etc/puppet
logdir       =  /var/log/puppet
vardir       =  /var/lib/puppet
ssldir       =  /var/lib/puppet/ssl
rundir       =  /var/run/puppet
factpath     =  $vardir/lib/facter
pluginsync   =  true
 
[master]
environment               =  production
environmentpath           =  $confdir/environments
autosign                  =  false
ssl_client_header         =  SSL_CLIENT_S_DN
ssl_client_verify_header  =  SSL_CLIENT_VERIFY
storeconfigs              =  true
storeconfigs_backend      =  puppetdb
reports                   =  puppetdb
 
[agent]
environment  =  production
server       =  ***
report       =  true

Affected Version:

dpkg -l puppet
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                         Version             Architecture        Description
+++-============================-===================-==
ii  puppet                       3.6.2-1puppetlabs1  all                 Centralized 

Comment by Tom De Vylder [ 2014/07/15 ]

I'm running into the same behavior with:

3.6.2-1puppetlabs1
3.6.1-1puppetlabs1
3.5.1-1puppetlabs1

(from apt.puppetlabs.com)

... using puppet apply on Puppetlabs' current Debian 7 Vagrant box (http://puppet-vagrant-boxes.puppetlabs.com/debian-73-x64-virtualbox-puppet.box).

Comment by cristi falcas [ 2014/08/07 ]

Any news about the status of this? Currently we can't use multiple environments because of this. Well, actually we use a "development" environment for our puppet masters where we keep all our modules in order to mitigate this.

Comment by Andrew Parker [ 2014/08/11 ]

Charlie Sharpsteen is going to take a look into this and see if this is definitively related to PUP-731 or not. If it is, then there isn't really anything we can do about it with the current master setups. If it isn't, then his investigation should uncover a bit clearer the source of the problem.

Comment by Charlie Sharpsteen [ 2014/08/11 ]

Eric's re-production steps for 3.5.1 still work on 3.6.2. Here is a backtrace from the master:

[root@poss-head-centos ~]# puppet master --no-daemonize --trace --verbose
Notice: Starting Puppet master version 3.6.2
Info: Inserting default '~ ^/catalog/([^/]+)$' (auth true) ACL
Info: Inserting default '~ ^/node/([^/]+)$' (auth true) ACL
Info: Inserting default '/file' (auth ) ACL
Info: Inserting default '/certificate_revocation_list/ca' (auth true) ACL
Info: Inserting default '~ ^/report/([^/]+)$' (auth true) ACL
Info: Inserting default '/certificate/ca' (auth any) ACL
Info: Inserting default '/certificate/' (auth any) ACL
Info: Inserting default '/certificate_request' (auth any) ACL
Info: Inserting default '/status' (auth true) ACL
Info: Inserting default '/v2.0/environments' (auth true) ACL
Info: Caching node for poss-head-centos.puppetdebug.vlan
Info: Caching node for poss-head-centos.puppetdebug.vlan
Error: Invalid parameter provider on Vcsrepo[dotfiles] at /etc/puppet/environments/fail/manifests/site.pp:10 on node poss-head-centos.puppetdebug.vlan
/puppetlabs/puppet/lib/puppet/resource.rb:442:in `validate_parameter'
/puppetlabs/puppet/lib/puppet/parser/resource.rb:264:in `block in validate'
/puppetlabs/puppet/lib/puppet/parser/resource.rb:263:in `each'
/puppetlabs/puppet/lib/puppet/parser/resource.rb:263:in `validate'
/puppetlabs/puppet/lib/puppet/parser/resource.rb:107:in `finish'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:442:in `block in finish'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:430:in `each'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:430:in `finish'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:128:in `block (2 levels) in compile'
/puppetlabs/puppet/lib/puppet/util/profiler/none.rb:6:in `profile'
/puppetlabs/puppet/lib/puppet/util/profiler.rb:43:in `profile'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:128:in `block in compile'
/puppetlabs/puppet/lib/puppet/context.rb:64:in `override'
/puppetlabs/puppet/lib/puppet.rb:234:in `override'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:106:in `compile'
/puppetlabs/puppet/lib/puppet/parser/compiler.rb:23:in `compile'
/puppetlabs/puppet/lib/puppet/indirector/catalog/compiler.rb:95:in `block (2 levels) in compile'
/puppetlabs/puppet/lib/puppet/util/profiler/none.rb:6:in `profile'
/puppetlabs/puppet/lib/puppet/util/profiler.rb:43:in `profile'
/puppetlabs/puppet/lib/puppet/indirector/catalog/compiler.rb:93:in `block in compile'
/puppetlabs/puppet/lib/puppet/util.rb:161:in `block in benchmark'
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime'
/puppetlabs/puppet/lib/puppet/util.rb:160:in `benchmark'
/puppetlabs/puppet/lib/puppet/indirector/catalog/compiler.rb:92:in `compile'
/puppetlabs/puppet/lib/puppet/indirector/catalog/compiler.rb:52:in `find'
/puppetlabs/puppet/lib/puppet/indirector/indirection.rb:201:in `find'
/puppetlabs/puppet/lib/puppet/network/http/api/v1.rb:105:in `do_find'
/puppetlabs/puppet/lib/puppet/network/http/api/v1.rb:50:in `block in call'
/puppetlabs/puppet/lib/puppet/context.rb:64:in `override'
/puppetlabs/puppet/lib/puppet.rb:234:in `override'
/puppetlabs/puppet/lib/puppet/network/http/api/v1.rb:49:in `call'
/puppetlabs/puppet/lib/puppet/network/http/route.rb:82:in `block in process'
/puppetlabs/puppet/lib/puppet/network/http/route.rb:81:in `each'
/puppetlabs/puppet/lib/puppet/network/http/route.rb:81:in `process'
/puppetlabs/puppet/lib/puppet/network/http/handler.rb:63:in `block in process'
/puppetlabs/puppet/lib/puppet/util/profiler/none.rb:6:in `profile'
/puppetlabs/puppet/lib/puppet/util/profiler.rb:43:in `profile'
/puppetlabs/puppet/lib/puppet/network/http/handler.rb:61:in `process'
/puppetlabs/puppet/lib/puppet/network/http/webrick/rest.rb:37:in `block in service'
<internal:prelude>:10:in `synchronize'
/puppetlabs/puppet/lib/puppet/network/http/webrick/rest.rb:36:in `service'
/opt/rh/ruby193/root/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/opt/rh/ruby193/root/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/puppetlabs/puppet/lib/puppet/network/http/webrick.rb:36:in `block (2 levels) in listen'
/opt/rh/ruby193/root/usr/share/ruby/webrick/server.rb:191:in `call'
/opt/rh/ruby193/root/usr/share/ruby/webrick/server.rb:191:in `block in start_thread'

So, compilation is dying in the very last stage when the master is validating the catalog contents before shipping it to the agent. Also worth noting is that the master environment will be "production" unless set to something else:

[30] pry(#<Puppet::Parser::Resource>)> Puppet[:environment]
=> "production"

My current theory is: the master can load types from the environment requested by the agent, but it can only load providers from it's own environment. Hence, the Vcsrepo Type is acting like an old primordial Type from the time before providers existed (Yumrepo used to be one of these until it was refactored in 3.5.0). If no providers are loaded, Type.providify never gets called and this method defines the provider parameter that the validator is failing to see.

If the master is launched with the same environment that the agent will use, the error disappears:

puppet master --no-daemonize --trace --verbose --environment fail

I'll dig some more tomorrow to see if my theory is correct. If it is, then this is a bit different than PUP-731 and I'll try to determine why the master is not loading providers from the agent environment.

Comment by Charlie Sharpsteen [ 2014/08/12 ]

So, what is happening here is that the Autoloader isn't switching environments correctly.

When the Vcsrepo type is loaded, a call of this form is used:

typeloader.load(name, Puppet.lookup(:current_environment))

Where the typeloader is an Autoloader instance. In this case, we explicitly pass the current environment that the master is compiling for.

Once the type is loaded, we then load all known providers for that type:

klass.providerloader.loadall

Where providerloader is another Autoloader instance. Note that no environment is passed to this call. Because no environment is passed, there are two places where nil gets used instead. This causes the Autoloader to search Puppet[:environment] instead of the current_environment.

So, to summarize: when the Puppet master loads a Type, it searches the environment that the agent requested. When it loads providers for that type, it searches the default environment instead of the one the agent requested.

Andrew Parker: This is a different issue from PUP-731 and one that we could fix. Handing back to you to decide if this should be scheduled for a release.

Comment by Andrew Parker [ 2014/08/12 ]

I've encountered this code before. Unfortunately we don't have time before 3.7 to get a fix in for this. I'm going to target it at 4.0.0 instead, since fixing it might also cause some API breaks.

Comment by Eric Sorenson [ 2014/08/12 ]

Wow, that's great. Sounds like a very targeted fix that could help a lot of people. Would it be possible to get a branch or "DO NOT MERGE" PR up with a quick spike? That way people affected by it could try out the branch to see if it works in their environments.

Comment by Charlie Sharpsteen [ 2014/08/12 ]

Eric Sorenson: I've put the patch I developed while investigating this issue up as PR 2963.

No tests, no guarantees. But, if someone tries it out and it works, please do let us know!

Comment by Richard Raseley [ 2014/11/13 ]

We are also hitting this issue when using puppet-3.7.3-1.el7 on CentOS Linux release 7.0.1406 (3.10.0-123.el7.x86_64), which is blocking us on a couple items.

I will try to find the time tomorrow to check and see if the patch (https://github.com/puppetlabs/puppet/pull/2963) mentioned by Charlie Sharpsteen resolves the issue.

Comment by Dirk Heinrichs [ 2014/11/21 ]

Any chance to get this fixed in 3.7.x instead of 4.0.0?

Comment by Eric Sorenson [ 2014/11/21 ]

Dirk Heinrichs there is probably not going to be another 3.x release until just before 4.0 comes out so it wouldn't change things much.

As an aside, did you try the patch? Did it work for you?

Kylo Ginsberg can you estimate how much work it would take to get the patch up to snuff? I'm not even sure how you'd test this in a before-and-after sense, but there are a ton of people interested in this bug and I'd hate to see it miss the train.

Comment by Kylo Ginsberg [ 2014/11/21 ]

Eric Sorenson I just threw an estimate down for velocity purposes. We reviewed Charlie Sharpsteen's PR at last PR triage and all (me/Henrik/JoshC) agreed that it makes sense on paper.

However, given that it's about environments - a central and sometimes nuanced area of the code - we did want Joshua Partlow to take a look at the PR for any reservations, etc.

Comment by Dirk Heinrichs [ 2014/11/23 ]

No, I didn't try the patch. The workaround referenced above solved it for me.

Comment by Joshua Partlow [ 2014/11/24 ]

Commented on the PR; Charlie's patch looks like the correct behavior to me. A test run of acceptance on rhel7 passed as well.

Comment by Gabriele Paggi [ 2014/12/11 ]

"So, to summarize: when the Puppet master loads a Type, it searches the environment that the agent requested. When it loads providers for that type, it searches the default environment instead of the one the agent requested."

Charlie Sharpsteen, what is the name of the default environment? Is it "production"?
I'm seeing the same issue with only one environment named prd. The workaround I'm using is to define a production environment pointing to the same directories as the prd one:

# Always create production pointing to the first defined environment
# to circumvent PUP-1515 (pluginsync, custom provider/type)
[production]
        manifest = /etc/puppet/environments/prd/manifests/
        modulepath = /etc/puppet/environments/prd/.....
[prd]
        manifest = /etc/puppet/environments/prd/manifests/
        modulepath = /etc/puppet/environments/prd/......

Comment by Charlie Sharpsteen [ 2014/12/11 ]

Gabriele Paggi The default environment for the master is usually, almost nearly always production. Specifically, the default is whatever the environment setting has been set to in the master section of puppet.conf. Since most people don't set that, production is used.

You can check what the value of the default environment is by running the following on a Puppet master:

puppet config print --section master environment

Comment by Gabriele Paggi [ 2014/12/12 ]

Charlie Sharpsteen Thanks! I didn't have the environment set in the master section indeed.

Comment by Josh Cooper [ 2014/12/19 ]

Merged to master in fcc85689

Comment by Eric Thompson [ 2015/01/09 ]

verified using eric's reproduction above on rhel7 at SHA: 65bf83f with:

[root@i8epbide7fr98pz puppet]# puppet agent -tv --vardir /tmp/var/ --environment fail
Debug: Routes Registered:
Debug: Route /\/v3/
Debug: Route /^\/v2\.0/
Debug: Evaluating match for Route /\/v3/
Debug: Evaluating match for Route /^\/environments$/
Debug: Did not match path ("/node/i8epbide7fr98pz.delivery.puppetlabs.net")
Debug: Evaluating match for Route /.*/
Info: access[^/v3/catalog/([^/]+)$]: allowing 'method' find
Info: access[^/v3/catalog/([^/]+)$]: allowing $1 access
Info: access[^/v3/node/([^/]+)$]: allowing 'method' find
Info: access[^/v3/node/([^/]+)$]: allowing $1 access
Info: access[/v3/certificate_revocation_list/ca]: allowing 'method' find
Info: access[/v3/certificate_revocation_list/ca]: allowing * access
Info: access[^/v3/report/([^/]+)$]: allowing 'method' save
Info: access[^/v3/report/([^/]+)$]: allowing $1 access
Info: access[/v3/file]: allowing * access
Info: access[/v3/certificate/ca]: adding authentication any
Info: access[/v3/certificate/ca]: allowing 'method' find
Info: access[/v3/certificate/ca]: allowing * access
Info: access[/v3/certificate/]: adding authentication any
Info: access[/v3/certificate/]: allowing 'method' find
Info: access[/v3/certificate/]: allowing * access
Info: access[/v3/certificate_request]: adding authentication any
Info: access[/v3/certificate_request]: allowing 'method' find
Info: access[/v3/certificate_request]: allowing 'method' save
Info: access[/v3/certificate_request]: allowing * access
Info: access[/v2.0/environments]: allowing 'method' find
Info: access[/v2.0/environments]: allowing * access
Info: access[/v3/environments]: allowing 'method' find
Info: access[/v3/environments]: allowing * access
Info: access[/]: adding authentication any
Info: Inserting default '/v3/status' (auth true) ACL
Debug: Caching environment 'fail' (cache ttl: Infinity)
Info: Caching node for i8epbide7fr98pz.delivery.puppetlabs.net
Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: node supports formats: pson yaml raw
Info: Retrieving pluginfacts
Debug: Routes Registered:
Debug: Route /\/v3/
Debug: Route /^\/v2\.0/
Debug: Evaluating match for Route /\/v3/
Debug: Evaluating match for Route /^\/environments$/
Debug: Did not match path ("/file_metadatas/pluginfacts")
Debug: Evaluating match for Route /.*/
Debug: No pluginfacts found in subpath '/etc/puppet/environments/fail/modules/vcsrepo/facts.d' (file / directory does not exist)
Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: file_metadata supports formats: pson yaml raw
Info: Retrieving plugin
Debug: Routes Registered:
Debug: Route /\/v3/
Debug: Route /^\/v2\.0/
Debug: Evaluating match for Route /\/v3/
Debug: Evaluating match for Route /^\/environments$/
Debug: Did not match path ("/file_metadatas/plugins")
Debug: Evaluating match for Route /.*/
Debug: Failed to load library 'msgpack' for feature 'msgpack'
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: file_metadata supports formats: pson yaml raw
Notice: /File[/tmp/var/lib/puppet]/ensure: created
Notice: /File[/tmp/var/lib/puppet/provider]/ensure: created
Notice: /File[/tmp/var/lib/puppet/provider/vcsrepo]/ensure: created
Debug: Routes Registered:
Debug: Route /\/v3/
Debug: Route /^\/v2\.0/
Debug: Evaluating match for Route /\/v3/
Debug: Evaluating match for Route /^\/environments$/
Debug: Did not match path ("/file_content/plugins/puppet/provider/vcsrepo.rb")
Debug: Evaluating match for Route /.*/
Notice: /File[/tmp/var/lib/puppet/provider/vcsrepo.rb]/ensure: defined content as '{md5}dbd72590771291f1db23a41ac048ed9d'

note this portion:

Notice: /File[/tmp/var/lib/puppet]/ensure: created
Notice: /File[/tmp/var/lib/puppet/provider]/ensure: created
Notice: /File[/tmp/var/lib/puppet/provider/vcsrepo]/ensure: created

Comment by Henrik Lindberg [ 2015/01/26 ]

Reopened as a backport is wanted to 3.7.5.

The issue has already been resolved on master.

Comment by Thomas Hallgren [ 2015/01/29 ]

PR-3539 is the same commit rebased to stable.

Comment by Henrik Lindberg [ 2015/01/29 ]

Merged to stable at 966f81b

Comment by Kurt Wall [ 2015/01/29 ]

Verified in master at SHA=af02828c7b16aa9c3b9823ba21b9cd60425b9745. Using eric0's reproducer, here's an agent run:

# puppet agent --test --environment production
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: removed
Info: Caching catalog for g9nzizk8lqji3m6.delivery.puppetlabs.net
Info: Applying configuration version '1422565122'
Notice: current environment is production
Notice: /Stage[main]/Main/Node[default]/Notify[environment]/message: defined 'message' as 'current environment is production'
Notice: Applied catalog in 0.02 seconds
 
# puppet agent --test --environment fail
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo.rb]/ensure: defined content as '{md5}dbd72590771291f1db23a41ac048ed9d'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/bzr.rb]/ensure: defined content as '{md5}9304caa8c45685d741248fb167c47842'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/cvs.rb]/ensure: defined content as '{md5}0fed7f40f3de82c6490ac241cea46b3a'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/dummy.rb]/ensure: defined content as '{md5}4490fb75f044bd43855b2962b8f6e301'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/git.rb]/ensure: defined content as '{md5}8bd810767eafda46956f7477ba917459'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/hg.rb]/ensure: defined content as '{md5}65a45f4382ed5874eeea3f2c3e0729f1'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/p4.rb]/ensure: defined content as '{md5}b324e3141913337d5592ebbcb2a7544e'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/svn.rb]/ensure: defined content as '{md5}da00dfb4a67fd01d4756e6497d5ed7f1'
Notice: /File[/var/lib/puppet/lib/puppet/type]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/type/vcsrepo.rb]/ensure: defined content as '{md5}0427f46f99137cea53a45d19f2c918f0'
Info: Caching catalog for g9nzizk8lqji3m6.delivery.puppetlabs.net
Info: Applying configuration version '1422565330'
Notice: current environment is fail
Notice: /Stage[main]/Main/Node[default]/Notify[environment]/message: defined 'message' as 'current environment is fail'
Notice: /Stage[main]/Main/Node[default]/Vcsrepo[dotfiles]/ensure: Creating repository from present
Error: Execution of '/usr/bin/git submodule update --init --recursive' returned 1: No submodule mapping found in .gitmodules for path '.vim/bundle/criticmarkup-vim'
Error: /Stage[main]/Main/Node[default]/Vcsrepo[dotfiles]/ensure: change from absent to present failed: Execution of '/usr/bin/git submodule update --init --recursive' returned 1: No submodule mapping found in .gitmodules for path '.vim/bundle/criticmarkup-vim'
Info: Node[default]: Unscheduling all events on Node[default]
Notice: Applied catalog in 1.78 seconds

Log from the master:

puppet master --no-daemonize --trace --verbose
Notice: Starting Puppet master version 3.7.4
Info: access[/puppet/v2.0/environments]: allowing 'method' find
Info: access[/puppet/v2.0/environments]: allowing * access
Info: access[/puppet/v3/environments]: allowing 'method' find
Info: access[/puppet/v3/environments]: allowing * access
Info: access[^/puppet/v3/catalog/([^/]+)$]: allowing 'method' find
Info: access[^/puppet/v3/catalog/([^/]+)$]: allowing $1 access
Info: access[^/puppet/v3/node/([^/]+)$]: allowing 'method' find
Info: access[^/puppet/v3/node/([^/]+)$]: allowing $1 access
Info: access[^/puppet/v3/report/([^/]+)$]: allowing 'method' save
Info: access[^/puppet/v3/report/([^/]+)$]: allowing $1 access
Info: access[/puppet/v3/file]: allowing * access
Info: access[/puppet/v3/status]: allowing 'method' find
Info: access[/puppet/v3/status]: allowing * access
Info: access[/puppet-ca/v1/certificate_revocation_list/ca]: allowing 'method' find
Info: access[/puppet-ca/v1/certificate_revocation_list/ca]: allowing * access
Info: access[/puppet-ca/v1/certificate/ca]: adding authentication any
Info: access[/puppet-ca/v1/certificate/ca]: allowing 'method' find
Info: access[/puppet-ca/v1/certificate/ca]: allowing * access
Info: access[/puppet-ca/v1/certificate/]: adding authentication any
Info: access[/puppet-ca/v1/certificate/]: allowing 'method' find
Info: access[/puppet-ca/v1/certificate/]: allowing * access
Info: access[/puppet-ca/v1/certificate_request]: adding authentication any
Info: access[/puppet-ca/v1/certificate_request]: allowing 'method' find
Info: access[/puppet-ca/v1/certificate_request]: allowing 'method' save
Info: access[/puppet-ca/v1/certificate_request]: allowing * access
Info: access[/]: adding authentication any
Info: Caching node for g9nzizk8lqji3m6.delivery.puppetlabs.net
Info: Caching node for g9nzizk8lqji3m6.delivery.puppetlabs.net
Notice: Compiled catalog for g9nzizk8lqji3m6.delivery.puppetlabs.net in environment production in 0.37 seconds
Info: Caching node for g9nzizk8lqji3m6.delivery.puppetlabs.net
Info: Caching node for g9nzizk8lqji3m6.delivery.puppetlabs.net
Notice: Compiled catalog for g9nzizk8lqji3m6.delivery.puppetlabs.net in environment fail in 0.04 seconds

Comment by Kurt Wall [ 2015/01/29 ]

I get the same result if I run against the fail environment before the production environment.

# puppet agent --test --environment fail
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo.rb]/ensure: defined content as '{md5}dbd72590771291f1db23a41ac048ed9d'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/bzr.rb]/ensure: defined content as '{md5}9304caa8c45685d741248fb167c47842'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/cvs.rb]/ensure: defined content as '{md5}0fed7f40f3de82c6490ac241cea46b3a'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/dummy.rb]/ensure: defined content as '{md5}4490fb75f044bd43855b2962b8f6e301'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/git.rb]/ensure: defined content as '{md5}8bd810767eafda46956f7477ba917459'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/hg.rb]/ensure: defined content as '{md5}65a45f4382ed5874eeea3f2c3e0729f1'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/p4.rb]/ensure: defined content as '{md5}b324e3141913337d5592ebbcb2a7544e'
Notice: /File[/var/lib/puppet/lib/puppet/provider/vcsrepo/svn.rb]/ensure: defined content as '{md5}da00dfb4a67fd01d4756e6497d5ed7f1'
Notice: /File[/var/lib/puppet/lib/puppet/type]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/type/vcsrepo.rb]/ensure: defined content as '{md5}0427f46f99137cea53a45d19f2c918f0'
Info: Caching catalog for g9nzizk8lqji3m6.delivery.puppetlabs.net
Info: Applying configuration version '1422566428'
Notice: current environment is fail
Notice: /Stage[main]/Main/Node[default]/Notify[environment]/message: defined 'message' as 'current environment is fail'
Notice: Applied catalog in 1.54 seconds

Comment by Eric Thompson [ 2015/02/26 ]

Nicholas Fagerlund i think the fix makes it work the way you'd expect. If you have a custom type/provider in another environment that is broken, it should not produce an error in another environment. Previously the autoloader would load types and providers from the default environment (production) during one phase of the compilation.
The fix is to ensure we autoload types and providers from the same environment that is specified.

Comment by E Salberg [ 2015/02/26 ]

Not to hijack this ticket, but I just ran into a situation (on PE 3.7.2) where I updated a module with a custom type/provider, and while I could see the updated type/provider in my environments, the agent could not utilize them until I restarted the pe-puppetserver process. If we're talking about loading providers from specified environments, could there be a situation of needing to kick the pe-puppetserver process in between, or was my situation different / unrelated?

Comment by Henrik Lindberg [ 2015/02/26 ]

E Salberg The problem you are describing is most likely something else. This ticket is for a problem when the compiler validates a resource and picks a provider from the wrong environment. Sounds more related to similar problems with pluginsync.

Comment by Robin Bowes [ 2015/03/09 ]

I appear to be running into this very issue (with the puppetboard module, that uses vcsrepo).

Is there a fix/workaround?

Comment by Henrik Lindberg [ 2015/03/09 ]

A fix for this issue will be included in the Puppet 3.7.5 and 4.0.0 releases.

Comment by Robin Bowes [ 2015/03/10 ]

Thanks Henrik.

Comment by Christophe Bordier [ 2015/03/10 ]

root@thalassa:/root # puppet agent --test
/opt/puppet/lib/ruby/site_ruby/1.9.1/puppet/defaults.rb:214: warning: Insecure world writable dir /usr in PATH, mode 0240777
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for thalassa
Error: Failed to apply catalog: Parameter provider failed on File[/home/cissys]: Invalid file provider 'svn' at /etc/puppetlabs/puppet/modules/sync_scripts/manifests/init.pp:9
Wrapped exception:
Invalid file provider 'svn'

Please commit a fix for this, as this is the only way to get files synchronized with a source repository.
This issue is blokcing and is significantly bad for a framework named as in version "3" "dot" "7"

Here is a link with the same error as the others users on this bug
https://drive.google.com/file/d/0B-wGewYKm47hLW01R21ubTkwbkE/view?usp=sharing

Comment by Henrik Lindberg [ 2015/03/10 ]

Christophe Bordier Regarding your comment - how did you conclude it was this particular issue? It is already fixed, and as noted above will be released in Puppet 3.7.5 / Puppet 4.0.0 quite soon. It would be very helpful if you would like to try out the fix and see if it cures your problem. Or is that what you have already tried, and hence request "please commit a fix for this" - can you explain?

Comment by Christophe Bordier [ 2015/03/11 ]

Here is a link with the same error as the others users on this bug
https://drive.google.com/file/d/0B-wGewYKm47hLW01R21ubTkwbkE/view?usp=sharing
Please note that yesterday this page was filled with "Not Fixed" with a sprint date in the past, in January
And thanks for the admin editing my posts I love you
Waiting for 3.7.5 to be released

Comment by Henrik Lindberg [ 2015/03/11 ]

In general, when a fix-version is assigned the fix is planned to go into that release. The closer to actual release we get the more accurate that field is for the pending release. If code has been merged you can pretty much count on that it will be released in the next release. We do not mark something as "fixed/closed" until we actually release. The state "Resolved" is the highest state of being done that we use until it has been released.

If I understand your comment correctly Christophe Bordier the things you pointed to does not add any new information or additional cases to the already fixed problem.

Comment by Christophe Bordier [ 2015/03/11 ]

You can't say it is "already fixed" it will be "already fixed" when Puppet consumers will be using 3.7.5..., not the case at the moment...
Currently it is revised in the Puppet contributors repository.
I'm a French guy and words are important to not confuse anyone.

Comment by Henrik Lindberg [ 2015/03/11 ]

We use specific terms for the states of tickets. "Resoved Fixed" (the state this ticket is in now) means that it is fixed, but not yet released.

Generated at Wed Feb 26 22:56:56 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.