Failure to fetch node definition may result in bad pluginsync and cascading failure

Puppet Version: all

Early on in the agent run process, Puppet calls an indirector to choose an environment to pluginsync against. If anything goes wrong with that call, pretty much anything at all, Puppet will pluginsync against the agent-configured environment (usually "production") and spit out a warning.

Warning: Unable to fetch my node definition, but the agent run will continue

Desired Behavior:

Puppet should pluginsync against the last-used environment, or alternatively, fail fast. At customer sites where "production" is not used, pluginsyncing against what is probably an empty environment has been observed to add ten minutes+ of time to a Puppet agent run, as it first deletes all of its plugins, and then has to re-sync them fresh from the correct environment after it switches over. This can negatively affect the delivery of changes that need to be performed within a change control window.

Actual Behavior:

Puppet pluginsyncs against production, or the environment value set with --environment or in puppet.conf.

Reproduction:

• Create an environment "r1" and "production"
• Pin a test agent to "r1"
• Edit /etc/puppetlabs/puppetserver/conf.d/auth.conf to disallow the test agent from requesting the /puppet/v3/node path
• Run puppet agent -t on the test agent

The agent will produce the expected error, pluginsync against production, retrieve a catalog, switch to r1, pluginsync against r1, retrieve a catalog, and apply it.

The kind of problem this can cause may be made clearly evident if you supply a fact in environment r1 that is needed to compile an r1 catalog. See below.

[root@pe-xl-compiler-2 ~]# puppet agent -t

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/pe-xl-compiler-2.dev36.puppet.vm (method :get). Please see the server logs for details.

Info: Retrieving pluginfacts

Notice: /File[/opt/puppetlabs/puppet/cache/facts.d/r1.txt]/ensure: removed

Info: Retrieving plugin

Info: Retrieving locales

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Fact r1_fact does not exist! (file: /etc/puppetlabs/code/environments/r1/manifests/site.pp, line: 29, column: 5) on node pe-xl-compiler-2.dev36.puppet.vm

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Specifying r1 with the environment flag mitigates the issue, and demonstrates what a better behavior might look like when this problem occurs.

[root@pe-xl-compiler-2 ~]# puppet agent -t --environment=r1

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/pe-xl-compiler-2.dev36.puppet.vm (method :get). Please see the server logs for details.

Info: Retrieving pluginfacts

Notice: /File[/opt/puppetlabs/puppet/cache/facts.d/r1.txt]/ensure: defined content as '{md5}13b8114b76258d0104dd0568b031361d'

Info: Retrieving plugin

Info: Retrieving locales

Info: Loading facts

Info: Caching catalog for pe-xl-compiler-2.dev36.puppet.vm

Info: Applying configuration version 'pe-xl-core-0-r1-bb4c21dbce5'

Notice: Applied catalog in 14.23 seconds