Details
-
Bug
-
Status: Closed
-
Normal
-
Resolution: Fixed
-
None
-
None
-
Coremunity
-
Platform Core KANBAN
-
Needs Assessment
-
Bug Fix
-
Catalog compilations for a newly created environment directory could fail if the environment was listed while the directory and its contents were being created. This issue only occurred when using an environment_timeout value greater than 0.
-
Needs Assessment
Description
If environments are listed via the environment_classes REST API while a new environment directory is deployed, it's possible for the default modulepath to be cached in settings, and cause future compilations with that environment to fail. It's can also cause agents to randomly delete pluginsync'ed files (due to omissions in the file metadata response).
On the puppetserver node, allow access to the REST API to listing and delete environments:
{
|
"allow": "ADD SERVER FQDN",
|
"match-request": {
|
"method": "delete",
|
"path": "/puppet-admin-api/v1/environment-cache",
|
"query-params": {},
|
"type": "path"
|
},
|
"name": "puppetlabs environment cache",
|
"sort-order": 500
|
},
|
{
|
"allow": "ADD SERVER FQDN",
|
"match-request": {
|
"method": "get",
|
"path": "/puppet/v3/environment_classes",
|
"query-params": {},
|
"type": "path"
|
},
|
"name": "puppetlabs environment classes",
|
"sort-order": 500
|
},
|
Then run:
#!/bin/sh
|
|
|
server="ADD SERVER NAME HERE" |
certname="${server}" |
|
|
# use server cert for privileged REST APIs
|
curl_opts="-kf --cert /etc/puppetlabs/puppet/ssl/certs/${server}.pem --key /etc/puppetlabs/puppet/ssl/private_keys/${server}.pem" |
|
|
api_environments="https://${server}:8140/puppet/v3/environments" |
api_classes="https://${server}:8140/puppet/v3/environment_classes" |
api_environment_cache="https://${server}:8140/puppet-admin-api/v1/environment-cache" |
api_catalog="https://${server}:8140/puppet/v3/catalog" |
api_catalog4="https://${server}:8140/puppet/v4/catalog" |
|
|
name="env_${RANDOM}" |
basedir="/etc/puppetlabs/code/environments" |
envdir="${basedir}/${name}" |
mkdir -p "${envdir}" |
chown puppet:puppet "${envdir}" |
echo "create random environment: ${name}" |
|
|
echo "list environments, cache modulepath" |
curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name} |
|
|
echo "finish deployment" |
envconf="${envdir}/environment.conf" |
cat <<EOF > "${envconf}" |
modulepath=site:dist:\$basemodulepath
|
EOF
|
|
|
mkdir "${envdir}/manifests" |
cat <<EOF > "${envdir}/manifests/init.pp" |
include role
|
EOF
|
|
|
mkdir "${envdir}/dist" |
mkdir -p "${envdir}/site/role/manifests" |
cat <<EOF > "${envdir}/site/role/manifests/init.pp" |
class role {
|
notify { 'in role': } |
}
|
EOF
|
chown -R puppet:puppet "${envdir}" |
|
|
echo mark environments as expired |
curl ${curl_opts} -X DELETE -H 'Accept: application/json' ${api_environment_cache} |
|
|
echo we see the modulepath from before the deploy finished |
curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name} |
|
|
echo compile v3 catalog fails |
curl -kf \
|
--cert "/etc/puppetlabs/puppet/ssl/certs/${certname}.pem" \ |
--key "/etc/puppetlabs/puppet/ssl/private_keys/${certname}.pem" \ |
-X POST \
|
-d "%257B%2522facts_format%2522%253A%2522application%252Fjson%2522%252C%2522facts%2522%253A%257B%2522name%2522%253A%2522${certname}%2522%257D%257D" \ |
-H 'Content-Type: application/x-www-form-urlencoded' \ |
-H 'Accept: application/json' \ |
${api_catalog}/${certname}\?environment\=${name}
|
Due to the bug, we expire all environments, but still see the cached environment's modulepath and compilation fails:
...
|
mark environments as expired
|
we see the modulepath from before the deploy finished
|
% Total % Received % Xferd Average Speed Time Time Time Current
|
Dload Upload Total Spent Left Speed
|
100 6900 100 6900 0 0 87341 0 --:--:-- --:--:-- --:--:-- 87341
|
{
|
"settings": {
|
"modulepath": [
|
"/etc/puppetlabs/code/environments/env_1077/modules",
|
"/etc/puppetlabs/code/modules",
|
"/opt/puppetlabs/puppet/modules"
|
],
|
"manifest": "/etc/puppetlabs/code/environments/env_1077/manifests",
|
"environment_timeout": 0,
|
"config_version": ""
|
}
|
}
|
compile v3 catalog fails
|
curl: (22) The requested URL returned error: 500 Server Error
|
And puppetserver.log contains:
2021-02-25T20:56:08.058Z ERROR [qtp1507614676-35] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::role for afferent-trot.delivery.puppetlabs.net (file: /etc/puppetlabs/code/environments/env_32225/manifests/init.pp, line: 1, column: 1) on node afferent-trot.delivery.puppetlabs.net
|
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:401:in `block in evaluate_classes'
|
org/jruby/RubyArray.java:2572:in `collect'
|
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:400:in `evaluate_classes'
|