Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10942

Listing environments during code deploys can prevent environment cache invalidation

    XMLWordPrintable

    Details

    • Template:
      PUP Bug Template
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Catalog compilations for a newly created environment directory could fail if the environment was listed while the directory and its contents were being created. This issue only occurred when using an environment_timeout value greater than 0.
    • QA Risk Assessment:
      Needs Assessment

      Description

      If environments are listed via the environment_classes REST API while a new environment directory is deployed, it's possible for the default modulepath to be cached in settings, and cause future compilations with that environment to fail. It's can also cause agents to randomly delete pluginsync'ed files (due to omissions in the file metadata response).

      On the puppetserver node, allow access to the REST API to listing and delete environments:

         {
                  "allow": "ADD SERVER FQDN",
                  "match-request": {
                      "method": "delete",
                      "path": "/puppet-admin-api/v1/environment-cache",
                      "query-params": {},
                      "type": "path"
                  },
                  "name": "puppetlabs environment cache",
                  "sort-order": 500
              },
              {
                  "allow": "ADD SERVER FQDN",
                  "match-request": {
                      "method": "get",
                      "path": "/puppet/v3/environment_classes",
                      "query-params": {},
                      "type": "path"
                  },
                  "name": "puppetlabs environment classes",
                  "sort-order": 500
              },
      

      Then run:

      #!/bin/sh
       
      server="ADD SERVER NAME HERE"
      certname="${server}"
       
      # use server cert for privileged REST APIs
      curl_opts="-kf --cert /etc/puppetlabs/puppet/ssl/certs/${server}.pem --key /etc/puppetlabs/puppet/ssl/private_keys/${server}.pem"
       
      api_environments="https://${server}:8140/puppet/v3/environments"
      api_classes="https://${server}:8140/puppet/v3/environment_classes"
      api_environment_cache="https://${server}:8140/puppet-admin-api/v1/environment-cache"
      api_catalog="https://${server}:8140/puppet/v3/catalog"
      api_catalog4="https://${server}:8140/puppet/v4/catalog"
       
      name="env_${RANDOM}"
      basedir="/etc/puppetlabs/code/environments"
      envdir="${basedir}/${name}"
      mkdir -p "${envdir}"
      chown puppet:puppet "${envdir}"
      echo "create random environment: ${name}"
       
      echo "list environments, cache modulepath"
      curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name}
       
      echo "finish deployment"
      envconf="${envdir}/environment.conf"
      cat <<EOF > "${envconf}"
      modulepath=site:dist:\$basemodulepath
      EOF
       
      mkdir "${envdir}/manifests"
      cat <<EOF > "${envdir}/manifests/init.pp"
      include role
      EOF
       
      mkdir "${envdir}/dist"
      mkdir -p "${envdir}/site/role/manifests"
      cat <<EOF > "${envdir}/site/role/manifests/init.pp"
      class role {
        notify { 'in role': }
      }
      EOF
      chown -R puppet:puppet "${envdir}"
       
      echo mark environments as expired
      curl ${curl_opts} -X DELETE -H 'Accept: application/json' ${api_environment_cache}
       
      echo we see the modulepath from before the deploy finished
      curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name}
       
      echo compile v3 catalog fails
      curl -kf \
       --cert "/etc/puppetlabs/puppet/ssl/certs/${certname}.pem" \
       --key "/etc/puppetlabs/puppet/ssl/private_keys/${certname}.pem" \
       -X POST \
       -d "%257B%2522facts_format%2522%253A%2522application%252Fjson%2522%252C%2522facts%2522%253A%257B%2522name%2522%253A%2522${certname}%2522%257D%257D" \
       -H 'Content-Type: application/x-www-form-urlencoded' \
       -H 'Accept: application/json' \
       ${api_catalog}/${certname}\?environment\=${name}
      

      Due to the bug, we expire all environments, but still see the cached environment's modulepath and compilation fails:

      ...
      mark environments as expired
      we see the modulepath from before the deploy finished
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  6900  100  6900    0     0  87341      0 --:--:-- --:--:-- --:--:-- 87341
      {
        "settings": {
          "modulepath": [
            "/etc/puppetlabs/code/environments/env_1077/modules",
            "/etc/puppetlabs/code/modules",
            "/opt/puppetlabs/puppet/modules"
          ],
          "manifest": "/etc/puppetlabs/code/environments/env_1077/manifests",
          "environment_timeout": 0,
          "config_version": ""
        }
      }
      compile v3 catalog fails
      curl: (22) The requested URL returned error: 500 Server Error
      

      And puppetserver.log contains:

      2021-02-25T20:56:08.058Z ERROR [qtp1507614676-35] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::role for afferent-trot.delivery.puppetlabs.net (file: /etc/puppetlabs/code/environments/env_32225/manifests/init.pp, line: 1, column: 1) on node afferent-trot.delivery.puppetlabs.net
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:401:in `block in evaluate_classes'
      org/jruby/RubyArray.java:2572:in `collect'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:400:in `evaluate_classes'
      

        Attachments

          Activity

            People

            Assignee:
            josh Josh Cooper
            Reporter:
            josh Josh Cooper
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Zendesk Support