Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-4810

Puppet caches parse results when environment_timeout is set to 0

    Details

    • Template:
    • Story Points:
      3
    • Sprint:
      Language 2015-07-08, Language 2015-07-22
    • CS Priority:
      Reviewed
    • Release Notes:
      Bug Fix

      Description

      When environment_timeout is set to 0, caching of parsed manifests is essentially a no-op as any object put into the cache is immediately marked as expired. However, Puppet still builds a cache when the timeout is set to 0 which consumes memory. This memory usage can be significant, especially when classifier sync hits the resource_types/* endpoint on each environment.

      Reproduction Case

      • Install puppet-agent and puppetserver.
      • Edit /opt/puppetlabs/puppet/puppet.conf and set environment_timeout to 0 in the master section.
      • Edit /opt/puppetlabs/puppet/auth.conf and allow unrestricted access to the environments and resource_types endpoints:

      path /puppet/v3/environments
      auth any
      method find
      allow *
       
      path /puppet/v3/resource_type
      auth any
      method search
      allow *
      

      • Create a large number of classes in the production environment, and then a large number of copies of that environment (the number created is a bit excessive since the classes are simple, but real-world environments have shown this behavior with only hundreds of classes):

      mkdir -p /etc/puppetlabs/code/environments/production/modules/test_classes/manifests
       
      cat <<EOF | /opt/puppetlabs/puppet/bin/erb > /etc/puppetlabs/code/environments/production/modules/test_classes/manifests/init.pp
      <% 100000.times do |i| %>
      class test_class_<%= i %> () {}
      <% end %>
      EOF
       
      for i in $(seq 100);do cp -r /etc/puppetlabs/code/environments/production /etc/puppetlabs/code/environments/test_${i}; done
      

      • Restart the puppetserver.
      • Simulate classifier syncs using the following script:

      #!/bin/bash
       
      get_env_list() {
        curl -s -k \
          https://localhost:8140/puppet/v3/environments |\
          /opt/puppetlabs/puppet/bin/ruby -rjson -e \
          'puts JSON.load($stdin.read())["environments"].keys.sort'
      }
       
      get_env_classes() {
        start_time=`date +%s`
        curl -s -k \
          "https://localhost:8140/puppet/v3/resource_types/*?environment=$1" > /tmp/${1}_classes.json;
        echo "Sync time: $(expr `date +%s` - $start_time) seconds";
      }
       
      get_all_classes() {
        for env in $(get_env_list); do
          echo "Fectching classes for: $env";
          get_env_classes $env
        done
      }
       
      while :; do get_all_classes; done
      

      Outcome

      During class sync, each class is persisted to the cache despite environment_timeout being set to zero. This causes the Puppet Server to run out of memory:

      Expected Outcome

      When `environment_timeout` is set to 0, no memory is wasted on caching classes.

      Applying a quick spike for this change:

      diff --git a/puppet/environments.rb b/puppet/environments.rb
      index fa5a028..0ac12b7 100644
      --- a/puppet/environments.rb
      +++ b/puppet/environments.rb
      @@ -304,7 +304,8 @@ module Puppet::Environments
             if result = @cache[name]
               return result.value
             elsif (result = @loader.get(name))
      -        @cache[name] = entry(result)
      +        e = entry(result)
      +        @cache[name] = e unless e.is_a?(NotCachedEntry)
               result
             end
           end
      

      Results in the Puppet Server being able to sustain syncing a large number of classes:

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  chuck Charlie Sharpsteen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: