Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-8662

puppet plugin download doesn't re-use HTTP connections

    Details

    • Template:
      PUP Bug Template
    • Epic Link:
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Inspection
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      puppet plugin download will reuse HTTPS connections significantly speeding up the download process.
    • QA Risk Assessment:
      Needs Assessment

      Description

      The puppet plugin download command doesn't re-use connections to the
      Puppet Server when downloading plugins from modules. This adds considerable time to
      downloads along with extra load on the Puppet Server due to multiple redundant TLS handshakes.

      Reproduction Case

      • Install puppet-agent 5.5.0 and puppetserver 5.3.0 on a CentOS 7
        node:

      rpm -Uvh http://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm
      yum install -y puppet-agent-5.5.0 puppetserver-5.3.0
      

      • Configure the puppet agent to connect to the local host and start
        puppetserver:

      /opt/puppetlabs/bin/puppet config set --section main server $(hostname -f)
      systemctl start puppetserver
      

      • Run puppet agent -t to verify things work.
      • Install the puppetlabs-windows module, which brings in a ton of
        plugins that will have to be synced:

      /opt/puppetlabs/bin/puppet module install puppetlabs-windows
      

      • Empty the plugin cache, and time a puppet agent run along with the
        number of network connections it opens:

      rm -rf /opt/puppetlabs/puppet/cache/lib
      time /opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -e Info: -e 'opening connection'
      

      • Empty the cache again and time a puppet plugin download run along
        with the number of network connections opened:

      rm -rf /opt/puppetlabs/puppet/cache/lib
      time /opt/puppetlabs/bin/puppet plugin download --verbose --http_debug 2>&1|grep -e Info: -e 'opening connection'
      

      Outcome

      The puppet agent run uses a handful of connections:

      # rm -rf /opt/puppetlabs/puppet/cache/lib
      # time /opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -e Info: -e 'opening connection'
       
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      Info: Using configured environment 'production'
      Info: Retrieving pluginfacts
      Info: Retrieving plugin
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      Info: Retrieving locales
      Info: Loading facts
      Info: Caching catalog for wzwhytp204ipraf.delivery.puppetlabs.net
      Info: Applying configuration version '1523665266'
       
      real	0m43.990s
      user	0m16.685s
      sys	0m1.497s
      

      The puppet plugin download run uses hundreds of connections and takes much
      longer than the puppet agent run despite only doing a subset of the work:

      # rm -rf /opt/puppetlabs/puppet/cache/lib
      # time /opt/puppetlabs/bin/puppet plugin download --verbose --http_debug 2>&1|grep -e Info: -e 'opening connection'
       
      Info: Retrieving pluginfacts
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      Info: Retrieving plugin
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      ... omitted hundreds of open calls for brevity ...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
      Info: Retrieving locales
      opening connection to wzwhytp204ipraf.delivery.puppetlabs.net:8140...
       
      real  3m39.345s
      user  1m29.576s
      sys 0m2.405s
      

      Expected Outcome

      The puppet plugin download action should complete much faster than
      puppet agent -t when populating a cold plugin cache.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  josh Josh Cooper
                  Reporter:
                  chuck Charlie Sharpsteen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Zendesk Support