Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-9707

Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins

    XMLWordPrintable

Details

    • Improvement
    • Status: Needs Information
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • Needs Assessment

    Description

      Background

      In PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script.

      On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down.

      Suggested Improvement

      The instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed.

      Out of scope

      My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball. Then in the future we can improve puppetserver to make the tarball if we want to. SERVER-2539

      Impact

      In order to show the impact of using the current bulk_pluginsync implementation we install the puppetlabs-windows module and then perform pluginsync via puppet or via bulk_pluginsync.

      When we look at the results we can see a puppet agent run without bulk_pluginsync takes around 90 seconds and with bulk_pluginsync it takes about 14 seconds. The download and extraction of the pluginsync tarball takes under 1 second and a reasonable part of the agent run after bulk_pluginsync seems to be validating the plugins.

      Puppet-agent ( one file at a time )

      /etc/puppetlabs/code/environments/production/modules
      └─┬ puppetlabs-windows (v6.1.0)
        ├── puppet-download_file (v3.2.0)
        ├── puppet-windows_env (v3.2.0)
        ├── puppet-windowsfeature (v3.2.2)
        ├── puppetlabs-acl (v2.1.0)
        ├─┬ puppetlabs-chocolatey (v3.3.0)
        │ ├── puppetlabs-powershell (v2.3.0)
        │ ├── puppetlabs-registry (v2.1.0)
        │ └── puppetlabs-stdlib (v5.2.0)
        ├─┬ puppetlabs-dsc (v1.9.2)
        │ └── puppetlabs-reboot (v2.1.2)
        ├── puppetlabs-iis (v4.5.1)
        └── puppetlabs-wsus_client (v1.1.0)
      [root@master201910-centos ~]#
      [root@master201910-centos ~]#
      [root@master201910-centos ~]#
      [root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/
      facts.d/       lib/           locales/       preview/       puppet-module/ state/
      [root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/
      [root@master201910-centos ~]# time puppet agent -t
      Info: Using configured environment 'production'
      Info: Retrieving pluginfacts
      Info: Retrieving plugin
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_build.rb]/ensure: defined content as '{md5}cdcc1ff07bc245c66cc1d46be56b3af5'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_version.rb]/ensure: defined content as '{md5}d05c8cbf788f47d33efd46a935dda61e'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/choco_install_path.rb]/ensure: defined content as '{md5}36ece697e936cc950f8946ca2ac11729'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/choco_temp_dir.rb]/ensure: defined content as '{md5}299356b5cbc6f0aed554294a69824c00'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/chocolateyversion.rb]/ensure: defined content as '{md5}941fd5e3ab6f1d54a5886d803d19bf22'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/facter_dot_d.rb]/ensure: defined content as '{md5}9f65108ecd7b8c5ce9c90ab4cd9d45be'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/iis_version.rb]/ensure: defined content as '{md5}8642e756efacbdbcca8d5ccc93026101'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/inventory.rb]/ensure: defined content as '{md5}e6c76fe99ac2a64fac1b168f36d47235'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb]/ensure: defined content as '{md5}12d57e65f9b90c13e7808c347fbdc0b0'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_build.rb]/ensure: defined content as '{md5}ee54c728457b32d6622c3985448918fa'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_concat_basedir.rb]/ensure: defined content as '{md5}0ccd3500f29b9dd346a45a61268c7c18'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_postgresql_info.rb]/ensure: defined content as '{md5}d65daa8950859bc7557b064203356565'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_razor_server_version.rb]/ensure: defined content as '{md5}ec91d8b92e03d5f952c789308d26dcd0'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_server_version.rb]/ensure: defined content as '{md5}17c2795fe8a56b731ae0fc81ba147e6a'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_version.rb]/ensure: defined content as '{md5}245bb8ea121e8a55e1a38668cc480530'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/platform_symlink_writable.rb]/ensure: defined content as '{md5}fc1e2766ff9994fa5df95cdc14b9bcd2'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/platform_tag.rb]/ensure: defined content as '{md5}2f3b09a0f0b539117456e311ad28f190'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/powershell_version.rb]/ensure: defined content as '{md5}50bab11a2d7c70d7cbbae618f060ceb8'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_files_dir_present.rb]/ensure: defined content as '{md5}3900e124be2f377638dd1522079856bf'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_settings.rb]/ensure: defined content as '{md5}d86bacf2b962a4744b701868cda31729'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/root_home.rb]/ensure: defined content as '{md5}b384092f0470b9e2177edaf546de2ec1'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/service_provider.rb]/ensure: defined content as '{md5}66cc42526eae631e306b397391f1f01c'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/should_install_shiro_ini.rb]/ensure: defined content as '{md5}a932ec299c2a33762e2713b57be9facd'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/staging_http_get.rb]/ensure: defined content as '{md5}2c27beb47923ce3acda673703f395e68'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/uses_win32console.rb]/ensure: defined content as '{md5}074d6dbc1aa6699acc1662a333a7eb3c'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/util]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/util/puppet_settings.rb]/ensure: defined content as '{md5}f69007bcc02031cd772fa7f2dffa3c0c'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/windows.rb]/ensure: defined content as '{md5}d8880f6f32905f040f3355e2a40cf088'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/windows_env.rb]/ensure: defined content as '{md5}d1d3a97851e655e4fc9c6a4945e0e670'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera/backend]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera/backend/hocon_backend.rb]/ensure: defined content as '{md5}47ff267a77a66b85dd98258c6e08a127'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/application]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/application/enterprise.rb]/ensure: defined content as '{md5}b0e6f1b94578dde29a3af39fe7652fb0'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face/enterprise]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face/enterprise.rb]/ensure: defined content as '{md5}c447263b49afebada8a4706654925a74'
      ...
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/powershell_manager.rb]/ensure: defined content as '{md5}e4194779ea96a79651ce0f82ed5180b5'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/powershell_version.rb]/ensure: defined content as '{md5}519ebd8f8b718c1afb2ce42f75b85c2a'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/hash.rb]/ensure: defined content as '{md5}1fb84a18a918aae2e76a3225aa0a2dbd'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/name.rb]/ensure: defined content as '{md5}473f314d46724c31b1f496b59031f64a'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/path.rb]/ensure: defined content as '{md5}fd3e53ba75d36e095d051ffded98c133'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/positive_integer.rb]/ensure: defined content as '{md5}9317877654cf840b4d10e912f1d20285'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/read_only.rb]/ensure: defined content as '{md5}297c977e0339cb123040998917400c70'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/string.rb]/ensure: defined content as '{md5}bf1cc4e3fc4cd8cd3159dd9c7bc8bf54'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/timeformat.rb]/ensure: defined content as '{md5}5d16de4de8c0c77bdfbed2f0c3f63f90'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config.rb]/ensure: defined content as '{md5}95b89bf089e175f17ee9047ca4f0fee6'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config/modify.rb]/ensure: defined content as '{md5}c5d7317c5a67dadac2f7770f32633591'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config/services.rb]/ensure: defined content as '{md5}aa0c7cd67e156bdb9fd8ce1560c17f03'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/defaults.rb]/ensure: defined content as '{md5}137d7250ef9872d5adb2f79a7767aa97'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/hiera_adapter.rb]/ensure: defined content as '{md5}c914281cb82008ad28b7bf81d0909a9a'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/meep_scope.rb]/ensure: defined content as '{md5}a6af7e4ca161715f37ec9213bf23745a'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/scope.rb]/ensure: defined content as '{md5}90beadb6836522b94e30dc717834dcb5'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/validate.rb]/ensure: defined content as '{md5}ec3acfa929bd8d0001c43276da9be870'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/compatible_powershell_version.rb]/ensure: defined content as '{md5}d85ac07f9b23964804f400737c846fb8'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/powershell_manager.rb]/ensure: defined content as '{md5}1d6efc75a7fb8f95b02307a74e135fd3'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/powershell_version.rb]/ensure: defined content as '{md5}0c8298e41ff0216f1f6ad32d2a5ce010'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell_manager.rb]/ensure: defined content as '{md5}71fc45bf2d401b718561080ac43bb0d2'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell_version.rb]/ensure: defined content as '{md5}662d10a58e5e4512f3e59bbd72bf4732'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/registry.rb]/ensure: defined content as '{md5}1132a6f66c4e7d549ac90f9f58bd73e2'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1/puppet-enterprise-support.ps1]/ensure: defined content as '{md5}167b5cbb2af5909322e474425d404fa5'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1/puppet-enterprise-support.sh]/ensure: defined content as '{md5}5d3869be504e11fcfd44124a68ab1991'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v3]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v3/puppet-enterprise-support.rb]/ensure: defined content as '{md5}4376e0c821978eb58b8070fcd27959dd'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/dsc]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/dsc/init_ps.ps1]/ensure: defined content as '{md5}ddaf94c38f20a293e040aed56d755edb'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/iis]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/iis/init_ps.ps1]/ensure: defined content as '{md5}afa7397ca63c05a6fd35425191cc2d90'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/powershell]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/powershell/init_ps.ps1]/ensure: defined content as '{md5}065293e35b7d035f5c54a26146106fec'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/aio_build.rb]/ensure: defined content as '{md5}0b71fe86c349f3f41401313c0ac03a71'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/pe_build.rb]/ensure: defined content as '{md5}4f4652af20c4f0391b9ca2976940a710'
      Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/pe_server_version.rb]/ensure: defined content as '{md5}f3d3fc8776512ae73d3293c97b8f3dfe'
      Info: Retrieving locales
      Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja]/ensure: created
      Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-stdlib.po]/ensure: defined content as '{md5}805e5d893d2025ad57da8ec0614a6753'
      Info: Loading facts
      Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::pe_metric_curl_cron_jobs for master201910-centos.puppetdebug.vlan (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 21, column: 5) on node master201910-centos.puppetdebug.vlan
      Warning: Not using cache on failed catalog
      Error: Could not retrieve catalog; skipping run
       
      real	1m44.931s
      user	0m25.433s
      sys	0m5.240s
      ...
      real	1m30.611s
      user	0m24.526s
      sys	0m5.231s
      ...
      real	1m29.392s
      user	0m24.535s
      sys	0m5.022s
      

      bulk_pluginsync ( download a tarball and extract it)

      [root@master201910-centos ~]# time ./bulk_psync.sh
      bulk downloading plugins
      extracting plugins
      Info: Using configured environment 'production'
      Info: Retrieving pluginfacts
      Info: Retrieving plugin
      Info: Retrieving locales
      Info: Loading facts
      Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::pe_metric_curl_cron_jobs for master201910-centos.puppetdebug.vlan (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 21, column: 5) on node master201910-centos.puppetdebug.vlan
      Warning: Not using cache on failed catalog
      Error: Could not retrieve catalog; skipping run
       
      real	0m13.542s
      user	0m7.250s
      sys	0m0.647s
      ...
      real	0m14.502s
      user	0m7.206s
      sys	0m0.767s
      ...
      real	0m13.055s
      user	0m6.994s
      sys	0m0.574s
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              nick.walker Nick Walker
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Zendesk Support