Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-9364

puppet module install is really slow

    Details

    • Template:
      PUP Bug Template
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      Puppet module install will now only download the release metadata it needs to perform dependency resolution, drastically reducing the amount of data transferred. For stdlib the reduction is from 25MB to 68KB, and any module that depends on stdlib will benefit when installed.
      Show
      Puppet module install will now only download the release metadata it needs to perform dependency resolution, drastically reducing the amount of data transferred. For stdlib the reduction is from 25MB to 68KB, and any module that depends on stdlib will benefit when installed.
    • QA Risk Assessment:
      Needs Assessment

      Description

      Puppet Version: 6.0.4
      Puppet Server Version: -
      OS Name/Version: Mac OS 10.14.1 (Probably all?)

      When running a lot of beaker tests, i realized that most time is used by `puppet module install ...` commands.

      Desired Behavior:

      Fast installation of module by using puppet module install ...

      Actual Behavior:

       $ ls -la modules/
      total 0
      drwxr-xr-x   2 mbaur staff   64 Dec 15 15:37 .
      drwxr-xr-x+ 76 mbaur staff 2432 Dec 15 15:37 ..
      mbaur(2018-12-15 15:37:26)  ~
      $ time puppet module install puppetlabs-firewall -v 1.14.0 --modulepath $HOME/modules/
      Notice: Preparing to install into /Users/mbaur/modules ...
      Notice: Downloading from https://forgeapi.puppet.com ...
      Notice: Installing -- do not interrupt ...
      /Users/mbaur/modules
      └─┬ puppetlabs-firewall (v1.14.0)
        └── puppetlabs-stdlib (v5.1.0)real	0m26.044s
      user	0m2.615s
      sys	0m1.211s

       

      I think the biggest problem is, that the command posted above makes 8 API calls to forgeapi.puppet.com which just take way to long:

      $ puppet module install puppetlabs-firewall -v 1.14.0 --modulepath $HOME/modules/ --debug | grep forgeapi
      Notice: Downloading from https://forgeapi.puppet.com ...
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version&limit=20&offset=20
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=20
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=40
      Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=60
      Debug: HTTP GET https://forgeapi.puppet.com/v3/files/puppetlabs-firewall-1.14.0.tar.gz
      Debug: HTTP GET https://forgeapi.puppet.com/v3/files/puppetlabs-stdlib-5.1.0.tar.gz
      

      I tried to measure the data which gets transfer:

      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version -> 1.4M
      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version&limit=20&offset=20 -> 621K
      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version -> 2.4M
      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=20 -> 1.1M
      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=40 -> 496K
      https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=60 -> 10K
      https://forgeapi.puppet.com/v3/files/puppetlabs-firewall-1.14.0.tar.gz -> 121K
      https://forgeapi.puppet.com/v3/files/puppetlabs-stdlib-5.1.0.tar.gz -> 242K
       
      Complete: 6.3M
      

      I think there is just way to much data which we need to retrieve for every module.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  josh Josh Cooper
                  Reporter:
                  baurmatt Matthias Baur
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Zendesk Support