[PUP-9364] puppet module install is really slow - Puppet Jira Server

XML

Word

Printable

Type: Bug
Status: Closed
Priority: Normal
Resolution: Fixed
Affects Version/s: None
Fix Version/s: PUP 5.5.10, PUP 6.0.5, PUP 6.2.0
Component/s: Docs, Modules
Labels:
- resolved-issue-added
Environment:

Mac 10.14.1
Internet Connection Speed: Tested on 50MBits and 1 Gbit

Release Notes:
Bug Fix
Release Notes Summary:

Hide
Puppet module install will now only download the release metadata it needs to perform dependency resolution, drastically reducing the amount of data transferred. For stdlib the reduction is from 25MB to 68KB, and any module that depends on stdlib will benefit when installed.

Show
Puppet module install will now only download the release metadata it needs to perform dependency resolution, drastically reducing the amount of data transferred. For stdlib the reduction is from 25MB to 68KB, and any module that depends on stdlib will benefit when installed.

Puppet Version: 6.0.4
Puppet Server Version: -
OS Name/Version: Mac OS 10.14.1 (Probably all?)

When running a lot of beaker tests, i realized that most time is used by `puppet module install ...` commands.

Desired Behavior:

Fast installation of module by using puppet module install ...

Actual Behavior:

 $ ls -la modules/

total 0

drwxr-xr-x   2 mbaur staff   64 Dec 15 15:37 .

drwxr-xr-x+ 76 mbaur staff 2432 Dec 15 15:37 ..

mbaur(2018-12-15 15:37:26)  ~

$ time puppet module install puppetlabs-firewall -v 1.14.0 --modulepath $HOME/modules/

Notice: Preparing to install into /Users/mbaur/modules ...

Notice: Downloading from https://forgeapi.puppet.com ...

Notice: Installing -- do not interrupt ...

/Users/mbaur/modules

└─┬ puppetlabs-firewall (v1.14.0)

  └── puppetlabs-stdlib (v5.1.0)real	0m26.044s

user	0m2.615s

sys	0m1.211s

I think the biggest problem is, that the command posted above makes 8 API calls to forgeapi.puppet.com which just take way to long:

$ puppet module install puppetlabs-firewall -v 1.14.0 --modulepath $HOME/modules/ --debug | grep forgeapi

Notice: Downloading from https://forgeapi.puppet.com ...

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version&limit=20&offset=20

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=20

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=40

Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=60

Debug: HTTP GET https://forgeapi.puppet.com/v3/files/puppetlabs-firewall-1.14.0.tar.gz

Debug: HTTP GET https://forgeapi.puppet.com/v3/files/puppetlabs-stdlib-5.1.0.tar.gz

I tried to measure the data which gets transfer:

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version -> 1.4M

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-firewall&sort_by=version&limit=20&offset=20 -> 621K

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version -> 2.4M

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=20 -> 1.1M

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=40 -> 496K

https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=60 -> 10K

https://forgeapi.puppet.com/v3/files/puppetlabs-firewall-1.14.0.tar.gz -> 121K

https://forgeapi.puppet.com/v3/files/puppetlabs-stdlib-5.1.0.tar.gz -> 242K

Complete: 6.3M

I think there is just way to much data which we need to retrieve for every module.

is blocked by

FORGE-471 API pagination links mangle value of "exclude_fields" query param