Uploaded image for project: 'Modules'
  1. Modules
  2. MODULES-9062

puppet_agent: Setting agent's package_version, then upgrading the master, results in failed catalog retrieval


    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: puppet_agent 2.1.1
    • Fix Version/s: None
    • Component/s: puppet_agent
    • Labels:
    • Template:
    • Acceptance Criteria:

      We should not break agent management if package_version is set to an already installed, supported agent version, for instance immediately after a PE master Z upgrade.

      We should produce useful notices, warnings, or errors when setting package_version to an invalid version for the PE master's current version.

      We should not break agent management if package_version is set to an already installed, supported agent version, for instance immediately after a PE master Z upgrade. We should produce useful notices, warnings, or errors when setting package_version to an invalid version for the PE master's current version.
    • Team:
      Night's Watch
    • Story Points:
    • Sprint:
      PR - 2019-05-29, PR - 2019-06-12, PR - 2019-06-25, PR - 2019-07-10, PR - 2019-07-23, NW - 2019-08-07, NW - 2019-08-21
    • Zendesk Ticket IDs:
    • Zendesk Ticket Count:
    • Release Notes:
      Not Needed
    • QA Risk Assessment:
      Needs Assessment


      Steps to reproduce

      1. Install a PE 2018.1.7 master and AIX 7.1 POWER agent.
      2. Add the pe_repo::platform::aix-7.1-power class to the master.
      3. Run puppet agent --test on the master.
      4. Install the puppet_agent module.
      5. Classify the agent node: puppet_agent::package_version = '5.5.10'
      6. Run puppet agent --test on the agent node and confirm that Puppet takes no action.
      7. Upgrade the master to PE 2018.1.8.
      8. Run puppet agent --test on the agent node.

      I haven't tried to reproduce on other operating systems.

      Expected behavior

      PE 2018.1.7 ships with agent 5.5.10. PE 2018.1.8 ships with agent 5.5.14. This is a supported upgrade path on all agent operating systems, and we support agents running 5.5.10 against a master running 2018.1.8.

      The puppet_agent module should either:

      • do nothing—agent package_version 5.5.10 is already installed and is still supported, so I expect that specifying the already-installed package_version would be idempotent and not cause Puppet to fail.
      • warn or notice that agent 5.5.10 is not available from the 2018.1.8 package repo, then continue—I expect the older agent to still be supported after upgrading the master and not fail, but would appreciate being notified that the agent can be upgraded.
      • check for agent 5.5.10 in 2018.1.7's packaging path—if I've provided package_version 5.5.10, and a 5.5.10 package is available on the master, nothing in the module documentation suggests that it would fail.

      Observed behavior

      The agent fails to retrieve the catalog and falls out of management:

      Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not get metadata for puppet:///pe_packages/2018.1.8/aix-7.1-power/puppet-agent-5.5.10-1.aix7.1.ppc.rpm

      The agent is looking for an RPM of the already installed agent, but the 5.5.10 package doesn't exist in the 2018.1.8 path on the upgraded master version.

      A 5.5.10 package still exists on the 2018.1.7 path on the master, but the module doesn't check for it:

      lrwxrwxrwx 1 root root 73 Apr 22 12:58 /opt/puppetlabs/server/data/packages/public/2018.1.7/aix-7.1-power -> /opt/puppetlabs/server/data/packages/public/2018.1.7/aix-7.1-power-5.5.10
      -rw-r--r-- 1 root root 16596 May 2 11:02 /opt/puppetlabs/server/data/packages/public/2018.1.7/aix-7.1-power.bash
      lrwxrwxrwx 1 root root 73 May 3 12:23 /opt/puppetlabs/server/data/packages/public/2018.1.8/aix-7.1-power -> /opt/puppetlabs/server/data/packages/public/2018.1.8/aix-7.1-power-5.5.14
      -rw-r--r-- 1 root root 16695 May 3 12:23 /opt/puppetlabs/server/data/packages/public/2018.1.8/aix-7.1-power.bash

      This appears to be the result of the $source line in aix.pp, specifically when running with otherwise default settings:

       $source = "${::puppet_agent::aix_source}/${pe_server_version}/aix-${aix_ver_number}-power/${::puppet_agent::package_name}-${::puppet_agent::package_version}-1.aix${aix_ver_number}.ppc.rpm"


      • puppet_agent::aix_source = puppet:///pe_packages
      • pe_server_version = 2018.1.8 (as reported by the master)
      • aix_ver_number = 7.1
      • puppet_agent::package_version = 5.5.10 (what the user had already provided under 2018.1.7)

      results in a path of:


      which does not exist, resulting in the 500 error, which breaks catalog retrieval.

      User impact

      Specifying a supported agent version to a supported version of the module in a supported manner, with a supported master, should not break management on the agent.

      We support running most older versions of Puppet agent against newer masters. If the package_version matches the agent's current version but the module assembles a path to an agent package that doesn't exist on the upgraded master, we should issue a notice or warning, but we should not break catalog retrieval.

      Agent runs without providing the package_version parameter should work, but we require users to provide package_version in order to upgrade agents, and our documentation does not tell users to remove or revise the parameter after completing an agent upgrade. Therefore, after upgrading agents once using the module, I'd then expect any subsequent PE master upgrades to immediately break management at least on AIX agents until package_version is removed or revised.

      If we intentionally want specifying an older package_version to break, we should revise our agent upgrade documentation to note that this is intentional, revise the steps to provide a resolution, and improve the current error message, which is unclear to users unfamiliar with how PE masters distribute agent packages and how that directory structure is organized.

      If we want users to remove or revise the package_version parameter after upgrading the master, we should document that as part of the agent upgrade documentation.

      Related tickets

      MODULES-8923 — Requests that the module be able to detect the appropriate package_version based on the master version. Implementing that feature would allow users to to upgrade agents immediately after the master is upgraded by unsetting package_version.

      Related documentation

      https://puppet.com/docs/pe/2018.1/upgrading_agents.html — The first note on the page instructs users to take steps that would result in this error if package_version is already set from a previous upgrade and has not been removed or revised (emphasis added):

      Before upgrading agents, first upgrade your master and verify that the master and agent software versions are compatible. Then after upgrade, run Puppet on your agents as soon as possible to verify that agents have the correct configuration and that your systems are behaving as expected.


          Issue Links



              • Assignee:
                gheorghe.popescu Gheorghe Popescu
                garrett.guillotte Garrett Guillotte
              • Votes:
                0 Vote for this issue
                7 Start watching this issue


                • Created:

                  Zendesk Support