Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-3931

Upgrading Puppet package within Puppet agent run on EL7 results in bad package state

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: PUP 3.7.4
    • Fix Version/s: PUP 3.7.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      EL7 (tested on CentOS 7)

    • Template:

      Description

      Under CentOS 7, having the puppet package upgrade under the daemonized instance of the puppet agent results in a broken package state.

      Installation via manual yum update or a one-off puppet run (not the daemonized agent) works fine, but when the update operation occurs within the daemonized puppet run, systemd seems to squash the running agent along with its yum process - the update never fully cleans up the old package, both are left showing as installed, and puppet keeps unsuccessfully trying to upgrade the old package.

      Manual:

      [root@test ~]# rpm -qa | grep puppet
      puppet-3.7.3-1.el7.noarch
      [root@test ~]# yum update puppet-3.7.4
      ...
      Resolving Dependencies
      --> Running transaction check
      ---> Package puppet.noarch 0:3.7.3-1.el7 will be updated
      ---> Package puppet.noarch 0:3.7.4-1.el7 will be an update
      ...
      Running transaction
        Updating   : puppet-3.7.4-1.el7.noarch                                                                                                                             1/2
        Cleanup    : puppet-3.7.3-1.el7.noarch                                                                                                                             2/2
        Verifying  : puppet-3.7.4-1.el7.noarch                                                                                                                             1/2
        Verifying  : puppet-3.7.3-1.el7.noarch                                                                                                                             2/2
       
      Updated:
        puppet.noarch 0:3.7.4-1.el7
       
      Complete!
      [root@test ~]# rpm -qa | grep puppet
      puppet-3.7.4-1.el7.noarch
      

      One-off:

      [root@test ~]# rpm -qa | grep puppet
      puppet-3.7.3-1.el7.noarch
      [root@test ~]# puppet apply -e "package { "puppet": ensure => latest }"
      ...
      [root@test ~]# rpm -qa | grep puppet
      puppet-3.7.4-1.el7.noarch
      

      Within daemonized agent (syslog):

      Jan 30 20:04:29 test systemd: Starting Puppet agent...
      Jan 30 20:04:29 test systemd: Started Puppet agent.
      Jan 30 20:04:31 test puppet-agent[3016]: Starting Puppet client version 3.7.3
      Jan 30 20:04:34 test puppet-agent[3023]: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
      Jan 30 20:04:34 test puppet-agent[3023]: (at /usr/share/ruby/vendor_ruby/puppet/type/package.rb:430:in `block (3 levels) in <module:Puppet>')
      Jan 30 20:04:51 test systemd: Reloading.
      Jan 30 20:04:51 test yum[3442]: Updated: puppet-3.7.4-1.el7.noarch
      Jan 30 20:04:51 test systemd: Stopping Puppet agent...
      Jan 30 20:04:51 test puppet-agent[3023]: Caught TERM; calling stop
      Jan 30 20:04:51 test puppet-agent[3016]: Caught TERM; calling stop
      Jan 30 20:04:51 test systemd: Starting Puppet agent...
      Jan 30 20:04:51 test systemd: Started Puppet agent.
      Jan 30 20:04:53 test puppet-agent[3517]: Starting Puppet client version 3.7.4
      Jan 30 20:04:56 test puppet-agent[3527]: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
      Jan 30 20:04:56 test puppet-agent[3527]: (at /usr/share/ruby/vendor_ruby/puppet/type/package.rb:430:in `block (3 levels) in <module:Puppet>')
      Jan 30 20:05:12 test puppet-agent[3527]: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install puppet-3.7.4-1.el7' returned 1: There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]: Transaction check error:
      Jan 30 20:05:12 test puppet-agent[3527]: package puppet-3.7.4-1.el7.noarch is already installed
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]: Error Summary
      Jan 30 20:05:12 test puppet-agent[3527]: -------------
      Jan 30 20:05:12 test puppet-agent[3527]: Wrapped exception:
      Jan 30 20:05:12 test puppet-agent[3527]: Execution of '/usr/bin/yum -d 0 -e 0 -y install puppet-3.7.4-1.el7' returned 1: There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]: Transaction check error:
      Jan 30 20:05:12 test puppet-agent[3527]: package puppet-3.7.4-1.el7.noarch is already installed
      Jan 30 20:05:12 test puppet-agent[3527]:
      Jan 30 20:05:12 test puppet-agent[3527]: Error Summary
      Jan 30 20:05:12 test puppet-agent[3527]: -------------
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure) change from 3.7.3-1.el7 to 3.7.4-1.el7 failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install puppet-3.7.4-1.el7' returned 1: There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure)
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure)
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure) Transaction check error:
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure)   package puppet-3.7.4-1.el7.noarch is already installed
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure)
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure) Error Summary
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Install/Package[puppet]/ensure) -------------
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Service/Service[puppet]) Dependency Package[puppet] has failures: true
      Jan 30 20:05:12 test puppet-agent[3527]: (/Stage[main]/Puppet_client::Service/Service[puppet]) Skipping because of failed dependencies
      Jan 30 20:05:13 test puppet-agent[3527]: Finished catalog run in 11.66 seconds
       
      [root@test ~]# rpm -qa | grep puppet
      puppet-3.7.4-1.el7.noarch
      puppet-3.7.3-1.el7.noarch
      

      I think the behavior under the old init scripts was to TERM just PID 3016 above and not also 3023, which seems to be what's causing the problem.

        Attachments

          Activity

            People

            • Assignee:
              qa qa
              Reporter:
              shanemadden Shane Madden
              QA Contact:
              Eric Thompson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support