Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10351

Puppet Agent fails to self-restart on config change or update

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: PUP 6.10.1, PUP 6.11.0, PUP 6.12.0, PUP 6.13.0
    • Fix Version/s: PUP 6.16.0, PUP 5.5.21
    • Component/s: None
    • Labels:
    • Environment:

      Verified on CentOS 7.7.1908 with Puppet Agent versions 6.13.0, 6.12.0, 6.11.0 and 6.10.1.

    • Template:
      PUP Bug Template
    • Epic Link:
    • Team:
      Night's Watch
    • Story Points:
      2
    • Sprint:
      NW - 2020-04-01, NW - 2020-04-15, NW - 2020-04-29, NW - 2020-05-13, NW - 2020-05-27
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      When two agent runs were taking place at about the same time, under certain circumstances, puppet agent could get in a repeatedly `restart_requested` state. It was fixed by cleaning up the puppet daemon code logic.
    • QA Risk Assessment:
      Needs Assessment

      Description

      When the configuration of Puppet Agent changes, or when Puppet Agent is upgraded to a newer version, it gets completely stuck. The Puppet Agent essentially stops all operations. The following messages get logged:

      $ systemctl status puppet
      ● puppet.service - Puppet agent
         Loaded: loaded (/usr/lib/systemd/system/puppet.service; enabled; vendor preset: disabled)
         Active: active (running) since Tue 2020-03-03 18:46:45 CET; 19h ago
       Main PID: 11433 (puppet)
         CGroup: /system.slice/puppet.service
                 └─11433 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
       
      Mar 04 09:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 10:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 10:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 11:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 11:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 12:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 12:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 13:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 13:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      Mar 04 14:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      

      The system log contains the following additional entries:

      Mar  3 18:50:35 host.example.com systemd: Reloading Puppet agent.
      Mar  3 18:50:35 host.example.com systemd: Reloaded Puppet agent.
      Mar  3 18:50:35 host.example.com puppet-agent[11433]: Caught HUP; storing restart
      Mar  3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Daemon/Service[puppet]) Triggered 'refresh' from 1 event
      Mar  3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Systemd/Service[puppet-run.timer]) Triggered 'refresh' from 1 event
      Mar  3 18:50:35 host.example.com systemd: Reloading.
      Mar  3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Systemd/Exec[systemctl-daemon-reload-puppet]) Triggered 'refresh' from 1 event
      Mar  3 18:50:36 host.example.com puppet-agent[4047]: Applied catalog in 256.53 seconds
      Mar  3 18:50:38 host.example.com puppet-agent[11433]: Processing restart
      Mar  3 18:50:38 host.example.com puppet-agent[11433]: Puppet::Agent::Locker.running? is deprecated as it is inherently unsafe. The only safe way to know if the lock is locked is to try lock and perform some action and then handle the LockError that may result.
      Mar  3 18:50:38 host.example.com puppet-agent[11433]:   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:37:in `running?')
      Mar  3 18:50:47 host.example.com puppet-agent[11433]: Config file /etc/puppetlabs/puppet/puppet.conf changed; triggering re-parse of all config files.
      Mar  3 19:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
      

      This was also reported on the Puppet Users list. Several people confirmed this issue over there. In that thread Josh Cooper asked to create a JIRA ticket. I couldn't find an existing ticket.

      This is a major issue, because it essentially breaks automation with Puppet. The Puppet Agent has to be manually restarted.

        Attachments

          Activity

            People

            Assignee:
            luchian.nemes Luchian Nemes
            Reporter:
            fraenki Frank Wall
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Zendesk Support