Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
PUP 6.10.1, PUP 6.11.0, PUP 6.12.0, PUP 6.13.0
-
None
-
Verified on CentOS 7.7.1908 with Puppet Agent versions 6.13.0, 6.12.0, 6.11.0 and 6.10.1.
-
Night's Watch
-
2
-
NW - 2020-04-01, NW - 2020-04-15, NW - 2020-04-29, NW - 2020-05-13, NW - 2020-05-27
-
Needs Assessment
-
Bug Fix
-
When two agent runs were taking place at about the same time, under certain circumstances, puppet agent could get in a repeatedly `restart_requested` state. It was fixed by cleaning up the puppet daemon code logic.
-
Needs Assessment
Description
When the configuration of Puppet Agent changes, or when Puppet Agent is upgraded to a newer version, it gets completely stuck. The Puppet Agent essentially stops all operations. The following messages get logged:
$ systemctl status puppet
|
● puppet.service - Puppet agent
|
Loaded: loaded (/usr/lib/systemd/system/puppet.service; enabled; vendor preset: disabled)
|
Active: active (running) since Tue 2020-03-03 18:46:45 CET; 19h ago
|
Main PID: 11433 (puppet)
|
CGroup: /system.slice/puppet.service
|
└─11433 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
|
|
|
Mar 04 09:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 10:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 10:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 11:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 11:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 12:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 12:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 13:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 13:46:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
Mar 04 14:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
The system log contains the following additional entries:
Mar 3 18:50:35 host.example.com systemd: Reloading Puppet agent.
|
Mar 3 18:50:35 host.example.com systemd: Reloaded Puppet agent.
|
Mar 3 18:50:35 host.example.com puppet-agent[11433]: Caught HUP; storing restart
|
Mar 3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Daemon/Service[puppet]) Triggered 'refresh' from 1 event
|
Mar 3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Systemd/Service[puppet-run.timer]) Triggered 'refresh' from 1 event
|
Mar 3 18:50:35 host.example.com systemd: Reloading.
|
Mar 3 18:50:35 host.example.com puppet-agent[4047]: (/Stage[main]/Puppet::Agent::Service::Systemd/Exec[systemctl-daemon-reload-puppet]) Triggered 'refresh' from 1 event
|
Mar 3 18:50:36 host.example.com puppet-agent[4047]: Applied catalog in 256.53 seconds
|
Mar 3 18:50:38 host.example.com puppet-agent[11433]: Processing restart
|
Mar 3 18:50:38 host.example.com puppet-agent[11433]: Puppet::Agent::Locker.running? is deprecated as it is inherently unsafe. The only safe way to know if the lock is locked is to try lock and perform some action and then handle the LockError that may result.
|
Mar 3 18:50:38 host.example.com puppet-agent[11433]: (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:37:in `running?')
|
Mar 3 18:50:47 host.example.com puppet-agent[11433]: Config file /etc/puppetlabs/puppet/puppet.conf changed; triggering re-parse of all config files.
|
Mar 3 19:16:47 host.example.com puppet-agent[11433]: Shutdown/restart in progress (:restart_requested); skipping run
|
This was also reported on the Puppet Users list. Several people confirmed this issue over there. In that thread josh asked to create a JIRA ticket. I couldn't find an existing ticket.
This is a major issue, because it essentially breaks automation with Puppet. The Puppet Agent has to be manually restarted.
