[PUP-9754] Log resident configuration upon daemon launch Created: 2019/06/14  Updated: 2019/10/09  Resolved: 2019/09/05

Status: Resolved
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: PUP 5.5.17, PUP 6.4.4, PUP 6.9.0

Type: Improvement Priority: Normal
Reporter: Garrett Guillotte Assignee: Ciprian Badescu
Resolution: Fixed Votes: 0
Labels: resolved-issue-added
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Acceptance Criteria:

Users can confirm Puppet agent's resident configuration state when running as a daemon.

Team: Night's Watch
Story Points: 2
Sprint: PR - 2019-07-23, NW - 2019-08-07, NW - 2019-08-21, NW - 2019-09-03, NW - 2019-09-18
CS Priority: Reviewed
Release Notes: Bug Fix
Release Notes Summary: This change is intended to log on debug level the configuration
used on startup when running in daemon mode (by default daemon mode can be enabled on POSIX systems and is disabled on Windows)

Log will sent to the output configured with --logdest

Configuration will be reloaded and also logged on SIGHUP
QA Risk Assessment: Needs Assessment

 Description   

Problem Statement

When attempting to diagnose issues with configuration drift within Puppet agent itself (such as changes to `puppet.conf` not picked up by the agent), there is no way to confirm that the resident configuration is the same as the current state of the configuration files.

For example, we can confirm configuration file settings with puppet config print and lookup PE keys with puppet lookup, but these only reflect what's currently written to the filesystem—if the daemon launched with different settings, there's no equivalent command to confirm its resident configuration.

Suggested Improvement

Either at info or debug logging levels, dump the configuration used by the daemon on launch and also when the daemon reloads the configuration.

Use cases

  • User Dougie wants to change a setting on disk (for instance, in puppet.conf or in PE Hiera keys) that doesn't have an obvious measurable impact, but he still wants to confirm that the change is picked up by the daemon after it is restarted. (For instance, Dougie might want to confirm that changing a setting in puppet.conf takes effect if the setting also exists in Hiera or the classifier.) If the daemon dumps its settings on launch with debug-level logging enabled, Dougie could confirm that the daemon's new settings match the settings on disk.
  • User Dougie launches the agent daemon, then user Bob changes the agent settings on disk. The agent daemon then runs uninterrupted for a length of time. Dougie is concerned that the setting on disk might have been changed, and if so wants to revert the change. However, he cannot remember the altered setting's original value. If the daemon dumped its settings on launch at info log level, Dougie could confirm that the settings on disk were changed and revert them to their original value.
  • User Dougie launches the agent daemon, then user Bob changes the agent settings on disk. The agent daemon then runs uninterrupted for a length of time. The agent eventually exhibits signs of a potential issue related to an agent setting, and Dougie calls Support. Using the settings that Bob set on disk, Support cannot reproduce the problem. It also does not appear that the daemon reloaded the settings from disk. Dougie does not remember how the settings Bob changed were configured when the running daemon was launched. If the daemon dumped its settings on launch at info log level, Support could confirm the settings in use by the daemon and attempt to reproduce the issue.


 Comments   
Comment by Josh Cooper [ 2019/06/19 ]

Logging every setting value could get spammy. We may want to limit this to only log settings whose values are different than the default and/or add a signal handler (maybe USR2?) to dump the configuration in response to the signal. Not sure about Windows.

This is another reason to run puppet from cron instead of running our homegrown daemonization code.

Comment by Ciprian Badescu [ 2019/07/26 ]

My understanding is that the config should be logged with debug only at puppet agent startup and not each catalog update in daemon mode. Garrett Guillotte, pls confirm.

On floaty centos-7-x86_64, this will add extra ~200 lines to already existing ~5000 in debug mode during puppet agent startup. Josh Cooper, does it sounds reasonable?

Comment by Garrett Guillotte [ 2019/08/09 ]

Ciprian Badescu Yes, log on startup is the intent.

Comment by Josh Cooper [ 2019/08/28 ]

Merged to 5.5.x in https://github.com/puppetlabs/puppet/commit/5144c75f8c1257fd1aa572d70714d7f4f1a97a7b

Comment by Jean Bond [ 2019/09/16 ]

Josh Cooper or Ciprian Badescu, I'm looking for a bit of clarification on the release note. It says:

(by default daemon mode can be enabled on POSIX systems and is disabled on Windows)

Does the mean daemon mode is enabled by default on POSIX, or it is disabled by default on POSIX and the user can enable it?
And for Windows, this says it is disabled by default. Can the user enable it?

Comment by Josh Cooper [ 2019/09/16 ]

Jean Bond The puppet agent process will daemonize by default on POSIX systems, though it can be prevented a few different ways, such as setting Puppet[:daemonize]=false, specifying puppet agent ---onetime, or more commonly puppet agent --test. The new behavior in this ticket only occurs when the agent daemonizes and the log level is set to debug.

Also Windows handles daemonizing differently, so this ticket doesn't apply to puppet on Windows.

Generated at Sat Aug 08 08:51:49 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.