Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
PDB 6.17.0
-
-
HA
-
3
-
HA 2021-10-20, HA 2021-11-03, HA 2021-11-17, HA 2021-12-01, HA 2021-12-15, HA 2022-01-05
-
Needs Assessment
-
45879
-
1
-
Bug Fix
-
Reject commands with empty certnames.
-
Needs Assessment
Description
PuppetDB "commands" are operations to update the data associated with puppet nodes. These commands are identified by certname, and odd things can happen if an empty string or other null value is provided as the certname. One consequence is a java.lang.NullPointerException during startup.
When PE DR is enabled, this exception prevents the PuppetDB service from exiting maintenance mode and responding to read and write requests.
Reproduction Case
- Install PE 2019.8.8 on a CentOS 7 node and add a DR replica
- Deactivate an empty certname: puppet node deactivate ''
- Re-start the pe-puppetdb service
Outcome
The deactivate command results in the following error message being recorded to the PuppetDB log:
2021-09-21T23:21:07.566Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-2 to stderr and log
|
java.lang.NullPointerException: null
|
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
|
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
|
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
|
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
|
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
|
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
|
at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
|
at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
|
at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
|
at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
|
at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
|
at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
|
at clojure.lang.AFn.run(AFn.java:22)
|
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
|
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
|
at java.base/java.lang.Thread.run(Thread.java:829)
|
This also leaves behind a command int the PuppetDB processing queue with an empty certname:
# find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -print -exec cat {} \;
|
|
|
/opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q/20-1632266467556-103_rm-node_3_.json
|
{"certname":"","producer_timestamp":"2021-09-21T23:21:07.453+00:00"}
|
Upon restart, the PuppetDB services fails to initialize the command processing pool and never exits maintenance mode:
2021-09-21T23:22:20.499Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-1 to stderr and log
|
java.lang.NullPointerException: null
|
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
|
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
|
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
|
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
|
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
|
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
|
at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
|
at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
|
at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
|
at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
|
at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
|
at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
|
at clojure.lang.AFn.run(AFn.java:22)
|
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
|
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
|
at java.base/java.lang.Thread.run(Thread.java:829)
|
2021-09-21T23:22:20.715Z INFO [p.p.c.services] Finished sweep of stale nodes (threshold: 7 days)
|
2021-09-21T23:22:20.717Z INFO [p.p.c.services] Starting purge deactivated and expired nodes (threshold: 14 days)
|
2021-09-21T23:22:20.760Z INFO [p.p.c.services] Finished purge deactivated and expired nodes (threshold: 14 days)
|
2021-09-21T23:22:20.761Z INFO [p.p.c.services] Starting sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
|
2021-09-21T23:22:20.849Z INFO [p.p.c.services] Finished sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
|
2021-09-21T23:22:20.849Z INFO [p.p.c.services] Starting gc packages
|
2021-09-21T23:22:20.852Z INFO [p.p.c.services] Finished gc packages
|
2021-09-21T23:22:20.853Z INFO [p.p.c.services] Starting database garbage collection
|
2021-09-21T23:22:20.909Z INFO [p.p.s.storage] Starting sweep of unused fact paths
|
2021-09-21T23:22:20.932Z INFO [p.p.s.storage] Finished sweep of unused fact paths
|
2021-09-21T23:22:20.940Z INFO [p.p.c.services] Finished database garbage collection
|
There should be a "PuppetDB finished starting, disabling maintenance mode" message in the above.
Expected Outcome
An empty certname in a command submission does not block PuppetDB startup
Recommended Workaround
Clear the PuppetDB command queue and re-start the service:
find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -delete |
systemctl restart pe-puppetdb
|
