Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-5282

Empty certname in PuppetDB command data results in NullPointerException during startup

    XMLWordPrintable

Details

    • Hide
      • When using the "new" command syntax, the HTTP endpoint should reject a command that has the empty string for a certname ie. &certname=& in the URI
      • When using the "old" command syntax, if we load the command into memory to read the certname in order to store it in stockpile we should reject it then. Otherwise the command processor should reject it and put it in the DLO without any retries.
      Show
      When using the "new" command syntax, the HTTP endpoint should reject a command that has the empty string for a certname ie. &certname=& in the URI When using the "old" command syntax, if we load the command into memory to read the certname in order to store it in stockpile we should reject it then. Otherwise the command processor should reject it and put it in the DLO without any retries.
    • HA
    • 3
    • HA 2021-10-20, HA 2021-11-03, HA 2021-11-17, HA 2021-12-01, HA 2021-12-15, HA 2022-01-05
    • Needs Assessment
    • 45879
    • 1
    • Bug Fix
    • Reject commands with empty certnames.
    • Needs Assessment

    Description

      PuppetDB "commands" are operations to update the data associated with puppet nodes. These commands are identified by certname, and odd things can happen if an empty string or other null value is provided as the certname. One consequence is a java.lang.NullPointerException during startup.

      When PE DR is enabled, this exception prevents the PuppetDB service from exiting maintenance mode and responding to read and write requests.

      Reproduction Case

      • Install PE 2019.8.8 on a CentOS 7 node and add a DR replica
      • Deactivate an empty certname: puppet node deactivate ''
      • Re-start the pe-puppetdb service

      Outcome

      The deactivate command results in the following error message being recorded to the PuppetDB log:

      2021-09-21T23:21:07.566Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-2 to stderr and log
      java.lang.NullPointerException: null
      	at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
      	at metrics.meters$mark_BANG_.invoke(meters.clj:72)
      	at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
      	at metrics.meters$mark_BANG_.invoke(meters.clj:72)
      	at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
      	at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
      	at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
      	at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
      	at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
      	at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
      	at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
      	at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
      	at clojure.lang.AFn.run(AFn.java:22)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      

      This also leaves behind a command int the PuppetDB processing queue with an empty certname:

      # find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -print -exec cat {} \;
       
      /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q/20-1632266467556-103_rm-node_3_.json
      {"certname":"","producer_timestamp":"2021-09-21T23:21:07.453+00:00"}
      

      Upon restart, the PuppetDB services fails to initialize the command processing pool and never exits maintenance mode:

      2021-09-21T23:22:20.499Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-1 to stderr and log
      java.lang.NullPointerException: null
      	at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
      	at metrics.meters$mark_BANG_.invoke(meters.clj:72)
      	at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
      	at metrics.meters$mark_BANG_.invoke(meters.clj:72)
      	at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
      	at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
      	at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
      	at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
      	at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
      	at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
      	at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
      	at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
      	at clojure.lang.AFn.run(AFn.java:22)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      2021-09-21T23:22:20.715Z INFO  [p.p.c.services] Finished sweep of stale nodes (threshold: 7 days)
      2021-09-21T23:22:20.717Z INFO  [p.p.c.services] Starting purge deactivated and expired nodes (threshold: 14 days)
      2021-09-21T23:22:20.760Z INFO  [p.p.c.services] Finished purge deactivated and expired nodes (threshold: 14 days)
      2021-09-21T23:22:20.761Z INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
      2021-09-21T23:22:20.849Z INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
      2021-09-21T23:22:20.849Z INFO  [p.p.c.services] Starting gc packages
      2021-09-21T23:22:20.852Z INFO  [p.p.c.services] Finished gc packages
      2021-09-21T23:22:20.853Z INFO  [p.p.c.services] Starting database garbage collection
      2021-09-21T23:22:20.909Z INFO  [p.p.s.storage] Starting sweep of unused fact paths
      2021-09-21T23:22:20.932Z INFO  [p.p.s.storage] Finished sweep of unused fact paths
      2021-09-21T23:22:20.940Z INFO  [p.p.c.services] Finished database garbage collection
      

      There should be a "PuppetDB finished starting, disabling maintenance mode" message in the above.

      Expected Outcome

      An empty certname in a command submission does not block PuppetDB startup

      Recommended Workaround

      Clear the PuppetDB command queue and re-start the service:

      find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -delete
      systemctl restart pe-puppetdb
      

      Attachments

        Activity

          People

            stel.abrego Stel Abrego
            chuck Charlie Sharpsteen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Zendesk Support