Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-3854

stockpile/discard used to calculate command queue depth

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: PDB 4.4.0
    • Fix Version/s: None
    • Component/s: PuppetDB
    • Labels:
      None
    • Template:
      Vim Tutorial
    • Team:
      Data Platform
    • Method Found:
      Needs Assessment
    • QA Risk Assessment:
      Needs Assessment

      Description

      Hello,

      we have noticed a high, fixed command queue depth on our PuppetDB instance.

      The metrics v1 API is used to monitor puppetlabs.puppetdb.mq:name=global.depth and it became stuck at a fixed point. The load was fairly low.

      After debugging, we found 20 discarded messages in /opt/puppetlabs/server/data/puppetdb/stockpile/discard/:

      # ll /opt/puppetlabs/server/data/puppetdb/stockpile/discard/
      total 724
      -rw------- 1 puppetdb puppetdb 43457 20. Feb 20:09 6684289-1519135763922_report_8_<NODE>.desy.de_err.txt
      -rw-r----- 1 puppetdb puppetdb  4465 20. Feb 15:09 6684289-1519135763922_report_8_<NODE>.desy.de.json
      -rw------- 1 puppetdb puppetdb 43457 20. Feb 21:02 6691938-1519138937721_report_8_<NODE>.desy.de_err.txt
      -rw-r----- 1 puppetdb puppetdb  3450 20. Feb 16:02 6691938-1519138937721_report_8_<NODE>.desy.de.json
      -rw------- 1 puppetdb puppetdb 43457 21. Feb 00:02 6726883-1519149736935_report_8_<NODE>.desy.de_err.txt
      -rw-r----- 1 puppetdb puppetdb  3440 20. Feb 19:02 6726883-1519149736935_report_8_<NODE>.desy.de.json
      

      After stopping PuppetDB and deleting the discarded files, the queue depth was back to normal values.
      The Puppet host responsible for this errors had garbage characters in a file.

      The documentation is not 100% clear, if depth should contain only queued commands or queued + discarded commands.

      Regards,
      Stefan

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  stdietrich Stefan Dietrich
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Zendesk Support