we have noticed a high, fixed command queue depth on our PuppetDB instance.
The metrics v1 API is used to monitor puppetlabs.puppetdb.mq:name=global.depth and it became stuck at a fixed point. The load was fairly low.
After debugging, we found 20 discarded messages in /opt/puppetlabs/server/data/puppetdb/stockpile/discard/:
After stopping PuppetDB and deleting the discarded files, the queue depth was back to normal values.
The Puppet host responsible for this errors had garbage characters in a file.
The documentation is not 100% clear, if depth should contain only queued commands or queued + discarded commands.