[PDB-3830] Command Queue Grows Steadily Over Time not related to load Created: 2018/02/05  Updated: 2018/08/24  Resolved: 2018/03/08

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: None
Fix Version/s: PDB 5.1.5, PDB 5.2.1

Type: Bug Priority: Normal
Reporter: Nick Walker Assignee: Rob Browning
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File queue_depth_discard_count.png    
Issue Links:
is duplicated by PDB-3854 stockpile/discard used to calculate c... Closed
relates to PDB-3854 stockpile/discard used to calculate c... Closed
Team: Data Platform
Story Points: 1
Sprint: Data Platform 2018-03-14
Method Found: Customer Feedback
Release Notes: Bug Fix
Release Notes Summary: PuppetDB now correctly reduces the command queue size when a malformed message is submitted to the queue. Previously if you submitted a message that PuppetDB discards, the command queue would increase when the message came in and would not reduce when the message was discarded.
QA Risk Assessment: Needs Assessment


 The Problem

Over time we notice that the queue depth grows just a little bit and never drops back down.  Over the course of a few days it'll be sitting around 300-400 despite capacity to work it down.  

After graphing some other metrics we see that discard count seems suspiciously close to the command queue which grows over time.

I suspect that discards are not correctly removed from the counter for queue depth.


Comment by Jeff Yarnell [ 2018/02/14 ]

Nick Walker Have you tested your patch for this?

Comment by Nick Walker [ 2018/02/15 ]

Jeff Yarnell yessir, I put my notes in the PR. The only question I had left was whether I should aim the patch at master or stable.

Comment by Jeff Yarnell [ 2018/02/15 ]

Russell Mull Please see Nick's question above.

Comment by Jeff Yarnell [ 2018/02/21 ]

Nick Walker In what PDB version did you see this?

Comment by Nick Walker [ 2018/02/21 ]

I think it should go into the next z release of PE 2017.3 so I think that would mean PDB 5.1.5?

Generated at Mon Jan 20 06:01:16 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.