-
Type:
Improvement
-
Status: Accepted
-
Priority:
Normal
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Template:customfield_10700 289456
-
Team:Froyo
-
CS Priority:Major
-
QA Risk Assessment:Needs Assessment
Puppet server 5.1.0 added a max-queued-requests setting that will terminate agent requests early when the number of requests waiting for a JRuby instance exceeded a limit imposed by the setting (SERVER-1767). The requests are failed with a HTTP 503 error that includes a Retry-After header set to a randomized number of seconds that the agent should sleep before retrying the request. This randomized sleep allows the server to "re-splay" thundering herds by spacing the agents out without requiring the intervention of human operators.
Currently, max-queued-requests is off by default. But, we should consider changing the default to enabled as the setting not only re-splays agent runs but ensures that some Jetty workers are free to handle requests that do not require a Ruby interpreter — such as performance metric snapsots.
The "ideal" default for this limit is likely some multiple of max-active-jrubies with a hard cap that ensures 25% – 33% of the Jetty worker pool is available to serve non-Ruby requests. We should also consider lowering the default for max-retry-delay from 30 minutes to 5 or 10 minutes so that agent runs are not overly delayed.
- is blocked by
-
PCP-855 PE Tasks not handling puppetserver max_queued_requests/max_retry_delay properly
-
- Open
-
-
SERVER-2405 Skip enforcement of max-queued-requests if agent version is too old
-
- Resolved
-