Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2405

Skip enforcement of max-queued-requests if agent version is too old

    XMLWordPrintable

Details

    • Froyo
    • Bug Fix
    • Puppet Server will no longer issue 503 responses to older agents that can't handle it, which allows the {{max-queued-requests}} setting to safely be used with older agents (though it still won't cause them to retry).
    • Needs Assessment

    Description

      Puppet server 5.1.0 added a max-queued-requests setting that will terminate agent requests early when the number of requests waiting for a JRuby instance exceeded a limit imposed by the setting (SERVER-1767). The requests are failed with a HTTP 503 error that includes a Retry-After header set to a randomized number of seconds that the agent should sleep before retrying the request. This randomized sleep allows the server to "re-splay" thundering herds by spacing the agents out without requiring the intervention of human operators.

      The issue is that agents older than Puppet 5.3 don't have logic for reacting to the presence of a Retry-After header. As a result these older agents will "fail fast" and reach the end of their runs very quickly. When an agent reaches the end of a run, it schedules the next run — which means older agents will end up scheduling at the same time and thus create a thundering herd.

      Due to this limitation, documentation on the max-queued-requests setting advises against using it unless the agent population is fully on 5.3 or newer (DOC-3845). We can improve this situation by updating the logic that enforces max-queued-requests to not impose the request limit if the X-Puppet-Version header indicates an older agent or is missing. This change would allow max-queued-requests to be used safely in populations that include older agents.

      Attachments

        Issue Links

          Activity

            People

              chuck Charlie Sharpsteen
              chuck Charlie Sharpsteen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support