Details
-
Type:
Task
-
Status: Resolved
-
Priority:
Normal
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: PUP 6.0.10, PUP 6.4.3, PUP 6.5.0
-
Component/s: None
-
Labels:
-
Template:customfield_10700 65130
-
Epic Link:
-
Team:Coremunity
-
Story Points:2
-
Sprint:Platform Core KANBAN
-
Release Notes:Bug Fix
-
Release Notes Summary:Disables ruby's automatic HTTP retry mechanism which could incorrectly cause the same report to be submitted multiple times, increasing the load on the puppetserver report processor.
Description
While reviewing https://github.com/puppetlabs/puppet/pull/3107 I discovered that ruby 2.1.5 will automatically retry idempotent http methods, which includes GET, HEAD, PUT, DELETE if any of the following exceptions are raised:
rescue Net::ReadTimeout, IOError, EOFError, |
Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE, |
# avoid a dependency on OpenSSL |
defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError, |
Timeout::Error => exception
|
According to the rfc7231:
A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.
The trouble is that we do not use PUT in an idempotent way. For example, each time we PUT a report, we store a *new* copy on disk and in puppetdb. In order to be idempotent, we would need to update the *one* report resource.
Normally that would not be a big deal. But when combined with ruby 2.1.5, it is possible that the agent can PUT a report, and the agent socket may timeout before the report processor finishes, or the load balancer may close the idle connection, e.g. PE-7326. In those cases ruby will catch the exception and retry the request, thankfully only once.
It does not appear ruby has a mechanism to disable this behavior...
Attachments
Issue Links
- blocks
-
PUP-3319 client should retry if connection to master fails
-
- Closed
-