Details
-
Task
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
None
-
Coremunity
-
2
-
Platform Core KANBAN
-
Bug Fix
-
Disables ruby's automatic HTTP retry mechanism which could incorrectly cause the same report to be submitted multiple times, increasing the load on the puppetserver report processor.
Description
While reviewing https://github.com/puppetlabs/puppet/pull/3107 I discovered that ruby 2.1.5 will automatically retry idempotent http methods, which includes GET, HEAD, PUT, DELETE if any of the following exceptions are raised:
rescue Net::ReadTimeout, IOError, EOFError, |
Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE, |
# avoid a dependency on OpenSSL |
defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError, |
Timeout::Error => exception
|
According to the rfc7231:
A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.
The trouble is that we do not use PUT in an idempotent way. For example, each time we PUT a report, we store a *new* copy on disk and in puppetdb. In order to be idempotent, we would need to update the *one* report resource.
Normally that would not be a big deal. But when combined with ruby 2.1.5, it is possible that the agent can PUT a report, and the agent socket may timeout before the report processor finishes, or the load balancer may close the idle connection, e.g. PE-7326. In those cases ruby will catch the exception and retry the request, thankfully only once.
It does not appear ruby has a mechanism to disable this behavior...
Attachments
Issue Links
- blocks
-
PUP-3319 client should retry if connection to master fails
-
- Closed
-