Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-3905

Disable ruby 2.1.5 automatically retrying idempotent http methods

    Details

    • Template:
    • Epic Link:
    • Team:
      Coremunity
    • Story Points:
      2
    • Sprint:
      Platform Core KANBAN
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Disables ruby's automatic HTTP retry mechanism which could incorrectly cause the same report to be submitted multiple times, increasing the load on the puppetserver report processor.

      Description

      While reviewing https://github.com/puppetlabs/puppet/pull/3107 I discovered that ruby 2.1.5 will automatically retry idempotent http methods, which includes GET, HEAD, PUT, DELETE if any of the following exceptions are raised:

            rescue Net::ReadTimeout, IOError, EOFError,
                   Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE,
                   # avoid a dependency on OpenSSL
                   defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError,
                   Timeout::Error => exception
      

      According to the rfc7231:

      A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.

      The trouble is that we do not use PUT in an idempotent way. For example, each time we PUT a report, we store a *new* copy on disk and in puppetdb. In order to be idempotent, we would need to update the *one* report resource.

      Normally that would not be a big deal. But when combined with ruby 2.1.5, it is possible that the agent can PUT a report, and the agent socket may timeout before the report processor finishes, or the load balancer may close the idle connection, e.g. PE-7326. In those cases ruby will catch the exception and retry the request, thankfully only once.

      It does not appear ruby has a mechanism to disable this behavior...

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                josh Josh Cooper
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support