Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-3905

Disable ruby 2.1.5 automatically retrying idempotent http methods

    XMLWordPrintable

Details

    • Coremunity
    • 2
    • Platform Core KANBAN
    • Bug Fix
    • Disables ruby's automatic HTTP retry mechanism which could incorrectly cause the same report to be submitted multiple times, increasing the load on the puppetserver report processor.

    Description

      While reviewing https://github.com/puppetlabs/puppet/pull/3107 I discovered that ruby 2.1.5 will automatically retry idempotent http methods, which includes GET, HEAD, PUT, DELETE if any of the following exceptions are raised:

            rescue Net::ReadTimeout, IOError, EOFError,
                   Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE,
                   # avoid a dependency on OpenSSL
                   defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError,
                   Timeout::Error => exception
      

      According to the rfc7231:

      A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.

      The trouble is that we do not use PUT in an idempotent way. For example, each time we PUT a report, we store a *new* copy on disk and in puppetdb. In order to be idempotent, we would need to update the *one* report resource.

      Normally that would not be a big deal. But when combined with ruby 2.1.5, it is possible that the agent can PUT a report, and the agent socket may timeout before the report processor finishes, or the load balancer may close the idle connection, e.g. PE-7326. In those cases ruby will catch the exception and retry the request, thankfully only once.

      It does not appear ruby has a mechanism to disable this behavior...

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              josh Josh Cooper
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support