Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-391

Improve error message for JRuby pool borrow timeout

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Normal
    • Resolution: Fixed
    • None
    • SERVER 1.1.0, SERVER 2.1.0
    • None
    • None
    • 1
    • Server Emerald 2015-03-04, Server Emerald 2015-04-15, Server Emerald 2015-04-29, Server Emerald 2015-05-13

    Description

      SERVER-246 introduced a timeout for allowing JRuby pool borrow attempts. The work for it, however, didn't directly cover what the external user experience should be if one of these timeouts were hit in the context of an incoming agent request. Presumably, the exception that the borrow attempt would throw would be translated into an HTTP failure response on the wire. The internal exception message has "Error: Attempt to borrow a JRuby instance from the pool timed out". This message should probably be more descriptive, e.g., including some informative text and what the user might be able to do to address the issue.

      This ticket would cover:

      1) Evaluate what the current behavior is for a failure.

      2) Investigate what would be involved in catching the internal exception and rethrowing as a more user-friendly failure in the HTTP response.

      3) Solicit opinions - maybe from UX - on what a more appropriate message for a consumer of the HTTP response would be.


      For reference, the following is what appears on the agent side when the JRuby pool borrow times out:

      Warning: Unable to fetch my node definition, but the agent run will continue:
      Warning: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Error: Attempt to borrow a JRuby instance from the pool timed out

      The full stacktrace that follows this exception message is printed out in the Server log.

      The agent run continues after receiving this warning/error, likely using some cached state from the previous run.

      To reproduce...

      First, the setup:

      • Install a custom "sleep" function on the Server:

        sleep.rb

        module Puppet::Parser::Functions
          newfunction(:sleep) do |args|
            sleep args.first
          end
        end
        

      • Sleep 15 seconds in a manifest:

        site.pp

        sleep(15)
        

      • Use only 1 JRuby instance, and lower the Puppet Server borrow timeout to 10 seconds:

        puppet-server.conf

        jruby-puppet: {
            max-active-instances: 1
            borrow-timeout: 10000
        }
        

      Now, trigger the timeout:

      1. Borrow the only JRuby directly with curl (since you can't do two simultaneous agent runs):

      $ curl -k -H "Accept: pson" https://localhost:8140/puppet/v3/catalog/my-agent\?environment\=production
      

      2. Immediately after the curl command, do an agent run:

      $ puppet agent --no-daemonize --onetime --verbose --server localhost --certname my-agent
      Warning: Unable to fetch my node definition, but the agent run will continue:
      Warning: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Error: Attempt to borrow a JRuby instance from the pool timed out
      Info: Retrieving plugin
      Info: Caching catalog for my-agent
      Info: Applying configuration version '1424979993'
      Notice: I'm still tired :(
      Notice: /Stage[main]/Foo/Notify[I'm still tired :(]/message: defined 'message' as 'I'm still tired :('
      Notice: Applied catalog in 0.02 seconds
      

      QA:

      Risk Assessment: Low
      Probability: Medium
      Severity: Low. Only affects the error message we throw when we time out a JRuby request.

      Attachments

        Issue Links

          Activity

            People

              qa qa
              jeremy.barlow Jeremy Barlow
              Erik Dasher Erik Dasher
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support