Details
-
Task
-
Status: Closed
-
Normal
-
Resolution: Fixed
-
None
-
None
-
None
-
1
-
Server Emerald 2015-03-04, Server Emerald 2015-04-15, Server Emerald 2015-04-29, Server Emerald 2015-05-13
Description
SERVER-246 introduced a timeout for allowing JRuby pool borrow attempts. The work for it, however, didn't directly cover what the external user experience should be if one of these timeouts were hit in the context of an incoming agent request. Presumably, the exception that the borrow attempt would throw would be translated into an HTTP failure response on the wire. The internal exception message has "Error: Attempt to borrow a JRuby instance from the pool timed out". This message should probably be more descriptive, e.g., including some informative text and what the user might be able to do to address the issue.
This ticket would cover:
1) Evaluate what the current behavior is for a failure.
2) Investigate what would be involved in catching the internal exception and rethrowing as a more user-friendly failure in the HTTP response.
3) Solicit opinions - maybe from UX - on what a more appropriate message for a consumer of the HTTP response would be.
For reference, the following is what appears on the agent side when the JRuby pool borrow times out:
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Error: Attempt to borrow a JRuby instance from the pool timed out
The full stacktrace that follows this exception message is printed out in the Server log.
The agent run continues after receiving this warning/error, likely using some cached state from the previous run.
To reproduce...
First, the setup:
- Install a custom "sleep" function on the Server:
sleep.rb
module Puppet::Parser::Functions
newfunction(:sleep) do |args|
sleep args.first
end
end
- Sleep 15 seconds in a manifest:
site.pp
sleep(15)
- Use only 1 JRuby instance, and lower the Puppet Server borrow timeout to 10 seconds:
puppet-server.conf
jruby-puppet: {
max-active-instances: 1
borrow-timeout: 10000
}
Now, trigger the timeout:
1. Borrow the only JRuby directly with curl (since you can't do two simultaneous agent runs):
$ curl -k -H "Accept: pson" https://localhost:8140/puppet/v3/catalog/my-agent\?environment\=production |
2. Immediately after the curl command, do an agent run:
$ puppet agent --no-daemonize --onetime --verbose --server localhost --certname my-agent
|
Warning: Unable to fetch my node definition, but the agent run will continue: |
Warning: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Error: Attempt to borrow a JRuby instance from the pool timed out
|
Info: Retrieving plugin
|
Info: Caching catalog for my-agent |
Info: Applying configuration version '1424979993' |
Notice: I'm still tired :(
|
Notice: /Stage[main]/Foo/Notify[I'm still tired :(]/message: defined 'message' as 'I'm still tired :(' |
Notice: Applied catalog in 0.02 seconds |
QA:
Risk Assessment: Low
Probability: Medium
Severity: Low. Only affects the error message we throw when we time out a JRuby request.
Attachments
Issue Links
- relates to
-
SERVER-246 change macro 'borrow's to 'borrow-with-timeout' in jruby service
-
- Closed
-
-
SERVER-389 Bump default-borrow-timeout to something significantly higher
-
- Closed
-
- links to