[SERVER-1052] Consider allowing JRuby compile.mode setting to be end-user configurable Created: 2015/12/03  Updated: 2016/09/27  Resolved: 2016/01/20

Status: Closed
Project: Puppet Server
Component/s: DOCS
Affects Version/s: None
Fix Version/s: SERVER 2.3.0

Type: Task Priority: Normal
Reporter: Jeremy Barlow Assignee: Erik Dasher
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relates to SERVER-993 Catalog compilation time Closed
relates to SERVER-1064 HACK; Experiment with configuring Pup... Resolved
Sub-team: emerald
Story Points: 2
Sprint: Server Emerald 2016-01-13, Server Emerald 2016-01-27
QA Contact: Erik Dasher


Through some work done for SERVER-993, we discovered that there could be a considerable performance benefit in at least some scenarios to having Puppet Server use the "compile.mode" setting of JIT instead of the Puppet Server default of OFF when initializing JRuby ScriptingContainers.

To allow for others to experiment with this for performance, it might be good to expose this as a configurable setting, e.g., via the jruby-puppet section in Trapperkeeper configuration, rather than having it just be hard-coded / unchangeable in Puppet Server code - see https://github.com/puppetlabs/puppet-server/blob/puppet-server-2.2.0/src/clj/puppetlabs/services/jruby/jruby_puppet_internal.clj#L35.

Probably best to stick with OFF as the default value for now until such time as we might have done sufficient characterization to recommend the use of JIT. Note that JIT is the default value used when running JRuby from the command line.

For more about the compile.mode setting, see https://github.com/jruby/jruby/wiki/JRubyCompiler#tweaking-and-troubleshooting.

Comment by Chris Price [ 2015/12/04 ]

Thomas Hallgren in the early prototypes of Puppet Server (2+ years ago), using the default optimization settings for JRuby was causing all kinds of really, really scary JIT-related PermGen errors that I had never even seen in the JVM before and did not know were possible. The goals for the initial release of Puppet Server focused on stability and feature parity with the Rack master, with the intention of coming back around and optimizing and improving performance later. So, we took the approach of choosing the absolute most conservative values for every option. Even with those conservative settings, perf testing showed us at parity or better than Rack in almost all cases. The recent performance ticket related to the Racc parsing is the first significant performance-related ticket we've ever gotten that has actually boiled down to a JRuby vs. MRI issue.

We've always intended to come back around and explore these settings, but haven't had the time to do so because we have so much feature-related work that tends to get prioritized above this. We still intend to, but, it's not entirely clear when. Direct Puppet and the Node Classifier-related memory issues for installations with large numbers of environments are a few things that we definitely have to get sorted before we'll be able to really give JRuby optimization the attention that it deserves.

We've had enough issues with JRuby (mostly occurring when we bump to a new version) in the past that we consider anything like this to be a high-risk endeavor that warrants a great deal of long-running test coverage before we roll out changes to the default production code / configuration, so we'll need to budget a significant chunk of time before we would consider rolling out a switch of the default settings to JIT.

Also, because we use JRuby via multiple ScriptingContainer objects controlled programatically, we are exercising a slightly different code path than what the CLI JRuby commands use, which means that even though flags like this might be well tested and stable for typical JRuby use cases, we still feel like we need to give them special testing and attention before rolling them out as defaults for Puppet Server.

So, I guess the tl;dr is just that we are erring on the side of being conservative and favoring stability over performance, and optimizing carefully as the opportunity presents itself.

For this ticket in particular, though, if we just expose the setting via configuration without changing the default value, then we're not really taking on any additional risk, but we can have users try it out as an experimental thing. So this one we could actually do sometime soon without incurring the cost of all of the expensive long-term stability testing.

Comment by Jeremy Barlow [ 2016/01/07 ]

Merged to puppet-server#master at af706e0.

Comment by Chris Price [ 2016/01/20 ]

Garrett Guillotte I couldn't find the dang "docs" tab in Jira to flag that this ticket is probably worthy of discussion in terms of whether we want to add any docs around it or not.

Comment by Garrett Guillotte [ 2016/01/20 ]

Chris Price Jeremy Barlow For the release notes fields, edit the ticket and you should see the docs tab between CS and Scope Change. The ticket's already got the DOCS component on it.

Glancing through the PR, the puppetserver.conf docs line describing compile-mode is good, but we should update the tuning guide to explain why someone would (or wouldn't) want to change the setting, and link to that guide from the reference for context.

Generated at Wed Nov 13 17:10:27 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.