Performance goal is to meet parity with Jruby 1.7
What we know:
- Problem is with JRuby's new interpreter. They expect us to use JIT, which has diminishing returns with max-requests-per-instance (a necessary setting in PE because customer code can be leaky).
- We know there's a lot more garbage than before, and that in 9k we spend much more time in GC pauses. It's known by maintainers that 9k uses more memory.
- We get better behavior with JIT no matter what. Turning this on seems safe.
Our approach going forward:
- Try to optimize max-requests-per-instance behavior by staggering jruby instance lifecycles
- Investigate further where JRuby interpreter might be poorly optimized by profiling particular functionality in Puppet server in smaller chunks (since we only saw the big aggregation from Gatling before)
- Investigate code patterns in Puppet: benchmark with jruby to see if there are patterns we use that are particularly problematic
We still want to decide:
- How we're going to mitigate or optimize the problem after we know more about the nature of it: tuning jruby compiler and JVM, moving ruby code into clojure, improving puppet's code
- What we're going to test in terms of catalogs and environments: both for making sure our solution is sufficient and for watching for regressions