Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Normal
-
Resolution: Fixed
-
Affects Version/s: SERVER 6.11.0
-
Fix Version/s: SERVER 6.12.0
-
Component/s: None
-
Labels:
-
Template:
-
Team:Froyo
-
Method Found:Needs Assessment
-
Release Notes:Bug Fix
-
Release Notes Summary:It was possible to get a StackOverflowError after running puppet server for a while. This was introduced with the upgrade to JRuby 9.2.11.1 in server 6.11 and has been fixed by turning off a new optimization that came with that version of JRuby.
-
QA Risk Assessment:Needs Assessment
Description
Twice since the update to JRuby 9.2.11.1, our internal Puppet deploy has hit an unrecoverable state where catalog compiles and environment_classes requests repeatedly throw StackOverflow errors. This was hit once after running two days, then again after running another week.
Catalog Compiles:
2020-05-01T00:00:22.514-07:00 WARN [qtp547561796-115247] [o.e.j.s.HttpChannel] /puppet/v3/catalog/pe-compiler-infranext-prod-2.infc-aws.puppet.net |
java.lang.StackOverflowError: null |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.lexer2.RUBY$method$scan$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/lexer2.rb:711) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105) |
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:365) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:153) |
at com.headius.racc.Cparse$Parser.call_lexer(Cparse.java:744) |
at com.headius.racc.Cparse$Parser.racc_yyparse(Cparse.java:727) |
at com.headius.racc.Cparse$Parser$INVOKER$i$1$0$racc_yyparse.call(Cparse$Parser$INVOKER$i$1$0$racc_yyparse.gen) |
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837) |
at org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514) |
at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1748) |
at org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
|
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:74) |
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.invokeOther1:__send__(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325) |
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.RUBY$method$yyparse$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:121) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175) |
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203) |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.invokeOther2:yyparse(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246) |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.RUBY$method$_parse$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105) |
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:141) |
...
|
environment_classes:
java.lang.StackOverflowError: null |
at java.base/java.security.AccessController.doPrivileged(Native Method)
|
at java.base/jdk.internal.logger.DefaultLoggerFinder.isSystem(DefaultLoggerFinder.java:142) |
at java.logging/java.util.logging.Logger.demandLogger(Logger.java:650) |
at java.logging/java.util.logging.Logger.getLogger(Logger.java:717) |
at java.logging/java.util.logging.Logger.getLogger(Logger.java:701) |
at com.headius.invokebinder.Binder.<init>(Binder.java:71) |
at com.headius.invokebinder.Binder.from(Binder.java:162) |
at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:98) |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.lexer2.RUBY$method$scan$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/lexer2.rb:711) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105) |
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:365) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:153) |
at com.headius.racc.Cparse$Parser.call_lexer(Cparse.java:744) |
at com.headius.racc.Cparse$Parser.racc_yyparse(Cparse.java:727) |
at com.headius.racc.Cparse$Parser$INVOKER$i$1$0$racc_yyparse.call(Cparse$Parser$INVOKER$i$1$0$racc_yyparse.gen) |
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837) |
at org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514) |
at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1748) |
at org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
|
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:74) |
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.invokeOther1:__send__(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325) |
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.RUBY$method$yyparse$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:121) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175) |
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) |
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203) |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.invokeOther2:yyparse(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246) |
at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.RUBY$method$_parse$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246) |
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95) |
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105) |
...
|
We suspect that this is related to the JRuby bump, though Java was also updated around the same time.
Current course of action is to roll back JRuby to 9.2.8.0, but we should continue to investigate the root cause here. JRuby recently did some work around block handling, and the spot in the lexer that seems to be common to both of these is a yield.