Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2793

StackOverflow errors after Puppet Server runs for a while

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: SERVER 6.11.0
    • Fix Version/s: SERVER 6.12.0
    • Component/s: None
    • Labels:
    • Template:
      PUP Bug Template
    • Team:
      Froyo
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      It was possible to get a StackOverflowError after running puppet server for a while. This was introduced with the upgrade to JRuby 9.2.11.1 in server 6.11 and has been fixed by turning off a new optimization that came with that version of JRuby.
    • QA Risk Assessment:
      Needs Assessment

      Description

      Twice since the update to JRuby 9.2.11.1, our internal Puppet deploy has hit an unrecoverable state where catalog compiles and environment_classes requests repeatedly throw StackOverflow errors. This was hit once after running two days, then again after running another week.

       

      Catalog Compiles:

      2020-05-01T00:00:22.514-07:00 WARN  [qtp547561796-115247] [o.e.j.s.HttpChannel] /puppet/v3/catalog/pe-compiler-infranext-prod-2.infc-aws.puppet.net
      java.lang.StackOverflowError: null
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.lexer2.RUBY$method$scan$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/lexer2.rb:711)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:365)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:153)
        at com.headius.racc.Cparse$Parser.call_lexer(Cparse.java:744)
        at com.headius.racc.Cparse$Parser.racc_yyparse(Cparse.java:727)
        at com.headius.racc.Cparse$Parser$INVOKER$i$1$0$racc_yyparse.call(Cparse$Parser$INVOKER$i$1$0$racc_yyparse.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)
        at org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
        at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1748)
        at org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:74)
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.invokeOther1:__send__(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325)
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.RUBY$method$yyparse$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:121)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.invokeOther2:yyparse(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246)
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.RUBY$method$_parse$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:141)
      ... 

      environment_classes:

      java.lang.StackOverflowError: null
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/jdk.internal.logger.DefaultLoggerFinder.isSystem(DefaultLoggerFinder.java:142)
        at java.logging/java.util.logging.Logger.demandLogger(Logger.java:650)
        at java.logging/java.util.logging.Logger.getLogger(Logger.java:717)
        at java.logging/java.util.logging.Logger.getLogger(Logger.java:701)
        at com.headius.invokebinder.Binder.<init>(Binder.java:71)
        at com.headius.invokebinder.Binder.from(Binder.java:162)
        at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:98)
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.lexer2.RUBY$method$scan$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/lexer2.rb:711)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:365)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:153)
        at com.headius.racc.Cparse$Parser.call_lexer(Cparse.java:744)
        at com.headius.racc.Cparse$Parser.racc_yyparse(Cparse.java:727)
        at com.headius.racc.Cparse$Parser$INVOKER$i$1$0$racc_yyparse.call(Cparse$Parser$INVOKER$i$1$0$racc_yyparse.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)
        at org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
        at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1748)
        at org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:74)
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.invokeOther1:__send__(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325)
        at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.racc.parser.RUBY$method$yyparse$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/racc/parser.rb:325)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:121)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.invokeOther2:yyparse(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246)
        at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.parser.parser_support.RUBY$method$_parse$0(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/parser_support.rb:246)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:95)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:105)
      ... 

      We suspect that this is related to the JRuby bump, though Java was also updated around the same time.

      Current course of action is to roll back JRuby to 9.2.8.0, but we should continue to investigate the root cause here. JRuby recently did some work around block handling, and the spot in the lexer that seems to be common to both of these is a yield.

        Attachments

          Activity

            People

            Assignee:
            patrick Patrick Carlisle
            Reporter:
            maggie Maggie Dreyer
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Zendesk Support