Uploaded image for project: 'Puppet Enterprise'
  1. Puppet Enterprise
  2. ENTERPRISE-453

trapperkeeper-webserver-jetty9 default for max-threads is too low for some systems

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: PE 3.7.0, PE 3.7.1, PE 3.7.2
    • Fix Version/s: PE 3.8.0
    • Component/s: PuppetDB
    • Labels:
    • Environment:

      HP DL380 G8 servers with 20CPU and 256GB of RAM

    • Template:
    • Story Points:
      1
    • Sprint:
      Server 2015-01-21, Server 2015-02-04

      Description

      Our current default for max-threads is a static value of 100. For some servers, apparently those with 256 GB of RAM, this default is insufficient and the service crashes during startup:

      2014-12-09 09:34:51,993 INFO  [p.t.s.w.jetty9-service] Initializing web server.
      2014-12-09 09:34:52,078 INFO  [p.t.s.w.jetty9-service] Starting web server.
      2014-12-09 09:34:52,081 INFO  [o.e.j.s.Server] jetty-9.1.z-SNAPSHOT
      2014-12-09 09:34:52,098 WARN  [o.e.j.u.c.AbstractLifeCycle] FAILED org.eclipse.jetty.server.Server@6251cb36: java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=100 < needed=121
      java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=100 < needed=121
      	at org.eclipse.jetty.server.Server.doStart(Server.java:320) ~[puppetdb.jar:na]
      	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69) ~[puppetdb.jar:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71-icedtea]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71-icedtea]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71-icedtea]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71-icedtea]
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) [puppetdb.jar:na]
      	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313) [puppetdb.jar:na]
      	at puppetlabs.trapperkeeper.services.webserver.jetty9_core$start_webserver.invoke(jetty9_core.clj:206) [na:na]
      	at puppetlabs.trapperkeeper.services.webserver.jetty9_service$reify__19641$f_positional__19646$fnk19754_positional__19755$fn__19756.invoke(jetty9_service.clj:15) [na:na]
      	at puppetlabs.trapperkeeper.services.webserver.jetty9_service$reify__19641$fn$reify__19831.start(jetty9_service.clj:15) [na:na]
      	at puppetlabs.trapperkeeper.services$eval16331$fn__16345$G__16321__16348.invoke(services.clj:12) [na:na]
      	at puppetlabs.trapperkeeper.services$eval16331$fn__16345$G__16320__16352.invoke(services.clj:12) [na:na]
      	at puppetlabs.trapperkeeper.internal$run_lifecycle_fn_BANG_.invoke(internal.clj:143) [na:na]
      	at puppetlabs.trapperkeeper.internal$run_lifecycle_fns.invoke(internal.clj:170) [na:na]
      	at puppetlabs.trapperkeeper.internal$build_app_STAR_$reify__17410.start(internal.clj:385) [na:na]
      	at puppetlabs.trapperkeeper.internal$boot_services_STAR_.invoke(internal.clj:401) [na:na]
      	at puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke(core.clj:114) [na:na]
      	at puppetlabs.trapperkeeper.core$run.invoke(core.clj:145) [na:na]
      	at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:160) [na:na]
      	at clojure.lang.RestFn.applyTo(RestFn.java:137) [puppetdb.jar:na]
      	at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na]
      	at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:374) [na:na]
      	at clojure.lang.RestFn.invoke(RestFn.java:421) [puppetdb.jar:na]
      	at clojure.lang.Var.invoke(Var.java:383) [puppetdb.jar:na]
      	at clojure.lang.AFn.applyToHelper(AFn.java:156) [puppetdb.jar:na]
      	at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na]
      	at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na]
      	at com.puppetlabs.puppetdb.core$run_command.invoke(core.clj:87) [na:na]
      	at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:95) [na:na]
      	at clojure.lang.RestFn.invoke(RestFn.java:436) [puppetdb.jar:na]
      	at clojure.lang.Var.invoke(Var.java:388) [puppetdb.jar:na]
      	at clojure.lang.AFn.applyToHelper(AFn.java:160) [puppetdb.jar:na]
      	at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na]
      	at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na]
      	at clojure.main$main_opt.invoke(main.clj:315) [puppetdb.jar:na]
      	at clojure.main$main.doInvoke(main.clj:420) [puppetdb.jar:na]
      	at clojure.lang.RestFn.invoke(RestFn.java:482) [puppetdb.jar:na]
      	at clojure.lang.Var.invoke(Var.java:401) [puppetdb.jar:na]
      	at clojure.lang.AFn.applyToHelper(AFn.java:171) [puppetdb.jar:na]
      	at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na]
      	at clojure.main.main(main.java:37) [puppetdb.jar:na]
      

      Adding a max-threads = 150 to the configuration of a trapperkeeper-powered service allows it to start up successfully. However, the default value brings Puppet Enterprise services down immediately after installation, before config files can be edited, which results in a failed installation.

      In PE 3.7.0, this issue affects PuppetDB, Console Services and Puppet Server since all three use the Jetty 9 component.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  chuck Charlie Sharpsteen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: