Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2387

puppetserver crashes on startup because it can't start multiple jruby instances on JRE 10

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: SERVER 6.0.2
    • Fix Version/s: SERVER 6.6.0
    • Component/s: Puppet Server
    • Labels:
      None
    • Template:
      PUP Bug Template
    • Team:
      Server
    • Method Found:
      Customer Feedback
    • Release Notes:
      Not Needed
    • Release Notes Summary:
      See release notes on SERVER-2423.
    • QA Risk Assessment:
      Needs Assessment

      Description

      Puppet Version: 6.0.2
      Puppet Server Version: 6.0.2
      OS Name/Version: Ubuntu 18.04 (bionic)

      The default JRE on Ubuntu Bionic is openjdk-11-jre, which is actually jdk10:

      root@p6:~# java --version
      openjdk 10.0.2 2018-07-17
      OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
      OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode)

      Using this JRE version with max-active-instances > 1 (either explicitly set or because you have more than 2 CPUs) makes puppetserver crash on startup wth:

       2018-11-21T22:05:59.225Z ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      

      you can work around this either by setting max-active-instances to 1 or by switching to JDK8. 

      Desired Behavior:

       ** Per the docs, any JDK 1.7 or greater should work.  Puppetserver should either be fixed or the docs should state compatible JDKs.

      Actual Behavior:

      root@p6:~# lsb_release -a
      No LSB modules are available.
      Distributor ID:	Ubuntu
      Description:	Ubuntu 18.04.1 LTS
      Release:	18.04
      Codename:	bionic
       
      root@p6:~# java --version
      openjdk 10.0.2 2018-07-17
      OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
      OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode)
      root@p6:~# dpkg -l |grep default-jre
      ii  default-jre                   2:1.10-63ubuntu1~02               amd64        Standard Java or Java compatible Runtime
      ii  default-jre-headless          2:1.10-63ubuntu1~02               amd64        Standard Java or Java compatible Runtime (headless)
       
      root@p6:~# dpkg -l |grep openjdk-11
      ii  openjdk-11-jre:amd64          10.0.2+13-1ubuntu0.18.04.3        amd64        OpenJDK Java runtime, using Hotspot JIT
      ii  openjdk-11-jre-headless:amd64 10.0.2+13-1ubuntu0.18.04.3        amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
      root@p6:~# dpkg -l |grep puppetserver
      ii  puppetserver                  6.0.2-1bionic                     all          Puppet Labs puppetserver. Contains: Puppet Server (puppetlabs/puppetserver 6.0.2,org.clojure/clojure 1.9.0,org.clojure/clojurescript 1.10.238,puppetlabs/puppetserver 6.0.2,puppetlabs/trapperkeeper-webserver-jetty9 2.3.0)
      root@p6:~# grep max-active-instances /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf 
          max-active-instances: 2
       
      root@p6:~# puppetserver foreground
      2018-11-21T23:03:22.064Z INFO  [main] [o.e.j.u.log] Logging initialized @25392ms to org.eclipse.jetty.util.log.Slf4jLog
      2018-11-21T23:03:22.321Z INFO  [main] [p.t.s.w.jetty9-core] Removing buggy security provider SunPKCS11 version 10
      2018-11-21T23:03:24.332Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
      2018-11-21T23:03:24.376Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'puppet-profiler', version 6.0.2
      2018-11-21T23:03:24.379Z INFO  [async-dispatch-2] [p.s.j.jruby-puppet-service] Initializing the JRuby service
      2018-11-21T23:03:24.397Z INFO  [async-dispatch-2] [p.s.j.jruby-pool-manager-service] Initializing the JRuby service
      2018-11-21T23:03:24.406Z INFO  [async-dispatch-2] [p.s.j.jruby-puppet-service] JRuby version info: jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a OpenJDK 64-Bit Server VM 10.0.2+13-Ubuntu-1ubuntu0.18.04.3 on 10.0.2+13-Ubuntu-1ubuntu0.18.04.3 +jit [linux-x86_64]
      2018-11-21T23:03:24.430Z INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-internal] Creating JRubyInstance with id 1.
      2018-11-21T23:03:24.449Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'jruby-metrics', version 6.0.2
      WARNING: An illegal reflective access operation has occurred
      WARNING: Illegal reflective access by org.jruby.ext.zlib.RubyZlib (file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar) to field java.util.zip.CRC32.crc
      WARNING: Please consider reporting this to the maintainers of org.jruby.ext.zlib.RubyZlib
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
      WARNING: All illegal access operations will be denied in a future release
      2018-11-21T23:03:31.694Z INFO  [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
      2018-11-21T23:03:36.282Z INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 1 of 2
      2018-11-21T23:03:36.283Z INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-internal] Creating JRubyInstance with id 2.
      2018-11-21T23:03:36.286Z INFO  [async-dispatch-2] [p.s.c.puppet-server-config-core] Initializing webserver settings from core Puppet
      2018-11-21T23:03:36.319Z INFO  [async-dispatch-2] [p.p.certificate-authority] CA already initialized for SSL
      2018-11-21T23:03:36.322Z INFO  [async-dispatch-2] [p.s.c.certificate-authority-service] CA Service adding a ring handler
      2018-11-21T23:03:36.379Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'ca', version 6.0.2
      2018-11-21T23:03:36.383Z INFO  [async-dispatch-2] [p.s.p.puppet-admin-service] Starting Puppet Admin web app
      2018-11-21T23:03:36.422Z INFO  [async-dispatch-2] [p.s.v.versioned-code-service] No code-id-command set for versioned-code-service. Code-id will be nil.
      2018-11-21T23:03:36.424Z INFO  [async-dispatch-2] [p.s.v.versioned-code-service] No code-content-command set for versioned-code-service. Attempting to fetch code content will fail.
      2018-11-21T23:03:36.633Z INFO  [async-dispatch-2] [p.p.certificate-authority] Master already initialized for SSL
      2018-11-21T23:03:36.639Z INFO  [async-dispatch-2] [p.s.m.master-service] Master Service adding ring handlers
      2018-11-21T23:03:36.640Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'master', version 6.0.2
      2018-11-21T23:03:36.712Z WARN  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Empty contextPath
      2018-11-21T23:03:36.720Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Starting web server(s).
      2018-11-21T23:03:36.735Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-cert'
      2018-11-21T23:03:36.735Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-key'
      2018-11-21T23:03:36.736Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-ca-cert'
      2018-11-21T23:03:36.736Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-crl-path'
      2018-11-21T23:03:36.953Z INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] Starting web server.
      2018-11-21T23:03:36.956Z INFO  [async-dispatch-2] [o.e.j.s.Server] jetty-9.4.z-SNAPSHOT; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.2+13-Ubuntu-1ubuntu0.18.04.3
      2018-11-21T23:03:37.046Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@235c4bc0{/puppet-ca,null,AVAILABLE}
      2018-11-21T23:03:37.048Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@7d1a78bf{/puppet-admin-api,null,AVAILABLE}
      2018-11-21T23:03:37.049Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@5591294a{/metrics,null,AVAILABLE}
      2018-11-21T23:03:37.067Z INFO  [async-dispatch-2] [o.e.j.s.session] DefaultSessionIdManager workerName=node0
      2018-11-21T23:03:37.067Z INFO  [async-dispatch-2] [o.e.j.s.session] No SessionScavenger set, using defaults
      2018-11-21T23:03:37.069Z INFO  [async-dispatch-2] [o.e.j.s.session] node0 Scavenging every 660000ms
      2018-11-21T23:03:37.114Z INFO  [async-dispatch-2] [p.t.s.m.jolokia] Using policy access restrictor classpath:/jolokia-access.xml
      2018-11-21T23:03:37.211Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.ServletContextHandler@78f39b3c{/metrics/v2,null,AVAILABLE}
      2018-11-21T23:03:37.213Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@5682cb1d{/puppet,null,AVAILABLE}
      2018-11-21T23:03:37.219Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@a0573fe{/,null,AVAILABLE}
      2018-11-21T23:03:37.270Z INFO  [async-dispatch-2] [o.e.j.u.s.SslContextFactory] x509=X509@52b084c9(private key,h=[puppet, puppetca, puppet-ca, puppet],w=[]) for InternalSslContextFactory@7921349c[provider=null,keyStore=null,trustStore=null]
      2018-11-21T23:03:37.401Z INFO  [async-dispatch-2] [o.e.j.s.AbstractConnector] Started ServerConnector@478a9f97{SSL,[ssl, http/1.1]}{0.0.0.0:8140}
      2018-11-21T23:03:37.401Z INFO  [async-dispatch-2] [o.e.j.s.Server] Started @40731ms
      2018-11-21T23:03:37.428Z INFO  [async-dispatch-2] [p.t.s.s.status-core] Starting background monitoring of cpu usage metrics
      2018-11-21T23:03:37.441Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for service 'status-service', version 1.1.0
      2018-11-21T23:03:37.443Z INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status service HTTP API at /status
      2018-11-21T23:03:37.453Z INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@57a40b24{/status,null,AVAILABLE}
      2018-11-21T23:03:37.500Z INFO  [async-dispatch-2] [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
      2018-11-21T23:03:37.501Z INFO  [async-dispatch-2] [p.s.l.legacy-routes-service] The legacy routing service has successfully started and is now ready to handle requests
      2018-11-21T23:03:37.530Z INFO  [async-dispatch-2] [p.s.a.analytics-service] Puppet Server Update Service has successfully started and will run in the background
      2018-11-21T23:03:39.656Z INFO  [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
      2018-11-21T23:03:40.006Z ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      org/jruby/javasupport/JavaPackage.java:127:in `const_missing'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:12:in `value'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confine/variable.rb:21:in `facter_value'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confine/variable.rb:57:in `test_value'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confine/variable.rb:45:in `valid?'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confine_collection.rb:48:in `block in valid?'
      org/jruby/RubyEnumerable.java:624:in `detect'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confine_collection.rb:48:in `valid?'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/confiner.rb:44:in `suitable?'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1971:in `block in suitableprovider'
      org/jruby/RubyHash.java:1350:in `each'
      org/jruby/RubyEnumerable.java:788:in `find_all'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1970:in `suitableprovider'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1760:in `defaultprovider'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1923:in `block in default'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:835:in `set_default'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:2394:in `initialize'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/file.rb:490:in `initialize'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:478:in `to_ral'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:633:in `block in to_catalog'
      org/jruby/RubyArray.java:1735:in `each'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:625:in `to_catalog'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:515:in `to_ral'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1005:in `use'
      uri:classloader:/puppetserver-lib/puppet/server/puppet_config.rb:45:in `initialize_puppet'
      uri:classloader:/puppetserver-lib/puppet/server/master.rb:29:in `initialize'
      NameError: Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      2018-11-21T23:03:40.011Z ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception!
      java.lang.IllegalStateException: There was a problem adding a JRubyInstance to the pool.
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481$fn__33485.invoke(jruby_agents.clj:75)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481.invoke(jruby_agents.clj:48)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33726$send_prime_pool_BANG___33731$fn__33732$fn__33733.invoke(jruby_agents.clj:233)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:389)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:374)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364)
      	at puppetlabs.trapperkeeper.internal$eval16091$shutdown_service__16096$fn$reify__16098$service_fnk__5906__auto___positional$reify__16103.shutdown_on_error(internal.clj:429)
      	at puppetlabs.trapperkeeper.internal$eval16021$fn__16033$G__16013__16041.invoke(internal.clj:397)
      	at puppetlabs.trapperkeeper.internal$eval16021$fn__16033$G__16012__16050.invoke(internal.clj:397)
      	at clojure.core$partial$fn__5561.invoke(core.clj:2617)
      	at clojure.core$partial$fn__5561.invoke(core.clj:2616)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33451$send_agent__33456$fn__33457$agent_fn__33458.invoke(jruby_agents.clj:42)
      	at clojure.core$binding_conveyor_fn$fn__5476.invoke(core.clj:2025)
      	at clojure.lang.AFn.applyToHelper(AFn.java:154)
      	at clojure.lang.RestFn.applyTo(RestFn.java:132)
      	at clojure.lang.Agent$Action.doRun(Agent.java:114)
      	at clojure.lang.Agent$Action.run(Agent.java:163)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base/java.lang.Thread.run(Thread.java:844)
      Caused by: org.jruby.embed.InvokeFailedException: (NameError) Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      	at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
      	at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:170)
      	at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1475)
      	at com.puppetlabs.jruby_utils.jruby.InternalScriptingContainer.callMethodWithArgArray(InternalScriptingContainer.java:180)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
      	at puppetlabs.services.jruby.jruby_puppet_core$eval35004$get_initialize_pool_instance_fn__35009$fn__35010$fn__35011.invoke(jruby_puppet_core.clj:151)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$eval33050$create_pool_instance_BANG___33059$fn__33062.invoke(jruby_internal.clj:211)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$eval33050$create_pool_instance_BANG___33059.invoke(jruby_internal.clj:177)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481$fn__33485.invoke(jruby_agents.clj:65)
      	... 20 common frames omitted
      Caused by: org.jruby.exceptions.RaiseException: (NameError) Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      2018-11-21T23:03:40.013Z INFO  [main] [p.t.internal] Beginning shutdown sequence
      2018-11-21T23:03:40.028Z INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopping metrics sampler job
      2018-11-21T23:03:40.032Z INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopped metrics sampler job
      2018-11-21T23:03:40.039Z INFO  [async-dispatch-3] [p.s.j.i.jruby-agents] Draining JRuby pool.
      2018-11-21T23:03:40.041Z ERROR [async-dispatch-3] [p.t.internal] Encountered error during shutdown sequence
      java.lang.InterruptedException: Lock can't be granted because a pill has been inserted
      	at com.puppetlabs.jruby_utils.pool.JRubyPool.lockWithTimeout(JRubyPool.java:368)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33557$borrow_all_jrubies__33562$fn__33563$fn__33564.invoke(jruby_agents.clj:126)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33557$borrow_all_jrubies__33562$fn__33563.invoke(jruby_agents.clj:125)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33557$borrow_all_jrubies__33562.invoke(jruby_agents.clj:117)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33637$drain_and_refill_pool_BANG___33646$fn__33649.invoke(jruby_agents.clj:191)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33637$drain_and_refill_pool_BANG___33646.invoke(jruby_agents.clj:173)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33678$flush_pool_for_shutdown_BANG___33683$fn__33684.invoke(jruby_agents.clj:212)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33678$flush_pool_for_shutdown_BANG___33683.invoke(jruby_agents.clj:200)
      	at puppetlabs.services.jruby_pool_manager.jruby_core$eval34587$flush_pool_for_shutdown_BANG___34592$fn__34593.invoke(jruby_core.clj:236)
      	at puppetlabs.services.jruby_pool_manager.jruby_core$eval34587$flush_pool_for_shutdown_BANG___34592.invoke(jruby_core.clj:231)
      	at puppetlabs.services.jruby.jruby_puppet_service$reify__35531$service_fnk__5906__auto___positional$reify__35545.stop(jruby_puppet_service.clj:52)
      	at puppetlabs.trapperkeeper.services$eval5704$fn__5731$G__5696__5734.invoke(services.clj:9)
      	at puppetlabs.trapperkeeper.services$eval5704$fn__5731$G__5695__5738.invoke(services.clj:9)
      	at puppetlabs.trapperkeeper.internal$eval15624$run_lifecycle_fn_BANG___15631$fn__15632.invoke(internal.clj:198)
      	at puppetlabs.trapperkeeper.internal$eval15624$run_lifecycle_fn_BANG___15631.invoke(internal.clj:181)
      	at puppetlabs.trapperkeeper.internal$eval16148$shutdown_BANG___16153$fn__16154$shutdown_fn__16156$fn__16171.invoke(internal.clj:440)
      	at puppetlabs.trapperkeeper.internal$eval16148$shutdown_BANG___16153$fn__16154$shutdown_fn__16156.invoke(internal.clj:439)
      	at puppetlabs.trapperkeeper.internal$eval15698$initialize_lifecycle_worker__15709$fn__15710$fn__15860$state_machine__12806__auto____15885$fn__15888$fn__15902.invoke(internal.clj:276)
      	at puppetlabs.trapperkeeper.internal$eval15698$initialize_lifecycle_worker__15709$fn__15710$fn__15860$state_machine__12806__auto____15885$fn__15888.invoke(internal.clj:260)
      	at puppetlabs.trapperkeeper.internal$eval15698$initialize_lifecycle_worker__15709$fn__15710$fn__15860$state_machine__12806__auto____15885.invoke(internal.clj:251)
      	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
      	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
      	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
      	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
      	at clojure.core.async$ioc_alts_BANG_$fn__13035.invoke(async.clj:383)
      	at clojure.core.async$do_alts$fn__12967$fn__12970.invoke(async.clj:252)
      	at clojure.core.async.impl.channels.ManyToManyChannel$fn__7535.invoke(channels.clj:135)
      	at clojure.lang.AFn.run(AFn.java:22)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base/java.lang.Thread.run(Thread.java:844)
      2018-11-21T23:03:40.055Z INFO  [async-dispatch-3] [p.t.s.w.jetty9-service] Shutting down web server(s).
      2018-11-21T23:03:40.059Z INFO  [async-dispatch-3] [p.t.s.w.jetty9-core] Shutting down web server.
      2018-11-21T23:03:40.062Z INFO  [async-dispatch-3] [o.e.j.s.AbstractConnector] Stopped ServerConnector@478a9f97{SSL,[ssl, http/1.1]}{0.0.0.0:8140}
      2018-11-21T23:03:40.062Z INFO  [async-dispatch-3] [o.e.j.s.session] node0 Stopped scavenging
      2018-11-21T23:03:40.062Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@57a40b24{/status,null,UNAVAILABLE}
      2018-11-21T23:03:40.062Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@a0573fe{/,null,UNAVAILABLE}
      2018-11-21T23:03:40.063Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@5682cb1d{/puppet,null,UNAVAILABLE}
      2018-11-21T23:03:40.064Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.ServletContextHandler@78f39b3c{/metrics/v2,null,UNAVAILABLE}
      2018-11-21T23:03:40.065Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@5591294a{/metrics,null,UNAVAILABLE}
      2018-11-21T23:03:40.065Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@7d1a78bf{/puppet-admin-api,null,UNAVAILABLE}
      2018-11-21T23:03:40.065Z INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@235c4bc0{/puppet-ca,null,UNAVAILABLE}
      2018-11-21T23:03:40.070Z INFO  [async-dispatch-3] [p.t.s.w.jetty9-core] Web server shutdown
      2018-11-21T23:03:40.078Z INFO  [clojure-agent-send-off-pool-1] [p.t.s.w.filesystem-watch-core] Closing watcher puppetlabs.trapperkeeper.services.watcher.filesystem_watch_core.WatcherImpl@5cc928ea
      2018-11-21T23:03:40.078Z INFO  [clojure-agent-send-off-pool-0] [p.t.s.w.filesystem-watch-core] Closing watcher puppetlabs.trapperkeeper.services.watcher.filesystem_watch_core.WatcherImpl@5146ec72
      2018-11-21T23:03:40.084Z INFO  [main] [p.t.internal] Finished shutdown sequence
      Exception in thread "main" java.lang.IllegalStateException: There was a problem adding a JRubyInstance to the pool.
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481$fn__33485.invoke(jruby_agents.clj:75)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481.invoke(jruby_agents.clj:48)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33726$send_prime_pool_BANG___33731$fn__33732$fn__33733.invoke(jruby_agents.clj:233)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:389)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:374)
      	at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364)
      	at puppetlabs.trapperkeeper.internal$eval16091$shutdown_service__16096$fn$reify__16098$service_fnk__5906__auto___positional$reify__16103.shutdown_on_error(internal.clj:429)
      	at puppetlabs.trapperkeeper.internal$eval16021$fn__16033$G__16013__16041.invoke(internal.clj:397)
      	at puppetlabs.trapperkeeper.internal$eval16021$fn__16033$G__16012__16050.invoke(internal.clj:397)
      	at clojure.core$partial$fn__5561.invoke(core.clj:2617)
      	at clojure.core$partial$fn__5561.invoke(core.clj:2616)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33451$send_agent__33456$fn__33457$agent_fn__33458.invoke(jruby_agents.clj:42)
      	at clojure.core$binding_conveyor_fn$fn__5476.invoke(core.clj:2025)
      	at clojure.lang.AFn.applyToHelper(AFn.java:154)
      	at clojure.lang.RestFn.applyTo(RestFn.java:132)
      	at clojure.lang.Agent$Action.doRun(Agent.java:114)
      	at clojure.lang.Agent$Action.run(Agent.java:163)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base/java.lang.Thread.run(Thread.java:844)
      Caused by: org.jruby.embed.InvokeFailedException: (NameError) Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      	at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
      	at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:170)
      	at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1475)
      	at com.puppetlabs.jruby_utils.jruby.InternalScriptingContainer.callMethodWithArgArray(InternalScriptingContainer.java:180)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
      	at puppetlabs.services.jruby.jruby_puppet_core$eval35004$get_initialize_pool_instance_fn__35009$fn__35010$fn__35011.invoke(jruby_puppet_core.clj:151)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$eval33050$create_pool_instance_BANG___33059$fn__33062.invoke(jruby_internal.clj:211)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$eval33050$create_pool_instance_BANG___33059.invoke(jruby_internal.clj:177)
      	at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval33476$prime_pool_BANG___33481$fn__33485.invoke(jruby_agents.clj:65)
      	... 20 more
      Caused by: org.jruby.exceptions.RaiseException: (NameError) Could not create resources for managing Puppet's files and directories in sections [:main, :master, :ssl, :metrics]: cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      cannot link Java class com.puppetlabs.Facter (java.lang.UnsatisfiedLinkError: Native Library /opt/puppetlabs/puppet/lib/libfacter.so.3.12.0 already loaded in another classloader)
      root@p6:~# 
       
      
      

       ** 

       

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  brandonrochon Brandon Rochon
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: