[SERVER-2404] puppetserver 6.1.0 dies with OpenJDK 11, but 6.0.2 worked Created: 2018/12/18  Updated: 2019/01/25  Resolved: 2019/01/09

Status: Closed
Project: Puppet Server
Component/s: Puppet Server
Affects Version/s: SERVER 6.1.0
Fix Version/s: SERVER 6.2.0

Type: Bug Priority: Normal
Reporter: James Ralston Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to SERVER-2421 Puppetserver fails to reload with dyn... Closed
Template: PUP Bug Template
Team: Server
Method Found: Needs Assessment
Release Notes: Bug Fix
Release Notes Summary: This fixes a dependency issue, allowing Puppet Server packages to be started under Java 11.
QA Risk Assessment: Needs Assessment

 Description   

Puppet Version: 6.1.0
Puppet Server Version: 6.1.0
OS Name/Version: Fedora 29 (using RHEL7 packages)

I run Puppet Server on a Fedora 29 host, by using the Puppet packages in the RHEL7 yum repository.

For the last month or so, I have been testing running Puppet Server 6.0.x using OpenJDK 11. As far as I can tell, this worked perfectly. The only artifact I noticed was this start-up message:

2018-12-17T11:38:50.020003-05:00 myhost puppetserver[1878]: WARNING: An illegal reflective access operation has occurred
2018-12-17T11:38:50.020003-05:00 myhost puppetserver[1878]: WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1894/jruby15087714324251502022jopenssl.jar) to constructor java.security.cert.CertificateFactory(java.security.cert.CertificateFactorySpi,java.security.Provider,java.lang.String)
2018-12-17T11:38:50.020003-05:00 myhost puppetserver[1878]: WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
2018-12-17T11:38:50.020003-05:00 myhost puppetserver[1878]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2018-12-17T11:38:50.021000-05:00 myhost puppetserver[1878]: WARNING: All illegal access operations will be denied in a future release

So I was excited to see the Puppet Server 6.1.0 release, with "official" support for OpenJDK 11.

But, irony of ironies, if I attempt to start Puppet Server 6.1.0 with OpenJDK 11, it outright dies. Here's the stack backtrace:

Dec 18 17:59:54 myhost systemd[1]: Starting puppetserver Service...
Dec 18 18:00:05 myhost puppetserver[4700]: Exception in thread "main" java.lang.ExceptionInInitializerError
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.defaults__init.load(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.defaults__init.<clinit>(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.loadClassForName(RT.java:2232)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:450)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:426)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load$fn__6548.invoke(core.clj:6046)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.invokeStatic(core.clj:6045)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.doInvoke(core.clj:6029)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invokeStatic(core.clj:5848)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invoke(core.clj:5843)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.invokeStatic(core.clj:5887)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.doInvoke(core.clj:5868)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.invokeStatic(core.clj:5925)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.doInvoke(core.clj:5909)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.invokeStatic(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.doInvoke(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:421)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.util$loading__6434__auto____14850.invoke(util.clj:1)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.util__init.load(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.util__init.<clinit>(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.loadClassForName(RT.java:2232)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:450)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:426)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load$fn__6548.invoke(core.clj:6046)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.invokeStatic(core.clj:6045)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.doInvoke(core.clj:6029)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invokeStatic(core.clj:5848)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invoke(core.clj:5843)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.invokeStatic(core.clj:5887)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.doInvoke(core.clj:5868)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.invokeStatic(core.clj:5925)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.doInvoke(core.clj:5909)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.invokeStatic(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.doInvoke(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:421)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.kitchensink.classpath$loading__6434__auto____14848.invoke(classpath.clj:1)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.kitchensink.classpath__init.load(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.kitchensink.classpath__init.<clinit>(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.loadClassForName(RT.java:2232)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:450)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:426)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load$fn__6548.invoke(core.clj:6046)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.invokeStatic(core.clj:6045)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.doInvoke(core.clj:6029)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invokeStatic(core.clj:5848)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invoke(core.clj:5843)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.invokeStatic(core.clj:5887)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.doInvoke(core.clj:5868)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.invokeStatic(core.clj:5925)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.doInvoke(core.clj:5909)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.invokeStatic(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.doInvoke(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:457)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.plugins$loading__6434__auto____14846.invoke(plugins.clj:1)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.plugins__init.load(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.plugins__init.<clinit>(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.loadClassForName(RT.java:2232)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:450)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:426)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load$fn__6548.invoke(core.clj:6046)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.invokeStatic(core.clj:6045)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.doInvoke(core.clj:6029)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invokeStatic(core.clj:5848)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invoke(core.clj:5843)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.invokeStatic(core.clj:5887)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.doInvoke(core.clj:5868)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.invokeStatic(core.clj:5925)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.doInvoke(core.clj:5909)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.invokeStatic(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.doInvoke(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:1096)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.core$loading__6434__auto____199.invoke(core.clj:1)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.core__init.load(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.core__init.<clinit>(Unknown Source)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.loadClassForName(RT.java:2232)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:450)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.load(RT.java:426)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load$fn__6548.invoke(core.clj:6046)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.invokeStatic(core.clj:6045)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load.doInvoke(core.clj:6029)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invokeStatic(core.clj:5848)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_one.invoke(core.clj:5843)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.invokeStatic(core.clj:5887)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_lib.doInvoke(core.clj:5868)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.invokeStatic(core.clj:5925)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$load_libs.doInvoke(core.clj:5909)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:659)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.invokeStatic(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$require.doInvoke(core.clj:5947)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.main$_main.invokeStatic(main.clj:6)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:4)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.Var.applyTo(Var.java:702)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.core$apply.invokeStatic(core.clj:657)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.main$main_opt.invokeStatic(main.clj:317)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.main$main_opt.invoke(main.clj:313)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.main$main.invokeStatic(main.clj:424)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.main$main.doInvoke(main.clj:387)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.Var.applyTo(Var.java:702)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.main.main(main.java:37)
Dec 18 18:00:05 myhost puppetserver[4700]: Caused by: java.lang.ClassNotFoundException: sun.misc.Launcher$ExtClassLoader
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2204)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at clojure.lang.RT.classForName(RT.java:2213)
Dec 18 18:00:05 myhost puppetserver[4700]: #011at dynapath.defaults$fn__14921.<clinit>(defaults.clj:14)
Dec 18 18:00:05 myhost puppetserver[4700]: #011... 151 more
Dec 18 18:00:05 myhost puppetserver[4700]: Background process 4707 exited before start had completed
Dec 18 18:00:05 myhost systemd[1]: puppetserver.service: Control process exited, code=exited status=1
Dec 18 18:00:05 myhost systemd[1]: puppetserver.service: Failed with result 'exit-code'.
Dec 18 18:00:05 myhost systemd[1]: Failed to start puppetserver Service.

If I switch the default JDK back to OpenJDK 8, then Puppet Server starts normally.

Desired Behavior:

Puppet Server 6.1.x should not die with OpenJDK 11—especially since Puppet Server 6.0.x worked just fine with OpenJDK 11!

Actual Behavior:

Puppet Server 6.1.x dies with OpenJDK 11.



 Comments   
Comment by Justin Stoller [ 2018/12/18 ]

Minor correction: we announced "experimental" support for Java 11.

Prior to 6.1.0 we had no automated testing of Java 11, if it passed it was purely accidental, we now have unit style tests for Java 11 and they are all currently passing. We have not made it to package level testing, hence the "experimental" nature of the support.

Comment by James Ralston [ 2018/12/18 ]

I just tested Puppet Server 6.1.0 on CentOS 7 with OpenJDK 11, and it crashes with a similar (if not identical) stack trace.

So the fact that I initially encountered this problem on Fedora 29 isn't relevant. The problem is with Puppet Server 6.1.0.

Comment by James Ralston [ 2018/12/18 ]

Justin Stoller, I understand your point.

But my point is that from a user perspective, this "enhancement" from the release notes:

Puppet Server now has experimental support for Java 11. This has been tested with low level tests but has not gone through high level packaged acceptance tests. Consequently, we consider this support “experimental”, with full support coming later in 2019 for the latest long term supported version of Java.

is an utter lie. From a user perspective, there is no possible sense that you can claim even experimental support for Java 11 in 6.1, when 6.1 outright crashes if you attempt to use Java 11. This is especially egregious considering that 6.0 did work with Java 11, even if that was only by accident.

Documentation of enhancements versus regressions in a new release must always be from the user's perspective. Or, to put it more colorfully: don't pee on the users' heads and tell them it's raining.

I suggest deleting the above bullet point from the release notes, and adding either of the below bullet points to the known issues:

For those of you who were batsh*t crazy enough to try using 6.0 with Java 11 and realized that it actually worked, you'll notice that 6.1 doesn't work with Java 11. So, 1) we admire your bravery/stupidity, and 2) sorry about that. But we had to break Java 11 support temporarily in order to work on Official™ support for Java 11 done right, which will land later in 2019. So quit your whining; it'll be worth it, we promise.

Or:

While Puppet Server 6.1 contains the initial groundwork for eventual full support for Java 11, Puppet Server 6.1 does not currently work with Java 11. This could potentially be viewed as a regression, as Puppet Server 6.0 did work with Java 11 in some circumstances, but this was only by accident. Therefore, if you were using Java 11 with Puppet Server 6.0, you will need to revert to Java 8 for Puppet Server 6.1. (Full support for Java 11 will come later in 2019.)

(I know which one I'd go with, but that's why I'm in engineering and not PR.)

Comment by Justin Stoller [ 2018/12/18 ]

What do you think the portion of users are who will use the package, vs running from source, when testing out Java 11?

Comment by Justin Stoller [ 2018/12/19 ]

I appreciate you opening this ticket, btw, we love to hear first hand the good and the bad of how working with Puppet Server is going.

I'm still curious about what you think the portion of users who would try out experimental Java 11 support would do so from the packaged artifacts vs running from source or compiling it themselves? You might have inferred from the "this is passing low level tests" that we have a high confidence in this code from source but aren't testing packaged artifacts of it (I don't think we'd mark it as experimental support if we were).

I'll look into updating the release notes to call out that the experimental support is only for those who want to try it out from source/compile it themselves.

I looked through the stacktrace and have built Puppet Server on a CentOS 7 box several times to confirm why this is happening and it looks like this was caused with our move across the projects to AOTing clojure code (with the hopes improved start up times).

While the code works if you compile Puppet Server's using JDK 11, it looks like there's a dependency of a dependency (dynapath) that is doing some "if running in Java 8 try this api and if running in Java 9 or above use this api". It looks like in this commit (38fae3f) they fixed the bug where they were doing that at compile time and now properly do it at run time.

I'll bump that dependency soon and it should come out in the next release of Puppet Server, though I still can't say if every will be perfect (it started in my test but I definitely didn't run our full acceptance suite against it).

Comment by James Ralston [ 2018/12/20 ]

Justin Stoller, I don't understand your question.

I suspect you are under the assumption that because the RHEL7 puppetserver package contains an explicit dependency on java-1.8.0-openjdk-headless:

$ rpm -q --requires puppetserver | grep java
java-1.8.0-openjdk-headless

…that puppetserver will always launch with OpenJDK 8.

This is not the case, due to this excerpt from /etc/sysconfig/puppetserver:

# Location of your Java binary (version 7 or higher)
JAVA_BIN="/usr/bin/java"

On Red Hat / CentOS / Fedora systems, /usr/bin/java is controlled through the alternatives(8) system:

$ ls -lsa /usr/bin/java
4 lrwxrwxrwx. 1 root root 22 Dec 19 11:43 /usr/bin/java -> /etc/alternatives/java*

This means that the puppetserver process will be launched with whatever JDK the system prefers, which will not necessarily be OpenJDK 8.

In my case, I installed the OpenJDK 11 packages as soon as they were available:

$ yum install java-11-openjdk-headless

If both OpenJDK 8 and OpenJDK 11 are installed, and alternatives(8) is permitted to choose the "best" Java automatically, OpenJDK 8 will win:

$ alternatives --display java | grep -v slave
java - status is auto.
 link currently points to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-11.fc29.x86_64/jre/bin/java
/usr/lib/jvm/java-11-openjdk-11.0.1.13-4.fc29.x86_64/bin/java - family java-11-openjdk.x86_64 priority 1
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-11.fc29.x86_64/jre/bin/java - family java-1.8.0-openjdk.x86_64 priority 1800191
Current `best' version is /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-11.fc29.x86_64/jre/bin/java.

But because I explicitly wanted to test to see what Java applications would break with OpenJDK 11, I explicitly set OpenJDK 11 to be the default:

$ alternatives --set java java-11-openjdk.x86_64
 
$ alternatives --display java | grep -v slave
java - status is manual.
 link currently points to /usr/lib/jvm/java-11-openjdk-11.0.1.13-4.fc29.x86_64/bin/java
/usr/lib/jvm/java-11-openjdk-11.0.1.13-4.fc29.x86_64/bin/java - family java-11-openjdk.x86_64 priority 1
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-11.fc29.x86_64/jre/bin/java - family java-1.8.0-openjdk.x86_64 priority 1800191
Current `best' version is /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-11.fc29.x86_64/jre/bin/java.
 
$ /usr/bin/java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode, sharing)

If it is the case that puppetserver should be launched only using OpenJDK 8, then /etc/sysconfig/puppetserver is wrong. It should instead use the alternatives(8) paths to ensure it launches with its desired JDK:

# Currently only OpenJDK 8 is guaranteed to work.  Other Java JDKs (e.g.,
# OpenJDK 11) may or may not work, so change this at your own risk!
JAVA_BIN="/etc/alternatives/jre_1.8.0_openjdk/bin/java"

Otherwise, all that is required for a system administrator to configure Puppet Server to launch with OpenJDK 11 is to run:

$ yum install java-11-openjdk-headless
$ alternatives --set java java-11-openjdk.x86_64

…and boom, Puppet Server is broken, using the official Puppet Server RPM packages, without touching a single Puppet Server configuration file. Running from source or manually compiling Puppet Server is not necessary.

I would argue that forward-thinking sysadmins will configure development/beta/test systems to use OpenJDK 11 as the default JDK, for the exact same reasons I did (to find incompatibilities in advance).

Comment by Justin Stoller [ 2018/12/20 ]

I'm aware of how alternatives work.

We also have FOSS users who run releases from source and other users that build their own packages.

My assumption was that the population of users who would be most likely to try Java 11 would include a higher percentage of those who also run from source or build their own packages than one typically sees. I assume those using packages are the majority, but I don't know if there's a meaningful portion of those trying out Java 11 using non official package based installation methods, or if it's just like three BSD dudes running from source and a government guy building his own packages, ya know?

Comment by James Ralston [ 2018/12/20 ]

I'm not sure the assumption that the population of users who would be most likely to try Java 11 would include a higher percentage of those who also run Puppet Server from source or build their own packages is a valid one.

Red Hat pushed java-11-openjdk RPMs to RHEL7 at the end of October. Red Hat fully supports OpenJDK 11. And while Red Hat and the OpenJDK folks have promised to continue security updates for OpenJDK 8, the writing is on the wall: OpenJDK 8 is legacy, and all sites should be preparing to transition to OpenJDK 11.

That's why I configured my test/development systems to use OpenJDK 11 by default. I wasn't specifically trying to break Puppet Server; I was attempting to find any applications that broke. It just so happened that I was running Puppet Server on one of my test/development systems. I actually expected Puppet Server to break with OpenJDK 11; I was pleasantly surprised when it didn't. I assumed that Puppet was already well aware of the need to transition from OpenJDK 8 to OpenJDK 11, and had started that process, even though they had made no announcements yet.

When Puppet Server 6.1 did have an explicit announcement that work to officially support OpenJDK 11 was underway, I figured that it was an extension of the work that had already landed in 6.0, and that Puppet was just making it official. That is why it was such a rude surprise when Puppet 6.1 broke with OpenJDK 11, and why I assumed that was an unintentional regression. (Why else would Puppet break the very thing they were announcing preliminary support for?)

In summary, this is less a technical issue than it is a complaint with how the release announcement was worded. Breaking something that previously worked (even if only by accident) is not an "enhancement". It's a regression, and should be noted as a "known issue" (as per my suggested phrasing in my second comment). The fact that it wasn't is why I assumed you did not know about it, and why I filed this ticket.

Comment by Justin Stoller [ 2018/12/21 ]

For background we did have some company sponsored work to prepare for, and potentially support Java 11 sooner rather than later, with Redhat announcing support for Java 8 until 2023 (only a year less than Java 11) the steam ran out of that endeavor last summer. While we'll probably have official Java 11 support sometime next year there's no commitments as to when exactly and the work necessary for us to build and test packages with Java 11 is currently unscheduled (assuming backwards compatibility in a minor releases we run the oldest minor that we can so we don't accidentally depend on a library that is only available in newer releases, consequently we run EL 7.2 in our infra).

To the extent we have "experimental" support is to the degree some of us developers have "seen the writing on the wall" and want to engage with the community and make the inevitable transition as easy as possible for everyone. We've done what we've can and tested this in our dev environments and in travis and have high confidence from a low level source perspective and hoped that that degree of confidence would be welcomed by the community. I'm sorry, that it worked before based on our earlier 9+ support work and that it doesn't work when installed from packages with this release, none of us were intending to "pee on the [your] head and tell [you] it's raining".

I've opened PRs to update the release notes and fix the dependency that's causing this issue.

Comment by Maggie Dreyer [ 2019/01/07 ]

The PR bumping that dependency in puppetserver is https://github.com/puppetlabs/puppetserver/pull/1935.

Comment by Kenn Hussey [ 2019/01/08 ]

Justin Stoller should this be moved to 'Ready for CI' (since it appears that the associated PR has been merged)?

Comment by Maggie Dreyer [ 2019/01/08 ]

We ran into an issue with the dynapath update, will need to keep investigating.

Comment by Maggie Dreyer [ 2019/01/09 ]

With the fix in SERVER-2421, puppetserver should now once again be able to start on Java 11. Note that reloading is still broken, and seems to have always been, see SERVER-2423.

Generated at Wed Nov 20 22:25:55 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.