Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-1456

Upgrade nss to workaround JDK InternalError at startup

    XMLWordPrintable

    Details

    • Type: CI Blocker
    • Status: Closed
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: SERVER 2.5.0
    • Component/s: None
    • Labels:
    • CI Pipeline/s:
      platform puppetserver
    • Sub-team:
    • Story Points:
      2
    • Sprint:
      Server Emerald 2016-08-10
    • Release Notes:
      Not Needed
    • Release Notes Summary:
      Hide
      When JDK version 1.7.0_111 or newer is installed on a host that has a sufficiently older version of the "nss" package - something older than 3.21.0 on EL 6 or 7, for example - the puppetserver service may fail to start with an error in log files - e.g., /var/log/messages on EL 7 - which contains the following:

      ---
      Caused by: java.lang.InternalError
        at sun.security.ec.SunEC.initialize(Native Method)
        ...
        at sun.security.jca.Providers.getFullProviderList(Providers.java:173)
        at java.security.Security.getProviders(Security.java:456)
        at digest$algorithms.invoke(digest.clj:76)
        ...
      ---

      To avoid this problem, try upgrading to the latest version of the "nss" package before starting/restarting the puppetserver service. For example, on EL systems, you could run:

      ---
      yum upgrade nss
      ---

      This problem has not been seen on EL 6 or 7 systems with "nss" version 3.21.0 and "java-1.7.0-openjdk" version 1.7.0_111.
      Show
      When JDK version 1.7.0_111 or newer is installed on a host that has a sufficiently older version of the "nss" package - something older than 3.21.0 on EL 6 or 7, for example - the puppetserver service may fail to start with an error in log files - e.g., /var/log/messages on EL 7 - which contains the following: --- Caused by: java.lang.InternalError   at sun.security.ec.SunEC.initialize(Native Method)   ...   at sun.security.jca.Providers.getFullProviderList(Providers.java:173)   at java.security.Security.getProviders(Security.java:456)   at digest$algorithms.invoke(digest.clj:76)   ... --- To avoid this problem, try upgrading to the latest version of the "nss" package before starting/restarting the puppetserver service. For example, on EL systems, you could run: --- yum upgrade nss --- This problem has not been seen on EL 6 or 7 systems with "nss" version 3.21.0 and "java-1.7.0-openjdk" version 1.7.0_111.

      Description

      We discovered that on CI pulling in the latest version of the JDK for 1.7, 1.7.0_111, for the Puppet Server 1.x pipeline run that the puppetserver service would fail to be started correctly.

      From /var/log/messages file:

      Jul 28 09:36:34 d8s93urohet13xn java: Exception in thread "main" java.lang.InternalError, compiling:(digest.clj:85:30)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.load(Compiler.java:7142)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:370)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:361)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:440)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:411)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load$fn__5066.invoke(core.clj:5641)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load.doInvoke(core.clj:5640)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:408)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_one.invoke(core.clj:5446)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib.doInvoke(core.clj:5485)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:142)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_libs.doInvoke(core.clj:5524)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:137)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$require.doInvoke(core.clj:5607)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:551)
      Jul 28 09:36:34 d8s93urohet13xn java: at puppetlabs.kitchensink.core$eval419$loading__4958__auto____420.invoke(core.clj:7)
      Jul 28 09:36:34 d8s93urohet13xn java: at puppetlabs.kitchensink.core$eval419.invoke(core.clj:7)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.eval(Compiler.java:6703)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.eval(Compiler.java:6692)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.load(Compiler.java:7130)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:370)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:361)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:440)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:411)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load$fn__5066.invoke(core.clj:5641)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load.doInvoke(core.clj:5640)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:408)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_one.invoke(core.clj:5446)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib.doInvoke(core.clj:5485)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:142)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_libs.doInvoke(core.clj:5524)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:137)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$require.doInvoke(core.clj:5607)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:703)
      Jul 28 09:36:34 d8s93urohet13xn java: at puppetlabs.trapperkeeper.core$eval3$loading__4958__auto____4.invoke(core.clj:1)
      Jul 28 09:36:34 d8s93urohet13xn java: at puppetlabs.trapperkeeper.core$eval3.invoke(core.clj:1)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.eval(Compiler.java:6703)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.eval(Compiler.java:6692)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.load(Compiler.java:7130)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:370)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.loadResourceScript(RT.java:361)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:440)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RT.load(RT.java:411)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load$fn__5066.invoke(core.clj:5641)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load.doInvoke(core.clj:5640)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:408)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_one.invoke(core.clj:5446)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_lib.doInvoke(core.clj:5485)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:142)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$load_libs.doInvoke(core.clj:5524)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.applyTo(RestFn.java:137)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:626)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$require.doInvoke(core.clj:5607)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:408)
      Jul 28 09:36:34 d8s93urohet13xn java: at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:6)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:457)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Var.invoke(Var.java:394)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.AFn.applyToHelper(AFn.java:165)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Var.applyTo(Var.java:700)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.core$apply.invoke(core.clj:624)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.main$main_opt.invoke(main.clj:315)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.main$main.doInvoke(main.clj:420)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.RestFn.invoke(RestFn.java:512)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Var.invoke(Var.java:409)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.AFn.applyToHelper(AFn.java:178)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Var.applyTo(Var.java:700)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.main.main(main.java:37)
      Jul 28 09:36:34 d8s93urohet13xn java: Caused by: java.lang.InternalError
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC.initialize(Native Method)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC.access$000(SunEC.java:49)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC$1.run(SunEC.java:61)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC$1.run(SunEC.java:58)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.security.AccessController.doPrivileged(Native Method)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC.<clinit>(SunEC.java:58)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.security.AccessController.doPrivileged(Native Method)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.ec.SunEC.<clinit>(SunEC.java:58)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.lang.Class.newInstance(Class.java:383)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:221)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.security.AccessController.doPrivileged(Native Method)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderList.loadAll(ProviderList.java:282)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.ProviderList.removeInvalid(ProviderList.java:299)
      Jul 28 09:36:34 d8s93urohet13xn java: at sun.security.jca.Providers.getFullProviderList(Providers.java:173)
      Jul 28 09:36:34 d8s93urohet13xn java: at java.security.Security.getProviders(Security.java:456)
      Jul 28 09:36:34 d8s93urohet13xn java: at digest$algorithms.invoke(digest.clj:76)
      Jul 28 09:36:34 d8s93urohet13xn java: at digest$create_fns.invoke(digest.clj:85)
      Jul 28 09:36:34 d8s93urohet13xn java: at digest$eval708.invoke(digest.clj:88)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.eval(Compiler.java:6703)
      Jul 28 09:36:34 d8s93urohet13xn java: at clojure.lang.Compiler.load(Compiler.java:7130)
      Jul 28 09:36:34 d8s93urohet13xn java: ... 74 more
      Jul 28 09:36:34 d8s93urohet13xn systemd: puppetserver.service: main process exited, code=exited, status=1/FAILURE
      Jul 28 09:36:35 d8s93urohet13xn systemd: puppetserver.service: control process exited, code=exited status=1
      Jul 28 09:36:35 d8s93urohet13xn systemd: Failed to start puppetserver Service.
      Jul 28 09:36:35 d8s93urohet13xn systemd: Unit puppetserver.service entered failed state.
      Jul 28 09:36:35 d8s93urohet13xn systemd: puppetserver.service failed.
      

      We found that our SUTs were running with an older install of the nss-related packages. The following bugs refer to updating nss as helping overcome a startup failure like these.

      https://bugzilla.redhat.com/show_bug.cgi?id=1332456
      https://bugzilla.redhat.com/show_bug.cgi?id=1332867

      In manual testing of a CentOS 6 VM showing this failure, we found that running yum update nss to update the OS to the latest nss version, 3.21.0, allowed the puppetserver service to subsequently be able to start properly.

      We discussed the possibility of addressing java's implicit dependency on nss in packaging. Given the frequency of jdk and nss package rolls and difficulty that would come from coordinating the exact versions of those that would work properly together across different OSes, we're choosing instead to just have our CI pipeline update to the latest nss before installing Puppet Server.

      We should also add a note somewhere to our documentation about this, though, so that end users are aware of this problem and account for it in their own deployments of Puppet Server. /CC Garrett Guillotte for thoughts on the best place to document this.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              jeremy.barlow Jeremy Barlow
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support