Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-1543

com.puppetlabs.http.client should allow adding certificates to supplement Puppet CA

    XMLWordPrintable

Details

    • Story
    • Status: Ready for Engineering
    • High
    • Resolution: Unresolved
    • None
    • None
    • None
    • Froyo
    • Reviewed

    Description

      The com.puppetlabs.http.client library was created and wrapped into Puppet::Server::HttpClient for Puppet Server due to limitations in the JRuby emulation layer for OpenSSL. However, the clients created using this library don't trust the CA certificates in the JVM cacerts keystore by default. This means that the client isn't able to successfully validate HTTPS connections that don't use certificates from the Puppet CA.

      A common use case is the built-in http report processor submitting a Puppet report to a 3rd party endpoint.

      Reproduction Case

      Install a PE 2016.2.1 monolithic master.

      Attempt to establish a HTTPS connection to a server that doesn't use a certificate from the Puppet CA:

      # connection_test.rb
      require 'puppet/server/puppet_config'
      require 'puppet/server/http_client'
      Puppet::Network::HttpPool.http_client_class = Puppet::Server::HttpClient
      Puppet::Server::PuppetConfig.initialize_puppet({})
      conn = Puppet::Network::HttpPool.http_instance('github.com', 443, true)
       
      response = conn.get('index.html', {})
       
      puts response.inspect
      

      Outcome

      Fetching index.html from github.com using HTTPS fails due to a SSL handshake error:

      # /opt/puppetlabs/server/bin/puppetserver ruby connection_test.rb
      18:23:30.475 [main] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] start execution
      18:23:30.483 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
      18:23:30.491 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
      18:23:30.491 [main] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] Request connection for {s}->https://github.com:443
      18:23:30.492 [main] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection request: [route: {s}->https://github.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
      18:23:30.624 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection leased: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
      18:23:30.627 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]}
      18:23:30.627 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][r:]: Set attribute http.nio.exchange-handler
      18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Event set [w]
      18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Set timeout 0
      18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [ACTIVE]: Connected
      18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Set attribute http.nio.http-exchange-state
      18:23:30.630 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - Start connection routing
      18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 Upgrade session 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][141][0]
      18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Connection route established
      18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] Attempt 1 to execute request
      18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Target auth state: UNCHALLENGED
      18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Proxy auth state: UNCHALLENGED
      18:23:30.676 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /index.html HTTP/1.1
      18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: close
      18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip, deflate
      18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: github.com:443
      18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpAsyncClient/4.1.1 (Java/1.8.0_101)
      18:23:30.677 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][141][0]: Event set [w]
      18:23:30.677 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] Request completed
      18:23:30.781 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Exception
      javax.net.ssl.SSLHandshakeException: General SSLEngine problem
             	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) ~[na:1.8.0_101]
             	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.8.0_101]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:303) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:507) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) [puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339) [puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317) [puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) [puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) [puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) [puppet-server-release.jar:na]
             	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
      Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
             	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[na:1.8.0_101]
             	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[na:1.8.0_101]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:281) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:351) ~[puppet-server-release.jar:na]
             	... 9 common frames omitted
      Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
             	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_101]
             	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_101]
             	at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[na:1.8.0_101]
             	... 17 common frames omitted
      Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
             	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_101]
             	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_101]
             	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_101]
             	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_101]
             	... 23 common frames omitted
      18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][r:r][ACTIVE][rw][NEED_WRAP][inbound done][][9][0][0][0]: Shutdown
      18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 0.0.0.0:33839<->192.30.253.113:443[CLOSED][][CLOSED][rw][NEED_WRAP][inbound done][][9][0][0][0]: Shutdown
      18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] connection aborted
      18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
      18:23:30.783 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection released: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
      18:23:30.783 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [CLOSED]: Disconnected
      18:23:30.785 [main] ERROR c.p.h.c.i.PersistentSyncHttpClient - Error executing http request
      javax.net.ssl.SSLHandshakeException: General SSLEngine problem
             	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) ~[na:1.8.0_101]
             	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.8.0_101]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:303) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:507) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) ~[puppet-server-release.jar:na]
             	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) ~[puppet-server-release.jar:na]
             	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
      Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
             	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_101]
             	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[na:1.8.0_101]
             	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]
             	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[na:1.8.0_101]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:281) ~[puppet-server-release.jar:na]
             	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:351) ~[puppet-server-release.jar:na]
             	... 9 common frames omitted
      Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
             	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_101]
             	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_101]
             	at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[na:1.8.0_101]
             	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[na:1.8.0_101]
             	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[na:1.8.0_101]
             	... 17 common frames omitted
      Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
             	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_101]
             	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_101]
             	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_101]
             	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_101]
             	... 23 common frames omitted
      Puppet::Server::HttpClientError: Error executing http request
        client_get at file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/http_client.rb:178
               get at file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/http_client.rb:85
            (root) at connection_test.rb:7
            invoke at jruby_puppet_core.clj:315
            invoke at jruby_puppet_core.clj:309
            invoke at subcommand.clj:38
          doInvoke at ruby.clj:7
            invoke at core.clj:630
            invoke at main.clj:316
          doInvoke at main.clj:421
      

      Expected Outcome

      The request should succeed, because the DigiCert High Assurance EV Root CA (used to sign the cert that signed the GitHub cert) is part of the Java 8 cacerts keystore:

      # /opt/puppetlabs/server/bin/keytool -list -keystore /opt/puppetlabs/puppet/ssl/puppet-cacerts -storepass changeit|grep 'digicert_high_assurance'
      digicert_high_assurance_ev_root_ca:2.16.2.172.92.38.106.11.64.155.143.11.121.242.174.70.37.119, Jul 19, 2016, trustedCertEntry,
      

      Result should be similar to:

      # /opt/puppetlabs/server/bin/puppetserver ruby connection_test.rb
      18:26:55.329 [main] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] start
      ... snip ...
      #<Net::HTTPOK 200  readbody=true>
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              chuck Charlie Sharpsteen
              Votes:
              3 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:

                Zendesk Support