[SERVER-1543] com.puppetlabs.http.client should allow adding certificates to supplement Puppet CA - Puppet Jira Server

XML

Word

Printable

Details

Type: Story
Status: Resolved
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
- nice-to-have

Epic Link:
Use System cert store
Team:
Froyo
Sprint:
Froyo - 4/27/2022

CS Priority:
Reviewed

Description

The com.puppetlabs.http.client library was created and wrapped into Puppet::Server::HttpClient for Puppet Server due to limitations in the JRuby emulation layer for OpenSSL. However, the clients created using this library don't trust the CA certificates in the JVM cacerts keystore by default. This means that the client isn't able to successfully validate HTTPS connections that don't use certificates from the Puppet CA.

A common use case is the built-in http report processor submitting a Puppet report to a 3rd party endpoint.

Reproduction Case

Install a PE 2016.2.1 monolithic master.

Attempt to establish a HTTPS connection to a server that doesn't use a certificate from the Puppet CA:

# connection_test.rb

require 'puppet/server/puppet_config'

require 'puppet/server/http_client'

Puppet::Network::HttpPool.http_client_class = Puppet::Server::HttpClient

Puppet::Server::PuppetConfig.initialize_puppet({})

conn = Puppet::Network::HttpPool.http_instance('github.com', 443, true)

response = conn.get('index.html', {})

puts response.inspect

Outcome

Fetching index.html from github.com using HTTPS fails due to a SSL handshake error:

# /opt/puppetlabs/server/bin/puppetserver ruby connection_test.rb

18:23:30.475 [main] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] start execution

18:23:30.483 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default

18:23:30.491 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context

18:23:30.491 [main] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] Request connection for {s}->https://github.com:443

18:23:30.492 [main] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection request: [route: {s}->https://github.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]

18:23:30.624 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection leased: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]

18:23:30.627 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]}

18:23:30.627 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][r:]: Set attribute http.nio.exchange-handler

18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Event set [w]

18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Set timeout 0

18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [ACTIVE]: Connected

18:23:30.628 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:]: Set attribute http.nio.http-exchange-state

18:23:30.630 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - Start connection routing

18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 Upgrade session 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][141][0]

18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Connection route established

18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] Attempt 1 to execute request

18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Target auth state: UNCHALLENGED

18:23:30.676 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - Proxy auth state: UNCHALLENGED

18:23:30.676 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /index.html HTTP/1.1

18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: close

18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip, deflate

18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: github.com:443

18:23:30.677 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpAsyncClient/4.1.1 (Java/1.8.0_101)

18:23:30.677 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][rw:][ACTIVE][rw][NEED_UNWRAP][0][0][141][0]: Event set [w]

18:23:30.677 [I/O dispatcher 1] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] Request completed

18:23:30.781 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Exception

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

       	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) ~[na:1.8.0_101]

       	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.8.0_101]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:303) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:507) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) [puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339) [puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317) [puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) [puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) [puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) [puppet-server-release.jar:na]

       	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

       	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[na:1.8.0_101]

       	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[na:1.8.0_101]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:281) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:351) ~[puppet-server-release.jar:na]

       	... 9 common frames omitted

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_101]

       	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_101]

       	at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[na:1.8.0_101]

       	... 17 common frames omitted

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_101]

       	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_101]

       	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_101]

       	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_101]

       	... 23 common frames omitted

18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 10.32.47.91:33839<->192.30.253.113:443[ACTIVE][r:r][ACTIVE][rw][NEED_WRAP][inbound done][][9][0][0][0]: Shutdown

18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.ManagedNHttpClientConnectionImpl - http-outgoing-0 0.0.0.0:33839<->192.30.253.113:443[CLOSED][][CLOSED][rw][NEED_WRAP][inbound done][][9][0][0][0]: Shutdown

18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.InternalHttpAsyncClient - [exchange: 1] connection aborted

18:23:30.782 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]

18:23:30.783 [I/O dispatcher 1] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection released: [id: http-outgoing-0][route: {s}->https://github.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]

18:23:30.783 [I/O dispatcher 1] DEBUG o.a.h.i.n.client.InternalIODispatch - http-outgoing-0 [CLOSED]: Disconnected

18:23:30.785 [main] ERROR c.p.h.c.i.PersistentSyncHttpClient - Error executing http request

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

       	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) ~[na:1.8.0_101]

       	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.8.0_101]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:303) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:507) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) ~[puppet-server-release.jar:na]

       	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) ~[puppet-server-release.jar:na]

       	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

       	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_101]

       	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[na:1.8.0_101]

       	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]

       	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[na:1.8.0_101]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:281) ~[puppet-server-release.jar:na]

       	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:351) ~[puppet-server-release.jar:na]

       	... 9 common frames omitted

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_101]

       	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_101]

       	at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[na:1.8.0_101]

       	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[na:1.8.0_101]

       	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[na:1.8.0_101]

       	... 17 common frames omitted

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_101]

       	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_101]

       	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_101]

       	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_101]

       	... 23 common frames omitted

Puppet::Server::HttpClientError: Error executing http request

  client_get at file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/http_client.rb:178

         get at file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/http_client.rb:85

      (root) at connection_test.rb:7

      invoke at jruby_puppet_core.clj:315

      invoke at jruby_puppet_core.clj:309

      invoke at subcommand.clj:38

    doInvoke at ruby.clj:7

      invoke at core.clj:630

      invoke at main.clj:316

    doInvoke at main.clj:421

Expected Outcome

The request should succeed, because the DigiCert High Assurance EV Root CA (used to sign the cert that signed the GitHub cert) is part of the Java 8 cacerts keystore:

# /opt/puppetlabs/server/bin/keytool -list -keystore /opt/puppetlabs/puppet/ssl/puppet-cacerts -storepass changeit|grep 'digicert_high_assurance'

digicert_high_assurance_ev_root_ca:2.16.2.172.92.38.106.11.64.155.143.11.121.242.174.70.37.119, Jul 19, 2016, trustedCertEntry,

Result should be similar to:

# /opt/puppetlabs/server/bin/puppetserver ruby connection_test.rb

18:26:55.329 [main] DEBUG o.a.h.impl.nio.client.MainClientExec - [exchange: 1] start

... snip ...

#<Net::HTTPOK 200  readbody=true>

Attachments

Issue Links

blocks

PUP-7737 Add HTTP report processor setting for including system truststore

Resolved

is blocked by

SERVER-2943 Allow clj-http-client to optionally load system stores from agent's bundle

Resolved

SERVER-2944 Make puppetserver http client respect `include_system_store` option

Resolved

is duplicated by

SERVER-1641 Puppet Server shouldn't rely on the system ca bundle

Closed

relates to

SERVER-2898 CA Certificates are not used by Puppetserver

Accepted

PUP-5069 Puppet's HTTP API does not allow callers to trust system cacerts

Resolved

TK-122 Add ability to configure CRL for SSL requests to clj-http-client

Closed

links to

Zendesk Ticket #21038

(2 relates to, 1 links to)

Activity

People

Assignee:: Justin Stoller

Reporter:: Charlie Sharpsteen

Votes:: 4 Vote for this issue

Watchers:: 26 Start watching this issue

Dates

Created:: 2016/08/31 6:31 PM

Updated:: 2022/04/27 10:13 AM

Resolved:: 2022/04/27 10:13 AM