Uploaded image for project: 'Trapperkeeper'
  1. Trapperkeeper
  2. TK-149

Runtime refresh of Jetty CRL (jetty 1.8)

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: TK-JETTY9 1.8.1
    • Component/s: None
    • Labels:
    • Template:
    • Team:
      Server
    • Story Points:
      3
    • Sprint:
      Server 2017-07-25, Platform Core 2017-08-08, Platform Core 2017-08-22
    • Release Notes:
      New Feature
    • QA Risk Assessment:
      Manual
    • QA Risk Assessment Reason:
      Fairly large change, but well covered with clojure integration tests as doced. Test once manually with multiple agents.

      Description

      The CRL option which can be set on the SslContextFactory in Jetty (see PE-3914) only allows the CRL to be established at Jetty server start time, by default. For backward compatibility with the CRL behavior in Puppet Server, only loading the CRL at service start time may be sufficient. However, we may want to improve upon this further by allowing the certs loaded into the CRL to be used by a Jetty server right away – without needing to restart Puppet Server and its underlying Jetty. Run-time reloading of the CRL could be desirable for other components built on the trapperkeeper-webservices-jetty9 service - including PuppetDB and PE Console Services.

      Ticket scope

      Here is a proposal for the scope of this ticket:

      1) Investigate approaches for safely, concurrently swapping out the SslContextFactory in the Jetty webserver without needing to restart a currently started Jetty ServerConnector.

      2) Use the trapperkeeper-filesystem-watcher service to register a watcher for the CRL file (and, optionally, the keystore, truststore, and associated parameters which map to files on disk).

      3) On receipt of a filesystem-watcher notification for a changed file, invoke some functionality to refresh Jetty's SslContextFactory instance(s) for the appropriate Server / connectors (without the need for a server restart).

      Additional Jetty background

      The entries in the CRL cert are only processed by Jetty when doStart() is called on the SslContextFactory. From this point on, the CRL entries are referenced by the factory in memory.

      I was able to force Jetty to reload the CRL entries from the cert on disk by having the Jetty webserver service create a custom, derived SslContextFactory which overrides a method that Jetty calls when handling a new client connection, newSSLEngine. Here is the implementation I used:

      (defn- ssl-context-factory-with-crl-refresh []
        (proxy [SslContextFactory] []
          (newSSLEngine
            ([] (proxy-super newSSLEngine))
            ([^InetSocketAddress address]
              (.stop this)
              (.start this)
              (proxy-super newSSLEngine address)))))
      


      The derived class is referenced in place of the Java SslContextFactory by changing the following line in jetty9-core's ssl-context-factory function:

        [context (ssl-context-factory-with-crl-refresh)]
      

      For the overload of newSSLEngine() which includes an address parameter, the underlying SslContextFactory is stopped and restarted. This causes the doStart() method on the class to be run again, loading entries from the CRL cert on disk back into memory and attaching them to a new SslContext object.

      This is a naive implementation which would recreate the SslContext for each connection that the server services. The performance of this solution at scale would probably not be desirable. To optimize this further, it may make sense to create a file watcher around the CRL file and only regenerate the SslContext when the content of the file has actually changed. If this were done asynchronously from the handling of a request on the server side, the impact to clients could be significantly reduced.

      Additionally, any solution in this area would need to handle concurrency. The example above does not do that. The Jetty server may be servicing several requests simultaneously and, so, any changes which would need to be made to the SslContext would need to be protected for concurrent access so as to avoid temporary problems in connection handling – e.g., connection failed or (even worse) unauthorized connection succeeds because the CRL hasn't been fully reattached to the context before a client request is validated against it.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  jeremy.barlow Jeremy Barlow
                  QA Contact:
                  Erik Dasher
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  18 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: