Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-3094

docker container: Critical: CA files stored in volatile storage

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Ready for Engineering
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Puppet Server
    • Labels:
      None
    • Template:
      PUP Bug Template
    • Team:
      Froyo
    • Story Points:
      2
    • Method Found:
      Needs Assessment
    • QA Risk Assessment:
      Needs Assessment

      Description

      Summary:
      After upgrading the puppetserver container, the CA directory is gone and therefore nothing work.


      This issue concerns puppetserver run as a docker container.

      The project https://github.com/puppetlabs/puppetserver/ does not have its own Issue tracking at github (as pupperware do), so I file this ticket here at tickets.puppetlabs.com instead:

      Puppet Server Version: All (the container currently use 7.4.0-1bionic)
      OS Name/Version: Any linux (probably the same problem in Windows)

      This bug is so huge I really can't believe it exists.
      It also means that no one must have ever upgraded their puppetserver container, or they would have experienced the same.

      CRITICAL fault:

      Inside the container we have:

      # ls -l /etc/puppetlabs/puppet/ssl
      lrwxrwxrwx 1 puppet puppet   31 Sep 20 14:59 ca -> /etc/puppetlabs/puppetserver/ca
      drwxr-xr-x 2 puppet puppet 4096 Sep 20 14:59 certificate_requests
      drwxr-xr-x 2 puppet puppet 4096 Sep 20 14:59 certs
      -rw-r--r-- 1 puppet puppet 1994 Sep 20 14:59 crl.pem
      drwxr-x--- 2 puppet puppet 4096 Sep 20 14:59 private
      drwxr-x--- 2 puppet puppet 4096 Sep 20 14:59 private_keys
      drwxr-xr-x 2 puppet puppet 4096 Sep 20 14:59 public_keys
      

      The directory /etc/puppetlabs/puppet is mounted in the persistent volume compose-services_puppetserver-config.
      However, see how the ca directory is symlinked away to another place, /etc/puppetlabs/puppetserver/ca.
      This place is volatile!!! It's part of the container's own filesystem, which is gone if you e.g. upgrade the container to a new version of puppetserver.

      After an upgrade, and the new container start, that place is empty.
      So the ca symlink now point to an empty directory, and the entire puppet server stop working, 'cause all certificate lookup calls (like "GET /puppet-ca/v1/certificate/ca HTTP/1.0") now get a 404 not found.

      Desired Behavior:
      All important files must be stored in a persistent volume.

      Actual Behavior:
      The ca files are stored in the container image's own filesystem.
      As long as one keep this docker overlay2 layer, things work fine (including reboots).
      But if you upgrade the container to a new version (like from v7.3.0 to 7.4.0), the new puppetserver 7.4.0 will not work.
      PS: The old overlay filesystem is also removed when the new image is pulled and started, so you can't even extract a copy of your old CA files. They are gone.

      Example of my setup:
      I configure pupperware to install puppetserver v7.3.0 (and db7.5.2) and bring it up with: docker-compose up -d.
      All is good. CA certificate files are created. All agents can connect and run puppet just fine.

      Puppetserver 7.4.0 is released. I take the stack down:
      docker-compose down
      I configure pupperware to install puppetserver v7.4.0 (and db7.6.0).
      docker-compose up -d
      Now the 7.4.0 image is pulled and the stack is started.

      The new 7.4.0-server, with its new container image, lack the CA files where the symlink point, so everything stop working. No agents can connect.


      Since /etc/puppetlabs/puppet is mounted to a persistent volume, files in the subdir /etc/puppetlabs/puppet/ssl/* is already persistent. I don't understand why /etc/puppetlabs/puppet/ssl/ca is symlinked off to another place.
      Why not simply skip the symlink and leave the files in ssl/? Problem solved?
      The only explaination I can think of is a human ooops: Perhaps this symlink is some old attempt to place the CA files in the persistent volume compose-services_puppetserver-data (not -config), but instead of linking to /opt/puppetlabs/server/data/puppetserver/ca it became /etc/puppetlabs/puppetserver/ca, i.e. the wrong 'puppetserver' dir.


      Just a paste of my own debug history:
      In the 7.3.0-server, I create a file, via exec inside the container:

      touch /etc/puppetlabs/puppet/ssl/ca/foobar.tmp
      exit
      updatedb
      locate foobar.tmp
      /var/lib/docker/overlay2/e7e17620cd11d7d4d888532c95eff0d099044bba98e077c1039116a75d37ecec/diff/etc/puppetlabs/puppetserver/ca/foobar.tmp
      /var/lib/docker/overlay2/e7e17620cd11d7d4d888532c95eff0d099044bba98e077c1039116a75d37ecec/merged/etc/puppetlabs/puppetserver/ca/foobar.tmp
      

      My file (and the ca files) are placed in overlay e7e17620...
      I now take down the stack and upgrade.
      7.4.0 (and db7.6.0) is pulled and started.
      I exec into 7.4.0 and my file (and the ca files) are gone.
      I create foobar2.txt and exit back to the docker mother.

      # locate foobar2.tmp
      /var/lib/docker/overlay2/66717a12d1632c7f18ef12162020d683f3096780da2478605a5ded3def7510b9/diff/etc/puppetlabs/puppetserver/ca/foobar2.tmp
      /var/lib/docker/overlay2/66717a12d1632c7f18ef12162020d683f3096780da2478605a5ded3def7510b9/merged/etc/puppetlabs/puppetserver/ca/foobar2.tmp
      

      foobar2.txt is created in the image filesystem of puppetserver:7.4.0 (overlay 66717a12...).
      The old overlay e7e17620... no longer exists, so I can't even try to get the original CA files back. The files are gone.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            elof Elof Ofel
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:

                Zendesk Support