Status: Ready for Engineering
Affects Version/s: None
Fix Version/s: None
Component/s: Puppet Server
Template:PUP Bug Template customfield_10700 416503
Method Found:Needs Assessment
QA Risk Assessment:Needs Assessment
After upgrading the puppetserver container, the CA directory is gone and therefore nothing work.
This issue concerns puppetserver run as a docker container.
The project https://github.com/puppetlabs/puppetserver/ does not have its own Issue tracking at github (as pupperware do), so I file this ticket here at tickets.puppetlabs.com instead:
Puppet Server Version: All (the container currently use 7.4.0-1bionic)
OS Name/Version: Any linux (probably the same problem in Windows)
This bug is so huge I really can't believe it exists.
It also means that no one must have ever upgraded their puppetserver container, or they would have experienced the same.
Inside the container we have:
The directory /etc/puppetlabs/puppet is mounted in the persistent volume compose-services_puppetserver-config.
However, see how the ca directory is symlinked away to another place, /etc/puppetlabs/puppetserver/ca.
This place is volatile!!! It's part of the container's own filesystem, which is gone if you e.g. upgrade the container to a new version of puppetserver.
After an upgrade, and the new container start, that place is empty.
So the ca symlink now point to an empty directory, and the entire puppet server stop working, 'cause all certificate lookup calls (like "GET /puppet-ca/v1/certificate/ca HTTP/1.0") now get a 404 not found.
All important files must be stored in a persistent volume.
The ca files are stored in the container image's own filesystem.
As long as one keep this docker overlay2 layer, things work fine (including reboots).
But if you upgrade the container to a new version (like from v7.3.0 to 7.4.0), the new puppetserver 7.4.0 will not work.
PS: The old overlay filesystem is also removed when the new image is pulled and started, so you can't even extract a copy of your old CA files. They are gone.
Example of my setup:
I configure pupperware to install puppetserver v7.3.0 (and db7.5.2) and bring it up with: docker-compose up -d.
All is good. CA certificate files are created. All agents can connect and run puppet just fine.
Puppetserver 7.4.0 is released. I take the stack down:
I configure pupperware to install puppetserver v7.4.0 (and db7.6.0).
docker-compose up -d
Now the 7.4.0 image is pulled and the stack is started.
The new 7.4.0-server, with its new container image, lack the CA files where the symlink point, so everything stop working. No agents can connect.
Since /etc/puppetlabs/puppet is mounted to a persistent volume, files in the subdir /etc/puppetlabs/puppet/ssl/* is already persistent. I don't understand why /etc/puppetlabs/puppet/ssl/ca is symlinked off to another place.
Why not simply skip the symlink and leave the files in ssl/? Problem solved?
The only explaination I can think of is a human ooops: Perhaps this symlink is some old attempt to place the CA files in the persistent volume compose-services_puppetserver-data (not -config), but instead of linking to /opt/puppetlabs/server/data/puppetserver/ca it became /etc/puppetlabs/puppetserver/ca, i.e. the wrong 'puppetserver' dir.
Just a paste of my own debug history:
In the 7.3.0-server, I create a file, via exec inside the container:
My file (and the ca files) are placed in overlay e7e17620...
I now take down the stack and upgrade.
7.4.0 (and db7.6.0) is pulled and started.
I exec into 7.4.0 and my file (and the ca files) are gone.
I create foobar2.txt and exit back to the docker mother.
foobar2.txt is created in the image filesystem of puppetserver:7.4.0 (overlay 66717a12...).
The old overlay e7e17620... no longer exists, so I can't even try to get the original CA files back. The files are gone.