Affects Version/s: SERVER 2.y
Fix Version/s: SERVER 2.5.0
Component/s: Puppet Server
Release Notes:Bug Fix
Release Notes Summary:Joe will be adding some notes on this to the docs in the repo; we created
SERVER-1470to capture the write-up.
NOTE: repurposing this ticket as an epic to capture smaller chunks of work. Modifying the original description to capture our proposed path forward and some acceptance criteria. Original description follows afterward.
OSS users who have modified their bootstrap.cfg file to support a multi-master setup (and thus have disabled the CA on their compile masters) are subject to a painful upgrade experience, because if the new package contains any changes to bootstrap.cfg, then the packaging system can't overlay the new config file due to the user's local changes. If the user accepts the packaged version of the config file, their CA configuration will be broken, and if they opt to keep their local changes then the server may not start due to missing or invalid services that changed in the new package.
The crux of the issue is that bootstrap.cfg currently contains some configuration that we don't expect users to ever change, and some configuration that we do expect them to change. We need a better mechanism for separating those two classes of configuration.
- Users who are not managing / modifying their bootstrap.cfg file have been able to upgrade seamlessly, and the changes that result from this ticket must continue to allow for a seamless upgrade for these users.
- Users who have been experiencing this issue will have to deal with it one more time when we roll out the changes described by this epic, but in subsequent releases we should be able to deliver a seamless upgrade experience.
- We should at least look into some modules such as jlambert121-puppet, and see if we can propose a change that will be compatible with both older and newer releases. It seems like we'd need a way to detect the puppetserver version number in order to make that work, so I'm not 100% sure how feasible it is, but we should look into it.
- Trapperkeeper will be updated to support loading and combining multiple bootstrap .cfg files, including from a conf.d-style directory containing multiple files
- All of the base services that are required to run Puppet Server, which we do not expect a user to need to modify, will be moved to a puppetserver-bootstrap.cfg file, somewhere in the /opt directory, hidden from the user's normal interaction with configuration files.
- We will remove the file /etc/puppetlabs/puppetserver/bootstrap.cfg, and in its place we will create a directory, /etc/puppetlabs/puppetserver/services.d, which Puppet Server will use (in addition to the file in /opt) when loading services at boot time. By default this directory will contain a single file, ca.cfg, that will contain the default CA service. If users wish to disable the CA (e.g. for compile masters), they would modify this file but the file in /opt would remain intact.
On the first upgrade after these changes are released, users who have disabled the CA will need to tweak how they are managing their CA settings to be aware of this new ca.cfg file instead of the old bootstrap.cfg file, but in subsequent releases, any packaged changes to the puppetserver-bootstrap.cfg file in /opt should apply cleanly without interfering with any CA customizations.
The TL;DR version: the monolithic bootstrap.cfg file makes Puppet server upgrades a flag-day nightmare to perform for sites that employ more than a single Puppet server.
So here's the problem:
Each Puppet server version has a particular set of services that must be declared in the bootstrap.cfg file, or else the Puppet server will fail to start.
For example, Puppet server 2.3.0 needs this new service:
If this service is not specified in the bootstrap.cfg file, Puppet server 2.3.0 will fail to start.
But Puppet server has no forward compatibility when it comes to services specified in the bootstrap.cfg file, because the Puppet server will also fail to start if a service is specified in the bootstrap.cfg file that it doesn't know how to load.
This places sysadmins in a "damned if you do, damned if you don't" situation when it comes to Puppet server upgrades: if the bootstrap.cfg file is being managed by Puppet, it is impossible to create a single bootstrap.cfg file template that will work across Puppet 2.3.0 and Puppet < 2.3.0 servers: the absence of the versioned-code service definition will cause Puppet server 2.3.0 to die, and the presence of the versioned-code service definition will cause Puppet server < 2.3.0 to die.
The fundamental problem here is that the bootstrap.cfg is erroneously commingling two completely different types of data:
- required service definitions that depend on the specific version of the Puppet server installed on the system, and must not be removed or added by the sysadmin
- optional service definitions that depend on the intended use of the Puppet server, and are intended to be removed or added by the sysadmin
The versioned-code service definition is an example of #1, but the certificate-authority and certificate-authority-disabled services are examples of #2. And as long as both types of service definitions are commingled in the same file, it is only possible to upgrade that file cleanly (e.g., when a new version of Puppet server changes the required set of service definitions) if the file has never been modified by the sysadmin.
Except, oops: sites that have multiple Puppet servers have no choice but to modify the bootstrap.cfg file on all Puppet servers except the CA server.
Fortunately, the way out of this mess is simple: split the bootstrap.cfg into a directory of individual files, a la the conf.d directory. All *.service files in the subdirectory will be loaded, in lexicographical order. Required, version-specific service definitions can be placed into a single file, while service definitions that a sysadmin may wish to modify should be placed into separate files. E.g.:
For sites that have never modified the bootstrap.cfg file, this change should be transparent. For sites that have modified the bootstrap.cfg file, every Puppet server upgrade is already a hands-on PITA anyway, so it's not going to be any worse than what we already have to deal with. And after the transition, future upgrades should no longer have this issue.
Issues in Epic
||SERVER-1247||Packaging changes to support bootstrap.cfg improvements in TK and EZBake||Closed||Erik Dasher|
||TK-375||Regression in 1.4.0 when loading bootstrap.cfg from resources/ classpath||Resolved||qa|
||EZ-72||Implement support for overriding bootstrap-config path in app's project.clj||Resolved||John Duarte|
||EZ-78||Update docs for bootstrap UX upgrade work||Resolved||Joe Pinsonault|
||TK-211||Trapperkeeper doesn't error if two services implementing the same protocol are started||Resolved||Erik Dasher|
||TK-349||TK should not fail during startup if an unrecognized service is found in bootstrap config||Resolved||Erik Dasher|
||EZ-71||Investigate possibility of projects overriding bootstrap-config path||Closed||Joe Pinsonault|
||EZ-73||allow projects to package bootstrap files in service.d directories under both /etc and /opt||Closed||Unassigned|
||TK-351||Ensure all bootstrap related errors log what file they come from||Resolved||Erik Dasher|
||TK-347||Support directories and paths in TK's "bootstrap-config" CLI argument||Resolved||Unassigned|
||SERVER-1248||Test bootstrap.cfg modifications||Resolved||Unassigned|