Affects Version/s: None
Method Found:Needs Assessment
Release Notes:Bug Fix
Release Notes Summary:Previously package upgrades on CentOS 6 might fail when upgrading from from older versions of PuppetDB (like 5.2) to more recent versions (6.3 or newer). That should no longer be the case.
QA Risk Assessment:Needs Assessment
During the PDB upgrade_oldest acceptance tests on centos6 for 6.3.x and master we try to upgrade the PDB package from 5.2.0 to the latest snapshot build 6.3.3-SNAPSHOT. This upgrade fails in our tests and results in PDB ending up in a state where running service puppetdb stop is unable to stop PDB and it must be killed by hand. This problem only appears to affect platforms which are running sysvinit and not systemd.
It appears that due to changes with how PDB is invoked in later versions the main namespace provided via the -m argument has changed.
In versions prior to PDB 6.1.0 the main namespace was puppetlabs.puppetdb.main in later versions it has changed to puppetlabs.puppetdb.cli.services.
When testing upgrades from 5.2.0 -> 6.3.2 on a centos6 by hand I was able to see that the command service puppetdb stop was looking for the pid of PDB using the new main namespace when it was still running under the old name. One possibility is that the newer stop script is being run when trying to upgrade which is resulting in service not finding the old name and the right pid to kill.
1. install pdb 5.2.0 on centos6
2. grab a package version later than 6.1
3. yum install the package while pdb is running (this will put the upgrade in a broken state)
4. add a set -x to /opt/puppetlabs/server/apps/puppetdb/cli/apps/stop
5. run service puppetdb stop
6. compare the output from the pgrep call in the stop script to the main namespace pdb is being run under.
We may need to make changes to lein-ezbake templates that create our stop/start scripts to account for the name change on platforms running sysvinit. see lein-ezbake templates
Workaround in case anyone hits this issue before it is fixed:
If you manually stop PDB before trying to install the upgraded package this issue can be avoided.
If an upgrade was tried while PDB was running and you're in the state where service puppetdb stop can't find and stop PDB, finding the pid, killing PDB, and restarting it should resolve the issue and allow the upgrade to proceed.