PDB-3911, a migration to the reports table will (hopefully) land in Kearney that changes how we generate the report hash (and therefore changes all the report hashes currently in PuppetDB). PuppetDB's HA sync uses the report hashes to determine what is different between the two PuppetDB's. This means that if one PuppetDB is left online (running the old code), and the other PuppetDB is shutdown and upgraded, it will perform the migration, change all the report hashes and then attempt to perform an initial sync with the old PuppetDB. Since every report hash has changed, it will attempt to sync ever report again. Once it completes that sync, it'll be able to start up. Then the next periodic sync from the old PuppetDB will do the same thing, but in reverse.
The initial idea of disabling sync between these two versions of PuppetDB checking the schema_migration number seems to be inadequate to prevent the old PuppetDB from syncing with the new PuppetDB.
Outstanding questions (that are more specific than the general, how does the installer upgrade an HA deployment of PuppetDB?)
- Do both PuppetDB's get shutdown during an upgrade at the same time? Or are they entirely separate?
- Can we use the installer to disable HA sync between the two PuppetDB's before we begin upgrading and then have the installer re-enable sync once both PuppetDB's are upgraded?