PuppetDB uses the current timestamp from the server as the expired and deactivated time. If there is a time sync issue on the PuppetDB server where it is set into the future and nodes are deactivated or expired, that timestamp will be used as the comparison to reactivate the node. Since the check is to see if the timestamp from the report is greater than the deactivated or expired time, nodes cannot be reactivated.
This can lead to a situation where all nodes are expired and will not be reactivated until the future date is hit. We recently encountered this where the date on the server was inadvertently set 1 year into the future and so all nodes were marked as expired during GC. After the time on the server was fixed, the nodes would not reactivate despite putting in new reports.
https://github.com/puppetlabs/puppetdb/blob/master/src/puppetlabs/puppetdb/scf/storage.clj#L1375-L1389 looks to be the query for this, and it would be nice if the deactivated and expired times were compared against NOW() to ensure that they are not in the future.
1. Install a monolithic puppet master
2. Set the time on the server to 1 year in the future
3. Deactivate a node
4. Set the time back to the current time
5. Run puppet on the agent to populate a new report
6. Observe that the node is still deactivated