I am considering deploying Puppet to an unreliable network, and it is important that we have some desired state management occurring on the nodes even when the Puppet master is down or uncontactable.
I have been testing Puppet's cached catalog to see if that would be a solution, the idea being it would execute the cached catalog when the agent fails to connect to the server. However, rather than using the cached catalog, it just borks out and refuses to do anything:
Error: Could not retrieve catalog from remote server: Connection refused - connect(2)
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Connection refused - connect(2)
Yes, there is a config option called use_cached_catalog, which when set to true will cause the Agent to only use the cached catalog and never update its configuration from the master, which is definitely not what I want - as the whole point of this kind of system is we should be able to push new configurations to the nodes when they are eventually able to check in with the master.
There is also another option, usecacheonfailure, however I have never been able to successfully get the Agent to use the cached catalog on any kind of failure - catalog compile failure or connect failure.