Puppet Version: 7.0.1 and 6.x including 6.19.1
Puppet Server Version: 7.0.1
OS Name/Version: CentOS 7.9.2009
We recently upgraded puppet server 7 and puppet db latest via yum. Immediately, agent runs went from taking ~1 min to >15 min. We rolled back to the previous RPMs and agent run times went back to normal.
Later, we performed the same yum update again and see the same drastic increase. The server usually runs in SELinux enforcing mode with auditd running. We have disabled both of those but the times remain very high. Server load and memory use do not seem to be affected.
Puppet agent runs with a puppetserver v7 run with similar speeds as with v6.
We have run the agent with a few different diagnostic levels:
- puppet agent -td --profile --summarize
- puppet agent -td --evaltrace
Both of these indicate File resources are taking a very long time. I have seen many online threads and even historical puppet issues that address this but none of the posts helped me ID the root cause and solution.
We had been using the default environment_timeout (0) in previous versions because we didn't know any better. After reading some tuning guides this week, we have tested with environment_timeout = unlimited. This does not seem to help.
We have disconnected puppetdb from the puppetserver but this did not help.