Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2942

Very slow agent runs (particularly file resources) after puppet server upgrade from 6 to 7

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: SERVER 7.0.1
    • Fix Version/s: None
    • Component/s: Puppet Server
    • Labels:
      None
    • Template:
      PUP Bug Template
    • Method Found:
      Needs Assessment
    • QA Risk Assessment:
      Needs Assessment

      Description

      Puppet Version: 7.0.1 and 6.x including 6.19.1
      Puppet Server Version: 7.0.1
      OS Name/Version: CentOS 7.9.2009

      We recently upgraded puppet server 7 and puppet db latest via yum.  Immediately, agent runs went from taking ~1 min to >15 min.  We rolled back to the previous RPMs and agent run times went back to normal.

      Later, we performed the same yum update again and see the same drastic increase.  The server usually runs in SELinux enforcing mode with auditd running.  We have disabled both of those but the times remain very high.  Server load and memory use do not seem to be affected.

      Desired Behavior:

      Puppet agent runs with a puppetserver v7 run with similar speeds as with v6.

      Actual Behavior:

      We have run the agent with a few different diagnostic levels:

      • puppet agent -td --profile --summarize
      • puppet agent -td --evaltrace

      Both of these indicate File resources are taking a very long time.  I have seen many online threads and even historical puppet issues that address this but none of the posts helped me ID the root cause and solution.

      We had been using the default environment_timeout (0) in previous versions because we didn't know any better.  After reading some tuning guides this week, we have tested with environment_timeout = unlimited.  This does not seem to help.

      We have disconnected puppetdb from the puppetserver but this did not help.

      $ rpm -qa | grep ^puppet | sort
      puppet-agent-7.1.0-1.el7.x86_64
      puppet-agent-oauth-0.5.1-3.el7.noarch
      puppet-bolt-2.40.2-1.el7.x86_64
      puppetdb-7.0.1-1.el7.noarch
      puppetdb-termini-7.0.1-1.el7.noarch
      puppetserver-7.0.1-1.el7.noarch

       

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            hammondr Ryan Hammond
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Zendesk Support