Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2766

Constant increase of catalog compilation time

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Normal
    • Resolution: Fixed
    • SERVER 6.3.1, SERVER 6.7.1, SERVER 6.9.2
    • SERVER 6.12.0
    • Puppet Server
    • None
    • PUP Bug Template
    • Froyo
    • Needs Assessment
    • Needs Assessment

    Description

      Puppet Version: 6.14.0
      Puppet Server Version: 6.9.1
      OS Name/Version: Ubuntu 16.04

      After an unspecific amount of runs the compilation time increases by 50-600%.

      Describe steps to reproduce:

      Execute a `puppet agent -t --noop` with a node concurrency of 15-40 for 2000 nodes.

      Desired Behavior:

      Compilation time should not constantly increase. It should stay nearly the same.

      Actual Behavior:

      Compilation time increases constantly and stays at a high value.

      Since quite a while we face a constant increase of the catalog compilation time with the puppetserver. After an unspecific amount of runs the compilation time increases by 50-600%. In the past the fixed this with your documenation meantioning to increase the `ReservedCodeCacheSize`. This did not completly solve the issue, but did ensure we did not need to restart the puppetserver regularly.

      After upgrading to puppetserver 6.9.1 this issue happens again very often, which forces us to restart the puppetserver often (1-2 times a day). But we cannot increase it futher because the we get an error telling us it can not be higher than 2GB.

      Examples:

      • compilation times of a node with many resources and puppetdb_query():

      097.50 seconds - restarted puppetserver
      106.81 seconds - many values at 100 seconds +/- 15 for the next 2 hours
      346.23 seconds - started a parallel (15) `puppet agent --noop` run
      311.15 seconds - many values around 300 seconds +/- 30 for the next 2 hours
      130.63 seconds - restarted the puppetserver
      101.60 seconds - many values at 100 seconds +/- 15 for the next 2 hours
      399.32 seconds - started a parallel (15) `puppet agent --noop` run
      545.80 seconds - stopped the parallel run
      554.95 seconds -
      613.68 seconds - puppet agent run (cron) times out (600 seconds limit)
      094.76 seconds - after restarted puppetserver

      • compilation times of a node with gentle amount of resources:

      8.67 seconds - restarted puppetserver
      8.79 seconds - many values at 8 seconds +/- 1
      22.33 seconds - started a parallel (15) `puppet agent --noop` run
      7.94 seconds - restarted puppetserver
      12.66 seconds - started a parallel (15) `puppet agent --noop` run
      13.91 seconds -
      13.26 seconds - stopped the parallel runs
      8.44 seconds - restarted puppetserver

      Setup:

      We run about 2.500 nodes served by two setups of puppetserver and puppetdb (each production and development has one). The production puppetserver serves only the production environment with about 175 modules. The development puppetserver serves 100+ environments also with about 175 modules.

      Settings:

      production puppetserver (Ubuntu Xenial, memory: 48GB, CPU: 32, Virtuzzo 6 VE):

      • max-active-instances: 32
      • max-requests-per-instance: 0
      • max-queued-requests: 64
      • max-retry-delay: 60
      • environment-class-cache-enabled: false
      • JAVA_ARGS: "-Xms20480m -Xmx20480m -XX:+UseG1GC -XX:ReservedCodeCacheSize=2048m"

      development puppetserver (Ubuntu Xenial, memory: 18GB, CPU: 12, Virtuzzo 6 VE):

      • max-active-instances: 12
      • max-requests-per-instance: 0
      • max-queued-requests: 24
      • max-retry-delay: 60
      • environment-class-cache-enabled: false
      • JAVA_ARGS: "-Xms7680m -Xmx7680m -XX:+UseG1GC -XX:ReservedCodeCacheSize=1024m"

      Java (same on all nodes):
      ~ $ java -version
      openjdk version "11.0.5" 2019-10-15
      OpenJDK Runtime Environment (build 11.0.5+10-post-Ubuntu-2ubuntu116.04)
      OpenJDK 64-Bit Server VM (build 11.0.5+10-post-Ubuntu-2ubuntu116.04, mixed mode, sharing)

      Attachments

        Activity

          People

            Unassigned Unassigned
            crazymind1337 Mike Fröhner
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Zendesk Support