Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-3647

Puppet stores files forever in state.yaml

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PUP 5.5.7, PUP 6.0.1
    • Component/s: None
    • Labels:
      None
    • Template:
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Customer Feedback
    • CS Priority:
      Major
    • CS Frequency:
      3 - 25-50% of Customers
    • CS Severity:
      3 - Serious
    • CS Business Value:
      4 - $$$$$
    • CS Impact:
      Many users would like to use recurse. Having it lead to poor performance isn't good.
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      Previously, the state.yaml file could grow unbounded. The new `statettl` setting controls how long entries are cached (default: 32 days). If you use resource schedules, please see the `statettl` documentation to see how this setting interacts with the schedule type.
      Show
      Previously, the state.yaml file could grow unbounded. The new `statettl` setting controls how long entries are cached (default: 32 days). If you use resource schedules, please see the `statettl` documentation to see how this setting interacts with the schedule type.

      Description

      We noticed the following today on one of our production clusters: https://cloudup.com/cgJm3eD4wu4. Essentially Puppet Agent has been using gradually more CPU and Memory over time and was taking longer and longer to execute. On some nodes its been using 2G during each run.

      Digging in, we discovered why. We have configuration like the following:

      file {
      	"/var/db/":
      		ensure => directory,
      		group => "dbuser",
      		owner => "dbuser",
      		recurse => true;
      }
      

      It turns out that the /var/db directory on this machine sometimes includes temporary files, because of the data system using this directory. This means there are a few new files in /var/db every time we run puppet agent.

      We discovered today that each time puppet sees a file, it writes a line to /var/lib/puppet/state/state.yaml. Since there were always new files each time puppet ran, this file grew and grew. Its about 125MB on some of our machines. With a state.yaml file of that size, puppet agent took this long:

      real 6m31.792s
      user 5m56.434s
      sys 0m33.867s

      After deleting /var/lib/puppet/state/state.yaml it dropped to:

      real 0m25.232s
      user 0m16.073s
      sys 0m4.112s

      We've removed our use of `recurse` for now, which wasn't really necessary in this case, but we consider this a dangerous situation and a bug in Puppet; it means that we need to be sure Puppet isn't touching new files over time. Happy to provide more details if its helpful.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  jacob.helwig Jacob Helwig
                  Reporter:
                  pbbakkum Peter Bakkum
                • Votes:
                  4 Vote for this issue
                  Watchers:
                  18 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: