Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-8014

Create a setting to clear short-lived puppet environments from memory

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PUP 6.19.0
    • Component/s: None
    • Labels:
    • Template:
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • CS Priority:
      Major
    • CS Frequency:
      5 - >90% of Customers
    • CS Severity:
      3 - Serious
    • CS Business Value:
      4 - $$$$$
    • CS Impact:
      There is no workaround that only gets rid of short lived environments. Being able to do so would help customers keep the environments they need while not being burdened by the environments they don't need sticking around.
    • Release Notes:
      New Feature
    • Release Notes Summary:
      Hide
      Adds a new `environment_timeout_mode` setting which makes it possible to clear short-lived puppet environments from memory, while caching frequently used environments indefinitely. Doing so will reduce Puppet server's memory footprint.

      If `environment_timeout` is 0 or unlimited, then there is no change in behavior. Otherwise, if the `environment_timeout` is a duration, such as 3 minutes, then the environment will be evicted 3 minutes after it is created. However, if `environment_timeout_mode` is set to `from_last_used`, then the environment will be evicted 3 minutes from when it was last used. We recommend either:

      Setting `environment_timeout` to unlimited and explicitly refreshing your Puppet server as part of your code deployment process.

      Or setting `environment_timeout` to a number that will keep your most actively used environments cached, but allow testing environments to fall out of the cache and reduce memory usage. A value of 3 minutes (3m) is a reasonable value. This option requires setting `environment_timeout_mode` to `from_last_used`.

      Once you set `environment_timeout` to a non-zero value, you need to tell Puppet server to read new code from disk using the `environment-cache` API endpoint after you deploy new code. See the docs for the Puppet Server [administrative API](https://puppet.com/docs/puppetserver/latest/admin-api/v1/environment-cache.html).
      Show
      Adds a new `environment_timeout_mode` setting which makes it possible to clear short-lived puppet environments from memory, while caching frequently used environments indefinitely. Doing so will reduce Puppet server's memory footprint. If `environment_timeout` is 0 or unlimited, then there is no change in behavior. Otherwise, if the `environment_timeout` is a duration, such as 3 minutes, then the environment will be evicted 3 minutes after it is created. However, if `environment_timeout_mode` is set to `from_last_used`, then the environment will be evicted 3 minutes from when it was last used. We recommend either: Setting `environment_timeout` to unlimited and explicitly refreshing your Puppet server as part of your code deployment process. Or setting `environment_timeout` to a number that will keep your most actively used environments cached, but allow testing environments to fall out of the cache and reduce memory usage. A value of 3 minutes (3m) is a reasonable value. This option requires setting `environment_timeout_mode` to `from_last_used`. Once you set `environment_timeout` to a non-zero value, you need to tell Puppet server to read new code from disk using the `environment-cache` API endpoint after you deploy new code. See the docs for the Puppet Server [administrative API]( https://puppet.com/docs/puppetserver/latest/admin-api/v1/environment-cache.html ).
    • QA Risk Assessment:
      Needs Assessment

      Description

      The Problem

      When running puppet server you have two levers to adjust how short-lived Puppet environments are cleared from memory.

      1. max-requests-per-instance in puppetserver.conf
      2. environment_timeout in puppet.conf/environment.conf

      Each option has it's own cons, because neither addresses the core problem we want to tackle. We want to clear Puppet environments from memory when they haven't been used for a while and there's no direct way to do that.

      With max-requests-per-instance we're destroying a whole JRuby which takes out all of the Puppet environments that were ever loaded into it.

      With environment_timeout we have to clear environments after a certain amount of time even if that environment is getting lots of catalogs compiled against it right now.

      The solution

      Create a setting that allows for expiring a Puppet environment after it has not been compiled against for a set period of time. If we were to call this setting environment_ttl and we set it to 30m then that would mean that if you ran an agent against your test environment and then then no other agents checked into that environment for 30m then the test environment would be marked expired.

      This gets to the bottom of the problem customers see when deploying short-lived Puppet environments.

      This setting would be a global setting not a per-environment setting.

      We decided to implement a new environment_timeout_mode setting that controls how environment_timeout behaves when its value is greater than 0 and less than unlimited. In 6.x, environments will be expired relative to when they are created. If environment_timeout_mode equals from_last_used, environments will be expired relative to when they were last used. The environment_timeout_mode setting affects values of environment_timeout set globally in puppet.conf and per-environment $codedir/environments/$environment/environment.conf.

      Caveats

      The setting would presumably interact with the environment_timeout setting. This setting could only be applied when the environment_timeout is not 0. If environment_timeout is 0 then this setting would emit a warning and do nothing.

      When the environment_timeout < environment_ttl then environment_timeout would take precedence because it would mark the environment expired before the environment_ttl code had a chance to run.

      When environment_timeout > environment_ttl then this is the primary use case for the setting and environment_ttl finally has a chance to help expire environments more quickly/efficiently than our current settings allow.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              josh Josh Cooper
              Reporter:
              nick.walker Nick Walker
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support