Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-366

improve logging config to limit max disk usage

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: SERVER 1.2.0, SERVER 2.6.0
    • Component/s: None
    • Labels:
      None
    • Template:
    • Sub-team:
    • Story Points:
      5
    • Sprint:
      Server Emerald 2016-08-24, Server Emerald 2016-09-07
    • CS Priority:
      Reviewed
    • Release Notes:
      New Feature
    • Release Notes Summary:
      Hide
      New Feature:
      Puppet Server 2.6.0 introduces changes to how log files are managed. Previously, logs would be backed up and compressed once a day through logrotate. Log rotation is now managed by logback. Improvements:
      * Logs are now backed up and compressed after exceeding 10MB
      * If the total size of all the logs exceeds 1GB, the oldest logs will be deleted

      Known Issue:
      Upgrade note for Debian users: tl;dr: You must manually remove /etc/logrotate.d/puppetserver to stop logrotate from trying to (harmlessly) manage your Puppet Server log files
      Show
      New Feature: Puppet Server 2.6.0 introduces changes to how log files are managed. Previously, logs would be backed up and compressed once a day through logrotate. Log rotation is now managed by logback. Improvements: * Logs are now backed up and compressed after exceeding 10MB * If the total size of all the logs exceeds 1GB, the oldest logs will be deleted Known Issue: Upgrade note for Debian users: tl;dr: You must manually remove /etc/logrotate.d/puppetserver to stop logrotate from trying to (harmlessly) manage your Puppet Server log files

      Description

      This is related to PE-8008.

      We've had some issues with disk usage going crazy when Puppet Server runs out of file handles; it starts logging tons and tons of exceptions about the issue. (Often the solution is simply to increase the ulimit, but users are still frustrated by the disk getting filled up by log files.)

      `logrotate` has been our go-to tool to try to solve this problem so far, but we've found that when the file handle errors start occuring, they come in much, much faster than logrotate manages them.

      We discovered that logback has support for what we want:

      http://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedFNATP

      If we ship a reasonable default logback configuration based on those features, we should be able to keep the log files from growing completely out of control.

      We'll need to experiment with the config to find one that we think makes sense, make sure that this isn't a big performance hit, and make sure we understand how it would interact with logrotate. We may need to modify ezbake such that specific projects can opt out of logrotate if it turns out that it's better to just have logback handle it.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              qa qa
              Reporter:
              chris Chris Price
              QA Contact:
              Erik Dasher
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support