Details

    • Template:
    • Team:
      Bolt
    • Sprint:
      Bolt Kanban
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Log rotation under systemd will now correctly signal pxp-agent to re-open the log file.
    • QA Risk Assessment:
      Needs Assessment

      Description

      The pxp-agent logrotate script for systemd never sends the USR2 signal after rotating the logs.

      Tested by running logrotate -f /etc/logrotate.d/pxp-agent. This results in a compressed log under /var/log/puppetlabs/pxp-agent, but no new logs. Explicitly sending the USR2 signal restarts logging.

      Reproduction Case

      • Install puppet-agent 5.4.0 on a CentOS 7 VM:

      rpm -Uvh http://yum.puppetlabs.com/puppet5/puppet-release-el-7.noarch.rpm
      yum install -y puppet-agent
      

      • Configure pxp-agent so that it can be started.

      # Bootstrap a CA and keypair
      /opt/puppetlabs/bin/puppet cert generate $(hostname -f)
       
      cat <<EOF > /etc/puppetlabs/pxp-agent/pxp-agent.conf
      {
          "broker-ws-uris": [
              "wss://broker.invalid:8142/pcp2/"
          ],
          "loglevel": "info",
          "master-uris": [
              "https://broker.invalid:8140"
          ],
          "pcp-version": "2",
          "ssl-ca-cert": "$(/opt/puppetlabs/bin/puppet config print localcacert)",
          "ssl-cert": "$(/opt/puppetlabs/bin/puppet config print hostcert)",
          "ssl-key": "$(/opt/puppetlabs/bin/puppet config print hostprivkey)"
      }
      EOF
      

      • Start the pxp-agent service: systemctl start pxp-agent. It will log connection failures, but that's fine. Has the bonus effect of putting a steady stream of traffic into the logs.
      • Trigger the logrotate configuration for pxp-agent laid down by the puppet-agent package: logrotate -v -f /etc/logrotate.d/pxp-agent

      Outcome

      Logrotation works:

      # logrotate -v -f /etc/logrotate.d/pxp-agent
       
      reading config file /etc/logrotate.d/pxp-agent
       
      Handling 1 logs
       
      rotating pattern: /var/log/puppetlabs/pxp-agent/*.log  forced from command line (30 rotations)
      empty log files are not rotated, old logs are removed
      considering log /var/log/puppetlabs/pxp-agent/pxp-agent.log
        log needs rotating
      rotating log /var/log/puppetlabs/pxp-agent/pxp-agent.log, log->rotateCount is 30
      dateext suffix '-20180312'
      glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.30.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.31.gz (rotatecount 30, logstart 1, i 30),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.30.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.29.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.30.gz (rotatecount 30, logstart 1, i 29),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.29.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.28.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.29.gz (rotatecount 30, logstart 1, i 28),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.28.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.27.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.28.gz (rotatecount 30, logstart 1, i 27),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.27.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.26.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.27.gz (rotatecount 30, logstart 1, i 26),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.26.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.25.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.26.gz (rotatecount 30, logstart 1, i 25),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.25.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.24.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.25.gz (rotatecount 30, logstart 1, i 24),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.24.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.23.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.24.gz (rotatecount 30, logstart 1, i 23),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.23.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.22.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.23.gz (rotatecount 30, logstart 1, i 22),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.22.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.21.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.22.gz (rotatecount 30, logstart 1, i 21),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.21.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.20.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.21.gz (rotatecount 30, logstart 1, i 20),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.20.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.19.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.20.gz (rotatecount 30, logstart 1, i 19),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.19.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.18.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.19.gz (rotatecount 30, logstart 1, i 18),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.18.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.17.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.18.gz (rotatecount 30, logstart 1, i 17),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.17.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.16.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.17.gz (rotatecount 30, logstart 1, i 16),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.16.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.15.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.16.gz (rotatecount 30, logstart 1, i 15),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.15.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.14.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.15.gz (rotatecount 30, logstart 1, i 14),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.14.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.13.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.14.gz (rotatecount 30, logstart 1, i 13),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.13.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.12.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.13.gz (rotatecount 30, logstart 1, i 12),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.12.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.11.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.12.gz (rotatecount 30, logstart 1, i 11),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.11.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.10.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.11.gz (rotatecount 30, logstart 1, i 10),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.10.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.9.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.10.gz (rotatecount 30, logstart 1, i 9),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.9.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.8.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.9.gz (rotatecount 30, logstart 1, i 8),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.8.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.7.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.8.gz (rotatecount 30, logstart 1, i 7),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.7.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.6.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.7.gz (rotatecount 30, logstart 1, i 6),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.6.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.5.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.6.gz (rotatecount 30, logstart 1, i 5),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.5.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.4.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.5.gz (rotatecount 30, logstart 1, i 4),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.4.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.3.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.4.gz (rotatecount 30, logstart 1, i 3),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.3.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.2.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.3.gz (rotatecount 30, logstart 1, i 2),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.2.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.1.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.2.gz (rotatecount 30, logstart 1, i 1),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.1.gz does not exist
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log.0.gz to /var/log/puppetlabs/pxp-agent/pxp-agent.log.1.gz (rotatecount 30, logstart 1, i 0),
      old log /var/log/puppetlabs/pxp-agent/pxp-agent.log.0.gz does not exist
      log /var/log/puppetlabs/pxp-agent/pxp-agent.log.31.gz doesn't exist -- won't try to dispose of it
      renaming /var/log/puppetlabs/pxp-agent/pxp-agent.log to /var/log/puppetlabs/pxp-agent/pxp-agent.log.1
      running postrotate script
      compressing log with: /bin/gzip
      

      But, the pxp-agent daemon never receives a SIGUSR2 and is left writing to a rotated filename that was removed by gzip:

      # ls -l /proc/$(systemctl show -p MainPID pxp-agent|cut -d= -f2)/fd
       
      total 0
      lr-x------ 1 root root 64 Mar 12 17:55 0 -> /dev/null
      lrwx------ 1 root root 64 Mar 12 17:55 1 -> socket:[22627]
      lrwx------ 1 root root 64 Mar 12 17:55 2 -> socket:[22627]
      l-wx------ 1 root root 64 Mar 12 17:55 3 -> /var/log/puppetlabs/pxp-agent/pxp-agent.log.1 (deleted)
      lrwx------ 1 root root 64 Mar 12 17:55 4 -> anon_inode:[eventfd]
      lrwx------ 1 root root 64 Mar 12 17:55 5 -> anon_inode:[eventpoll]
      lrwx------ 1 root root 64 Mar 12 17:55 6 -> anon_inode:[timerfd]
      

      Expected Outcome

      The daemon should be writing to /var/log/puppetlabs/pxp-agent/pxp-agent.log after rotation is triggered:

      # ls -l /proc/$(systemctl show -p MainPID pxp-agent|cut -d= -f2)/fd
       
      total 0
      lr-x------. 1 root root 64 Mar 12 17:17 0 -> /dev/null
      lrwx------. 1 root root 64 Mar 12 17:17 1 -> socket:[666596]
      lrwx------. 1 root root 64 Mar 12 17:17 2 -> socket:[666596]
      l-wx------. 1 root root 64 Mar 12 17:17 3 -> /var/log/puppetlabs/pxp-agent/pxp-agent.log
      lrwx------. 1 root root 64 Mar 12 17:17 4 -> anon_inode:[eventfd]
      lrwx------. 1 root root 64 Mar 12 17:17 5 -> anon_inode:[eventpoll]
      lrwx------. 1 root root 64 Mar 12 17:17 6 -> anon_inode:[timerfd]
      lrwx------. 1 root root 64 Mar 12 17:17 7 -> socket:[666604]
      

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  michael.smith Michael Smith
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Zendesk Support