Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-4502

PuppetDB command size metrics are missing

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PDB 5.2.9, PDB 6.5.0
    • Fix Version/s: PDB 5.2.10, PDB 6.3.5, PDB 6.7.1
    • Component/s: None
    • Labels:
      None
    • Template:
    • Team:
      PuppetDB
    • Method Found:
      Inspection
    • CS Priority:
      Reviewed
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      Fixed a bug that prevented command size metrics from being recorded and the max-command-size config option from working properly. This was done by adding a custom 'X-Uncompressed-Length' header to compressed requests sent from the terminus. The value of this header is the size in bytes of the command before it is compressed. This value is used to update the size metric and to decide when to discard commands based on
      'max-command-size'.
      Show
      Fixed a bug that prevented command size metrics from being recorded and the max-command-size config option from working properly. This was done by adding a custom 'X-Uncompressed-Length' header to compressed requests sent from the terminus. The value of this header is the size in bytes of the command before it is compressed. This value is used to update the size metric and to decide when to discard commands based on 'max-command-size'.
    • QA Risk Assessment:
      Needs Assessment

      Description

      PuppetDB has a global.size message queue metric that provides the distribution of command sizes that have been submitted:

      https://puppet.com/docs/puppetdb/6.5/api/metrics/v1/mbeans.html#message-queue-metrics

      This information is important for determining PuppetDB memory use and detecting changes that have caused an abnormal amount of command data to be submitted. In current configurations, this metric is not receiving any data.

      Reproduction Case

      • Install Puppet, Puppet Server, and PuppetDB 5 on a CentOS 7 system:

      yum install -y http://yum.puppetlabs.com/puppet5-release-el-7.noarch.rpm
      yum install -y puppetserver
       
      source /etc/profile.d/puppet-agent.sh
       
      puppet config set server $(puppet config print certname)
      systemctl start puppetserver
      puppet module install puppetlabs-puppetdb
       
      puppet apply <<EOF
      include puppetdb
      class { 'puppetdb::master::config':
        manage_config =>  true,
        enable_reports => true,
      }
      EOF
      

      • Run the puppet agent to submit data to PuppetDB: puppet agent -t
      • Check the global.size metric:

      curl -sS
      'http://localhost:8080/metrics/v2/read/puppetlabs.puppetdb.mq:name=global.size'
      |python -m json.tool
      

      Outcome

      The metric returns no data:

      curl -sS 'http://localhost:8080/metrics/v2/read/puppetlabs.puppetdb.mq:name=global.size' |python -m json.tool
      {
          "request": {
              "mbean": "puppetlabs.puppetdb.mq:name=global.size",
              "type": "read"
          },
          "status": 200,
          "timestamp": 1567818278,
          "value": {
              "50thPercentile": 0.0,
              "75thPercentile": 0.0,
              "95thPercentile": 0.0,
              "98thPercentile": 0.0,
              "999thPercentile": 0.0,
              "99thPercentile": 0.0,
              "Count": 0,
              "Max": 0,
              "Mean": 0.0,
              "Min": 0,
              "SnapshotSize": 0,
              "StdDev": 0.0
          }
      }
      

      Despite the PuppetDB access log showing that command data was submitted:

      # fgrep /pdb/cmd/ /var/log/puppetlabs/puppetdb/puppetdb-access.log
      10.16.126.201 - - [07/Sep/2019:01:00:11 +0000] "POST
      /pdb/cmd/v1?checksum=03c4cb56d666fdac92d41124d98c4b11c0563d83&version=5&certname=bwoy7ty3hsskdj0.delivery.puppetlabs.net&command=replace_facts&producer-timestamp=2019-09-07T01:00:11.496Z
      HTTP/1.1" 200 53 "-" "Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_222)" 53
      10.16.126.201 - - [07/Sep/2019:01:00:12 +0000] "POST
      /pdb/cmd/v1?checksum=2f074c729b6180641c927fcf5293bf25d69cdce5&version=9&certname=bwoy7ty3hsskdj0.delivery.puppetlabs.net&command=replace_catalog&producer-timestamp=2019-09-07T01:00:12.042Z
      HTTP/1.1" 200 53 "-" "Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_222)" 64
      

      Expected Outcome

      The Count component of the global.size metric is equal to the number of
      requests to the /pdb/cmd API recorded in the access log.

        Attachments

          Activity

            People

            • Assignee:
              zachary.kent Zachary Kent
              Reporter:
              chuck Charlie Sharpsteen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support