Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-3928

Facts blacklist should support providing a fact name regex to blacklist facts matching a pattern

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: PDB 5.1.z, PDB 5.2.6, PDB 6.0.1
    • Component/s: None
    • Labels:
      None
    • Template:
    • Team:
      PuppetDB
    • Sprint:
      Hopper/Triage
    • CS Priority:
      Major
    • CS Frequency:
      3 - 25-50% of Customers
    • CS Severity:
      4 - Major
    • CS Business Value:
      4 - $$$$$
    • CS Impact:
      Hide
      There is a significant performance boost available for customers if they block unneeded facts from being sent for storage. Without the ability to use regexs, it is not reasonable to blacklist all possible permutations of possible names for network and block device facts since they could be in the hundreds depending on installation size. These are most likely not going to be needed by customers in general.
      Show
      There is a significant performance boost available for customers if they block unneeded facts from being sent for storage. Without the ability to use regexs, it is not reasonable to blacklist all possible permutations of possible names for network and block device facts since they could be in the hundreds depending on installation size. These are most likely not going to be needed by customers in general.
    • Release Notes:
      New Feature
    • Release Notes Summary:
      Hide

      Add support for fact blacklist regexes. Omit facts whose name
      completely matches any of the expressions provided. Add a
      "facts-blacklist-type" database configuration option which defaults to
      "literal", producing the existing behavior, but can be set to "regex"
      to indicate that the facts-blacklist items are java patterns.
      Show
      Add support for fact blacklist regexes. Omit facts whose name completely matches any of the expressions provided. Add a "facts-blacklist-type" database configuration option which defaults to "literal", producing the existing behavior, but can be set to "regex" to indicate that the facts-blacklist items are java patterns.
    • QA Risk Assessment:
      Needs Assessment

      Description

      The Problem

      The puppet agent still sends legacy facts to the puppet master and they are forwarded onto PuppetDB. However, there's not many good reasons to store the legacy facts in addition to the modern structured facts.

      Take a look at the list of legacy facts here: https://puppet.com/docs/facter/3.9/core_facts.html

      Notice that alot of them include a network interface or something else unique which is bloating PuppetDB's fact storage full of unique facts. This makes lists of facts for populating things like autocomplete in the console UI larger and slower than they need to be.

      Suggested Improvement

      We already have a facts-blacklist in PuppetDB which could be used to workaround this problem if it allowed for a regex match on fact name so we could exclude facts like `mtu_` or `blockdevice_`

      https://puppet.com/docs/puppetdb/5.1/configure.html#facts-blacklist

      This isn't a full solution, really we should either stop sending legacy facts from the agent to the mater OR stop forwarding legacy facts from the master onto PuppetDB. Since those may have unintended consequences or may just be a breaking change, this improvement allows customers that want to take the initiative a way to remove these facts from being stored in PuppetDB.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  zachary.kent Zachary Kent
                  Reporter:
                  nick.walker Nick Walker
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: