Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-5260

Improve PQL parse performance with high number of filter clauses

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • None
    • PDB 6.21.0, PDB 7.10.1
    • PuppetDB
    • None
    • Bug Fix
    • Hide
      PuppetDB should require much less time and memory when parsing some PQL queries, , for example queries including many or clauses like `nodes {x or y or ...}`. Previously 5000 clauses could not be parsed with an 8GB heap, and much smaller queries still required exorbitant amounts of memory and CPU time.
      Show
      PuppetDB should require much less time and memory when parsing some PQL queries, , for example queries including many or clauses like `nodes {x or y or ...}`. Previously 5000 clauses could not be parsed with an 8GB heap, and much smaller queries still required exorbitant amounts of memory and CPU time.
    • Needs Assessment

    Description

      We've discovered that PQL parsing (via instaparse) can be extremely slow (say 20s for a pql query with ~1000 filter clauses). A brief investigation suggests that this may be a known issue with instaparse.

      It's not yet clear whether we might be able to address this by rearranging our grammar somehow, or we might want to rewrite the parser (either via manual recursive descent, or via something like antlr, flex/bison, etc.

      Attachments

        Issue Links

          Activity

            People

              rob.browning Rob Browning
              rob.browning Rob Browning
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support