Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-1631

Import/Export of PuppetDB data is too slow

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PDB 3.1.0
    • Component/s: None
    • Labels:
      None
    • Template:
    • Story Points:
      5
    • Sprint:
      PuppetDB 2015-07-01, PuppetDB 2015-07-15

      Description

      Importing and exporting PuppetDB data is just too slow. Importing/export 5k nodes can take hours.

      We need to get some specifics around performance of our current import/export processes and come up with ways to improve the wall clock time of our current approach. Not sure what the best approach would be here. Probably a good place to start is not going over HTTP for every import/export. Perhaps something that just dumps a tarball locally is the easiest way to improve export perf. We should also be able to import things in process with the commands service that now exists in PuppetDB.

      There are other viable options too, bulk JSON docs, binary formats, multi-threading etc.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              ryan.senior Ryan Senior
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support