Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-1631

Import/Export of PuppetDB data is too slow

    Details

    • Type: Task
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PDB 3.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Importing and exporting PuppetDB data is just too slow. Importing/export 5k nodes can take hours.

      We need to get some specifics around performance of our current import/export processes and come up with ways to improve the wall clock time of our current approach. Not sure what the best approach would be here. Probably a good place to start is not going over HTTP for every import/export. Perhaps something that just dumps a tarball locally is the easiest way to improve export perf. We should also be able to import things in process with the commands service that now exists in PuppetDB.

      There are other viable options too, bulk JSON docs, binary formats, multi-threading etc.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  ryan.senior Ryan Senior
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: