Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-1860

Improve Performance of File Serving in PuppetServer

    XMLWordPrintable

Details

    • Epic
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • Improve Performance of File Serving in PuppetServer
    • Platform Core
    • Done
    • Major
    • 5 - >90% of Customers
    • 3 - Serious
    • 3 - $$$$
    • Hide
      File serving is one of the areas we see impacting performance. This really shows up when you are adding a lot of new nodes and they all have to pluginsync a large number of modules. This can sometime take 5 minutes per node for just the DSC module. This takes up all the available jrubies. By doing this in clojure it would likely perform much better.

      By moving file serving away from using jrubies it might also give you a sort of control on how much performance will be used for catalog compilation rather than file serving.
      Show
      File serving is one of the areas we see impacting performance. This really shows up when you are adding a lot of new nodes and they all have to pluginsync a large number of modules. This can sometime take 5 minutes per node for just the DSC module. This takes up all the available jrubies. By doing this in clojure it would likely perform much better. By moving file serving away from using jrubies it might also give you a sort of control on how much performance will be used for catalog compilation rather than file serving.

    Description

      The Problem

      Tuning the number of JRubies required for PuppetServer is very difficult because serving files require borrowing a JRuby but adding extra JRubies to handle serving more files requires more heap and puppet code will be loaded into those JRubies.

      As a result you need more heap and you get more garbage created and collected because there is no way to separate the resources for file serving and catalog compilation.

      Also generally speaking, the current file serving implementation is not optimal because files are loaded into memory and then served

      In Scope

      1. Investigate which file serving functions would yield the most benefit from being in pure clojure instead of delegating to puppet / jruby

        • starting point is pluginsync
        • then files resources with a source attribute not stored in a module that is file synced

      2. Rewrite the file serving for these endpoints in Clojure so they do not require borrowing a jruby

      Possible Benefits

      Either solution will lower heap requirements and improve the stability of Puppet Server by segregating the resource needs.

      For small installs of PE we'll be able to more effectively run with lower resources.

      For large installs of PE we'll be able to drastically lower the heap requirements.

      Improve file serving from its current implementation to a more generic file-server like approach, hopefully reducing/removing the need to load full files into memory as opposed to streaming them off disk

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              nick.walker Nick Walker
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support