Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2026

Serve Files from Clojure

    XMLWordPrintable

Details

    • Epic
    • Status: Closed
    • Normal
    • Resolution: Won't Do
    • None
    • None
    • None
    • Clojure File Serving
    • Froyo
    • Platform Core Grooming
    • To Do
    • Major
    • 5 - >90% of Customers
    • 3 - Serious
    • 3 - $$$$
    • Hide
      File serving is one of the areas we see impacting performance. This really shows up when you are adding a lot of new nodes and they all have to pluginsync a large number of modules. This can sometime take 5 minutes per node for just the DSC module. This takes up all the available jrubies. By doing this in clojure it would likely perform much better.

      By moving file serving away from using jrubies it might also give you a sort of control on how much performance will be used for catalog compilation rather than file serving.
      Show
      File serving is one of the areas we see impacting performance. This really shows up when you are adding a lot of new nodes and they all have to pluginsync a large number of modules. This can sometime take 5 minutes per node for just the DSC module. This takes up all the available jrubies. By doing this in clojure it would likely perform much better. By moving file serving away from using jrubies it might also give you a sort of control on how much performance will be used for catalog compilation rather than file serving.
    • 23,000
    • Needs Assessment

    Description

      Serving files from clojure rather than ruby should be significantly faster and more scalable. In preliminary tests there was roughly a 70% reduction in pluginsyncing the windows module, dramatically reducing first run times. This will also allow us to stream file content, significantly reducing the https://tickets.puppetlabs.com/projects/SERVER/issues/SERVER-2026?filter=allopenissues#memory footprint for large files and allowing the serving of larger files.

      To serve files from modules (including plugins, custom facts, tasks, and locales) clojure must be able to determine where on disk those modules are (global and environment configuration), manage the lifecycle of that information (environment info caching) and safely access them (file system locking that works with code deployment workflows).

      Serving files from custom mount-points hopefully will follow the existing example of authorization, moving to a configuration file based within the Server's conf.d.

      Implementing a bug-for-bug port would not be significantly less work and the team, owning both agent and server, can make upgrades to the wire format transparent to the user. Existing, supported, workflows for file serving and code deployment with r10k and file-sync should continue to work as-is. Upgrading the configuration file for custom mount-points should be mechanical and automatable.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              justin Justin Stoller
              Votes:
              1 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support