Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-884

Add support for streaming output from external shell processes



    • Type: New Feature
    • Status: Closed
    • Priority: Normal
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Template:
    • Story Points:


      The current support for running external processes from Puppet Server uses a background thread to read buffered data from the STDOUT/STDERR streams, but it buffers all of the data in memory and does not expose it to the caller until the external process completes.

      For Direct Puppet we have some requirements around allowing the user to specify an external script to call to provide file content that should be returned directly in an HTTP response. It's possible that there may be extremely large files involved in this transaction, so we can't get away with buffering the STDOUT results in memory.

      We need to extend our capabilities around calling external shell commands such that we can stream the data from the command in some way. In all likelihood, this just means that we need to provide a way to pass in an OutputStream that we use to read the data into; in cases where we want to stream the data we can either use an OutputStream with a fixed buffer size (so that we know the maximum number of bytes that we'll read into memory at a given time), or a PipedOutputStream, or perhaps a combination of the two.

      There will probably be a bit of trickiness involved in this just due to the fact that our current API for this in Puppet Server is synchronous (blocks until the system command exits), and this change may force us to switch to an asynchronous model, which may require us to refactor some places where we currently call that code from, e.g., Puppet::Util::ExecutionStub.


          Issue Links



              Unassigned Unassigned
              chris Chris Price
              QA Contact:
              Erik Dasher Erik Dasher
              0 Vote for this issue
              1 Start watching this issue



                  Zendesk Support