Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-8338

Puppet apply doesn't stream file content

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • None
    • PUP 6.14.0
    • None
    • None
    • Coremunity
    • Platform Core KANBAN
    • Needs Assessment
    • Bug Fix
    • Hide
      This changes "puppet apply" to retrieve content for file resources with a "source" parameter in the same way that "puppet agent" does. Previously, "puppet apply" read the entire file contents into memory and cached it for the duration of the run. It also did not follow HTTP redirects or handle HTTP errors in the same way, such as HTTP service unavailable (503). Now "puppet apply" uses the same code path as "puppet agent" to stream file content from "puppet://" and "http(s)://" file sources. As a result, memory footprint should be reduced when using "puppet apply".
      Show
      This changes "puppet apply" to retrieve content for file resources with a "source" parameter in the same way that "puppet agent" does. Previously, "puppet apply" read the entire file contents into memory and cached it for the duration of the run. It also did not follow HTTP redirects or handle HTTP errors in the same way, such as HTTP service unavailable (503). Now "puppet apply" uses the same code path as "puppet agent" to stream file content from "puppet://" and "http(s)://" file sources. As a result, memory footprint should be reduced when using "puppet apply".
    • Needs Assessment

    Description

      Puppet Version: Puppet 4 (at least) and up
      Puppet Server Version: n/a
      OS Name/Version: all

      When running puppet apply with a file resource and source parameter, then puppet will load the entire file into memory, instead of streaming it.

      This was noticed while researching why the bug described in PUP-7482 only affected puppet agent but not puppet apply.

      Desired Behavior:

      When managing files with a source parameter, puppet apply should always stream file content from all of the following sources

      <absolute path> local file
      file://<URI-encoded path> local file
      puppet:///modules/mymodule local file
      http?://host:port/path/to/file remote file

      Actual Behavior:

      puppet apply does not stream file content from http-based sources. See https://github.com/puppetlabs/puppet/blob/5.3.3/lib/puppet/type/file/source.rb#L274-L276.

      At first glance it appears the Puppet::Type::File::ParameterSource#content method can be deleted entirely, and should always rely on chunk_file_from_disk or chunk_file_from_source.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. PUP-10367 Puppet apply recursive copy of directory fails in file resource

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. PUP-10415 puppet-apply: Could not set 'file' on ensure: Is a directory @ io_fread

          • Normal - Regular issue, some loss of functionality under specific circumstances.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. PUP-8420 Regression: puppet apply does not decompress response bodies for http file sources

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. PUP-7482 Agent sends wrong content-type when using http file sources

          • Normal - Regular issue, some loss of functionality under specific circumstances.
          • Closed

          Activity

            People

              josh Josh Cooper
              josh Josh Cooper
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support