Inline file metadata and retrieve content using content_uri

*Updated*

This ticket was repurposed to handle reworking how the compiler inlines metadata and how the agent uses that information to retrieve versioned file content. This ticket does the following:

• Inlines file metadata into the catalog in a new metadata section.
• Inlines recursive file metadata into the catalog in a new recursive_metadata section.
• Deletes the content_uri file parameter and moves it into the FileMetadata object where it belongs. This isn't a breaking change, because we never released a version of puppet with content_uri as a file parameter.
• Ensures source is set on metadata objects that the fileserver returns in the metadata section (but not the recursive_metadata section). This is necessary so that we know which source file_metadata came from, in the situation you have multiple sources, and puppet selects the first non-nil one.
• Modifies the agent to look in the catalog when resolving metadata. If no metadata exists for a file resource, it falls back to its previous behavior, requesting data from the server.
• If the agent needs to retrieve file content, and the file resource has metadata containing a content_uri parameter, then it uses it to download file content.

*Original*

Modify the catalog building process to inline content_uri for files. It should be of the form:

content_uri => 'puppet:///modules/ntp/files/ntp.conf'

Note the path is rooted in the per-environment directory, e.g. /etc/puppetlabs/code/environments/production. By default, content_uri should not specify a host and port in the URI, e.g. puppet:///modules/.... However, if source => 'puppet://<host>:<port>', then those values should be preserved in the content_uri.

Modify the catalog application process to retrieve content for file resources using the content_uri parameter instead of source. If the content_uri parameter is absent (for example because source refers to a local file or custom mount point), then the agent should fallback to using source and retrieve latest file content like it does now.

The agent should send the REST request of the form:

https://server:port/puppet/v3/static_file_content/<path>?code_id=<code_id>

where <path> is that path of the file to download, e.g. modules/ntp/files/ntpd.conf, and code_id is obtained from the catalog. Also note the endpoint is called static_file_content, a recent development, and it's in the v3 namespace.

The agent may send environment=production as a query param due to the indirector, but it is not necessary since that information will be contained in code_id.

The agent should stream the response back, reading chunks of the response body like we do for file_content: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type/file/content.rb#L216-L221. In other words, we don't want to read the entire file into a memory.

The agent should use DNS SRV records like it does for file_content requests. This is accomplished by following the file_content example above.

Also add rspec tests under spec/integration simulating the server responding with valid content, and with HTTP 404. The latter case should cause the resource to fail, and should be output in the log.

Also update API documentation showing expected request/response. Since this will only be a puppetserver REST endpoint, it probably makes sense to add it there. See https://github.com/puppetlabs/puppet/pull/4586/files#r50486882 /cc kevin.corcoran

The compiler should only line content_uri if code_id is non-nil/empty. If content_uri is present, but code_id is not, then that's a bug and we should probably fail the run.