[PUP-5119] Handle file sources outside per-environment codedir Created: 2015/08/30  Updated: 2016/04/27  Resolved: 2016/02/10

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: PUP 4.4.0

Type: New Feature Priority: Normal
Reporter: Josh Cooper Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PUP-5814 Handle symlinks that point out of the... Resolved
Template:
Epic Link: (Burnside) Direct Puppet: Client Static Catalog
Story Points: 3
Sprint: Client 2016-02-10
Release Notes: Not Needed

 Description   

When inlining file metadata, the compiler should skip file sources that are outside codedir, e.g. custom mount points. The agent should continue to retrieve latest metadata and content using the existing file_metadata and file_content endpoints without any changes to the agent.



 Comments   
Comment by Josh Cooper [ 2016/01/27 ]

Removed blocking relationship because we only need to resolve the file source and determine if it can be inlined or not. The actual inlining of checksums will be handled later.

Comment by Michael Smith [ 2016/01/27 ]

Prior discussion on this preferred not collecting the metadata on a file outside the codedir during static compilation. We'd like to preserve giving errors for absent files during static compilation as well.

Comment by Michael Smith [ 2016/01/27 ]

Since it's not clear what it means to "not skip" a file, this seems like it should define a contract for what it means to detect a file is outside the codedir and should be skipped, with skipping to be implemented in conjuction with PUP-5117.

Alternatively, the first actor can add iterating over the list of resources to inline (when static compilation is enabled), and other tickets can make use of that.

Comment by Branan Riley [ 2016/01/27 ]

My plan for this is pretty similar what we talked about at the meeting yesterday - pass along a statically_compiling option to the file_metadata terminus, and when that option is present we can skip the stat call and flag the returned metadata as not inlinable.

There's some weird additional logic when we're following symlinks that might make this gross, but I don't think that pushes this work far enough outside the 3-point range to warrant splitting it into another ticket

Comment by Michael Smith [ 2016/02/10 ]

In the end, passing an option to the file_metadata was going to be complicated. Until we have measurements that stat-ing extra files is a problem, we've gone with the simpler approach of just filtering out files after we've resolved metadata.

Comment by Michael Smith [ 2016/02/10 ]

FR Notes:

Created a codedir:

$ tree
.
├── environments
│   └── production
│       ├── manifests
│       │   └── test.pp
│       └── modules
│           └── a_module
│               └── files
│                   └── cupcake
└── modules
    └── a_core_module
        └── files
            └── foo
 
9 directories, 3 files
 
$ cat environments/production/manifests/test.pp
file { '/tmp/foo':
  source => 'puppet:///modules/a_core_module/foo',
  ensure => file,
}
 
file { '/tmp/foo2':
  source => 'file:///Users/michaelsmith/tmp/List.h',
  ensure => file,
}
 
file { '/tmp/foo3':
  source => 'puppet:///modules/a_module/cupcake',
  ensure => file,
}

With static_catalogs enabled, ran puppet agent. The catalog received by the agent is

{
    "catalog_uuid": "499264a0-503b-47d0-8551-3e7b6dd3f068",
    "classes": [
        "settings"
    ],
    "code_id": null,
    "edges": [
        {
            "source": "Stage[main]",
            "target": "Class[Settings]"
        },
        {
            "source": "Stage[main]",
            "target": "Class[Main]"
        },
        {
            "source": "Class[Main]",
            "target": "File[/tmp/foo]"
        },
        {
            "source": "Class[Main]",
            "target": "File[/tmp/foo2]"
        },
        {
            "source": "Class[Main]",
            "target": "File[/tmp/foo3]"
        }
    ],
    "environment": "production",
    "name": "michaels-mbp.corp.puppetlabs.net",
    "resources": [
        {
            "exported": false,
            "tags": [
                "stage",
                "main"
            ],
            "title": "main",
            "type": "Stage"
        },
        {
            "exported": false,
            "tags": [
                "class",
                "settings"
            ],
            "title": "Settings",
            "type": "Class"
        },
        {
            "exported": false,
            "parameters": {
                "name": "main"
            },
            "tags": [
                "class",
                "main"
            ],
            "title": "Main",
            "type": "Class"
        },
        {
            "exported": false,
            "file": "/Users/michaelsmith/puppetlabs/puppet/master/code/environments/production/manifests/test.pp",
            "line": 1,
            "parameters": {
                "checksum": "md5",
                "ensure": "file",
                "source": "puppet:///modules/a_core_module/foo"
            },
            "tags": [
                "file",
                "class"
            ],
            "title": "/tmp/foo",
            "type": "File"
        },
        {
            "exported": false,
            "file": "/Users/michaelsmith/puppetlabs/puppet/master/code/environments/production/manifests/test.pp",
            "line": 6,
            "parameters": {
                "ensure": "file",
                "source": "file:///Users/michaelsmith/tmp/List.h"
            },
            "tags": [
                "file",
                "class"
            ],
            "title": "/tmp/foo2",
            "type": "File"
        },
        {
            "exported": false,
            "file": "/Users/michaelsmith/puppetlabs/puppet/master/code/environments/production/manifests/test.pp",
            "line": 11,
            "parameters": {
                "checksum": "md5",
                "checksum_value": "39a1e84d5dbaf59fadbd3cfb32c19aff",
                "ensure": "file",
                "source": "puppet:///modules/a_module/cupcake"
            },
            "tags": [
                "file",
                "class"
            ],
            "title": "/tmp/foo3",
            "type": "File"
        }
    ],
    "tags": [
        "settings"
    ],
    "version": 1455142213
}

The foo3 resource has checksum_value inlined (as expected, since it's in the production environment); the others don't. The files end up with the expected contents

$ md5 /tmp/foo modules/a_core_module/files/foo
MD5 (/tmp/foo) = b1946ac92492d2347c6235b4d2611184
MD5 (modules/a_core_module/files/foo) = b1946ac92492d2347c6235b4d2611184
 
$ md5 /tmp/foo2 ~/tmp/List.h
MD5 (/tmp/foo2) = 0cc3e742fc8ac403d25de9ef29d6bfeb
MD5 (/Users/michaelsmith/tmp/List.h) = 0cc3e742fc8ac403d25de9ef29d6bfeb
 
$ md5 /tmp/foo3 environments/production/modules/a_module/files/cupcake
MD5 (/tmp/foo3) = 39a1e84d5dbaf59fadbd3cfb32c19aff
MD5 (environments/production/modules/a_module/files/cupcake) = 39a1e84d5dbaf59fadbd3cfb32c19aff

Generated at Sun Sep 27 16:09:30 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.