[PUP-6415] static_compiler: symlinks in recursed directories don't work Created: 2016/06/16  Updated: 2016/08/11  Resolved: 2016/07/11

Status: Closed
Project: Puppet
Component/s: Types and Providers
Affects Version/s: PUP 3.8.7, PUP 4.5.2
Fix Version/s: PUP 4.6.0

Type: Bug Priority: Normal
Reporter: Chris Boot Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Ruby 2.1, Linux

Issue Links:
relates to PUP-5978 Deprecate static compiler Closed
Acceptance Criteria:

The generated File resource should contain a target property so that the symlink can be created.

Story Points: 0
Sprint: Client 2016-06-29, Client 2016-07-13 (HA, 1.5.3)
Release Notes: Bug Fix
Release Notes Summary: This change fixes an issue in the (now deprecated) static compiler where symlinks in recursed directories did not end up with {{target}} attributes, causing Puppet to fail to manage them.

Note that this only affects the static compiler, and *not* the newer static catalog functionality which was added as part of the direct puppet workflow.


When using the static_compiler terminus with a resource definition like the following:

    file { '/tmp/somedir':
        ensure  => directory,
        owner   => 'root',
        group   => 'root',
        links   => 'manage',
        mode    => '0644',
        recurse => true,
        source  => 'puppet:///modules/mymodule/somedir/',

When the source directory contains a symlink, catalog application aborts because the symlink has no target:

Error: /Stage[main]/Mymodule::Myclass/File[/tmp/somedir/foo]/ensure: change from absent to link failed: Cannot create a symlink without a target

I have a patch for this bug that I will send in via GitHub shortly. Unfortunately, writing an rspec test to check for this condition is beyond my grasp of Ruby.

Comment by Chris Boot [ 2016/06/16 ]

Pull request in https://github.com/puppetlabs/puppet/pull/5026

Comment by Josh Cooper [ 2016/06/16 ]

Hi Chris, thanks for investigating and submitting a patch. I just wanted to make sure you know that the static compiler is deprecated for the reasons given in PUP-5978. There are two main issues, the static compiler doesn't handle containment correctly (so there's no guarantee that the recursed children will be processed before downstream resources), and it can't compile catalogs for Windows agents.

In Puppet 4.4.0 and up there is a simple solution, use static catalogs. It provides a a similar end user experience without the static compiler bugs.

Comment by Chris Boot [ 2016/06/16 ]

Hi Josh,

I'm disappointed to hear that the static compiler is deprecated, it seems to help solve some real-world performance issues for me among other things. I've read up briefly about the replacement static catalogs system and it doesn't seem as mature to me. I know this isn't necessarily the right place to ask questions about it, but how does it handle directories and server-side recursion? symlinks?

We've been trying to use the static compiler to drastically reduce the number of requests for metadata that our agents send to our masters. We use Puppet to manage our own clients' systems, and those systems are geographically very dispersed and many are on the end of unreliable or high-latency links where simply combining all the metadata into the catalogue makes things run far smoother both for our agents and our masters.

Please reconsider.


Comment by Josh Cooper [ 2016/06/16 ]

Hi Chris, the static catalog feature is designed to solve exactly the same problem as the static compiler does. Namely eliminating the file metadata requests. Both implementations add the file metadata to the catalog. The difference is that the static catalog only adds the metadata, while the static compiler adds "synthetic" file resources to the catalog containing the metadata.

The problem is that the static compiler duplicates agent-side logic, and does so incorrectly. As a result there are many, many problems. Also the static compiler invokes agent-side validation logic when generating file resources (instance of Puppet::Type::File). That is a fundamental design problem that can't be easily solved.

I'd really like to understand your concerns about the static catalog feature, as it was intended to be a clear replacement? About your specific question:

how does it handle directories and server-side recursion? symlinks?

The static catalog feature post-processes the catalog. When it sees a directory, server side recursion, symlinks, etc, it internally makes a file metadata request, essentially what the agent would have done. The static catalog then inlines the file metadata response directly in the catalog. When the agent processes the catalog, it looks in the catalog, sees that the file metadata is already present, and doesn't need to ask for it.

The agent then processes the catalog exactly the same as it would have if it had sent the file metadata request. So it correctly handles recursion, recurse depth limits, multiple sources, source_select => all, symlinks, etc.

Comment by Chris Boot [ 2016/06/18 ]

So I've now had the chance to try out Static Catalogues in my lab (er, at home) Puppet setup. It seems to work well and does indeed appear to provide a replacement for the static_compiler terminus, at least at first glance. I also now understand much better how it works and how it essentially collects all the file metadata and delivers it with the catalogue. It's nice to see a new approach to this that really seems to work.

Where this fails is that it requires use of the PuppetServer Java blob on the server, and Puppet 4.4+ on the client.

We can use the static_compiler today and it mostly works, and works with Puppet 2.7 agents (which we still have in the field). Yes, we can migrate from distribution-supported Puppet packages to the PC1 versions in many cases but we've not had any particular reason to before. It also doesn't look like Debian is intending on going down the Java/PuppetServer route at all at the time being, which will inconvenience people like us.

In my opinion it would be useful if you could apply the patch to 4.x branches at least, despite the static_compiler now being deprecated.

(Now if only the richardc/datacat module could/would use this, that would be awesome!)

Comment by Josh Cooper [ 2016/06/24 ]

Hi Chris, thanks for taking the time to investigate. We can definitely apply the patch to 4.x (and I don't see why not 3.x, since the static compiler is the only option there). Interesting to hear about puppetserver. Is there anything getting in the way of switching over? Or is it just you've never had reason to before?

Comment by Chris Boot [ 2016/06/25 ]

Essentially we strive to only use packages from trusted sources, and as much as possible only from distributors themselves. For Debian, most packages are already provided (via the main distribution or backports), or at a pinch we have Debian Developers (including myself) on staff who can contribute packages back to Debian that we'll also use. For Red Hat and CentOS we'll use EPEL but the ecosystem is very different. Unfortunately, as most distributors don't like to distribute bleeding edge packages (for good reasons generally), we won't get updated Puppet packages without going for 3rd party package sources like yours.

We've had bad experiences using 3rd party sources before. Often such sources include packages unrelated to what's shipped and play havoc with others packages' dependencies. Other times packages ship their own versions of libraries such as OpenSSL and then don't bother keeping them up to date. The quality of such packages and sources is very variable. This kind of thing makes us very reluctant to reach for 3rd party packages unless we absolutely have to. I'm not saying your package repo or packages are like this at all, but once burned...

Generated at Mon Jul 13 22:06:19 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.