[PUP-1963] Generated resources never receive dependency edges Created: 2014/03/17  Updated: 2019/06/03  Resolved: 2015/10/28

Status: Closed
Project: Puppet
Component/s: Catalog Application
Affects Version/s: PUP 3.4.3
Fix Version/s: PUP 4.3.0

Type: Bug Priority: Blocker
Reporter: redmine.exporter Assignee: Unassigned
Resolution: Fixed Votes: 4
Labels: redmine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks MODULES-1341 firewall with purge=>true fails when ... Resolved
Duplicate
is duplicated by PDB-134 Purging resources that were exported ... Closed
Relates
relates to PUP-4748 Allow resources to be purged even whe... Closed
relates to PUP-5447 Warning for dependency conflict with ... Closed
relates to MODULES-3030 Firewall module not purging unmanaged... Accepted
relates to PUP-1153 Incorrect ordering of group and user ... Closed
Template:
Epic Link: Transaction Events
Story Points: 3
Sprint: Client 2015-09-16, Client 2015-09-30, Client 2015-10-14, Client 2015-10-28, Client 2015-11-11
Release Notes: Bug Fix
Release Notes Summary: Resources created by another type (via the `generate` method) will now have any relationship metaparams respected, and will participate in `autorequire`
QA Contact: Eric Thompson

 Description   

I've preserved the original description below for posterity. Note however that this issue is now tracking only the first (of two) issues described in the original problem report.


First off I will quickly explain what it is I'm trying to do. I have installed the puppetlabs-apache-0.0.3 module that provides a new type called a2mod. My goal is to use virtual resources to collect a list of modules to enable/disable on a node. I would also however, like to ensure that all other apache modules are disabled on the system. I found the resources type has a purge option. I then updated the provider to have an instances section since it did not have one. It looks like this:

    class << self
        attr_accessor :defpath
    end
 
    @defpath = "/etc/apache2/mods-available/"
 
    def self.instances
        self.defpath = [self.defpath] unless self.defpath.is_a? Array
 
        instances = []
 
        self.defpath.each do |path|
            unless FileTest.directory?(path)
                Puppet.debug "Apache module path %s does not exist" % path
                next
            end
 
            check = [:ensure]
 
            if public_method_defined? :enabled?
                check << :enable
            end
 
            Dir.entries(path).each do |name|
                if name =~ /.load/
                    name = name.sub(".load","")
                    instances << new(:name => name, :path => path)
                end
            end
        end
        instances
    end

I then used the resources type to purge out unused apache modules. This seemingly does the job I wanted it to do. However, I wanted to notify my apache to restart if it had disabled any modules. I added the notify metaparameter to the configuration but it would never notify.

    resources { 'a2mod': tag => 'modtest', notify => Exec['apache2reload'], purge => true, loglevel => 'notice' }

I then tried to add a tag for testing purposes and it would not see the tag being set. I could however use loglevel but it would seem other metaparameters would not work properly. I'm assuming that all parameters on http://docs.puppetlabs.com/references/stable/metaparameter.html should be available to the resources type as indicated by the documentation for the resources type.

So, I'm wanting to understand if "all" metaparameters should work with the resources type or not. And, is there any possible way to do a purge that would allow me to do a notify of another resource.



 Comments   
Comment by Chad Metcalf [ 2014/03/17 ]

This seems to still be the case.

In a world where I have exported resources that are purged, I want to notify a resource that the exported resources set has changed, recalculate and do whatever.

resources { [ "foo", "bar"]: purge => true, notify => Service["baz"] }

Poked Charlie Sharpsteen about it.

Comment by Chad Metcalf [ 2014/03/24 ]

The customer I was talking to had a couple use cases but basically they all boil down to:

We're using exported resources for (dns, monitoring, whatever) and when there is a purge we want to notify (bind, nagios, whatever) to reload its configs to pickup on the new config.

Comment by Charlie Sharpsteen [ 2014/03/24 ]

Alright. This is an interesting one, that eventually led into the Transaction where the agent evaluates a catalog received from the master. The investigation turned up some strange behavior, so this update will be a bit detailed.

Catalogs and Edges

Relationships between Puppet resources are tracked in two separate places. The main catalog tracks "containment edges" which represent the hierarchical structure of the node, classes, defined types and the individual resources inside each of those structures. The main catalog also has an instance variable called relationship_graph that keeps track of "dependency edges" which are relationships defined by meta-parameters such as require, notify, etc.

The important thing to note is that the relationship_graph, and associated dependency edges, is built lazily the first time it is referenced by the main catalog.

Generated Resources

In certain cases, resource definitions are generated and added to the catalog during the agent run instead of during compilation on the master. Use of the Resources type is one of these cases. The Resources type needs to generate new resource instances for things it finds on the agent so that it can purge instances that are not managed by the catalog.

The important thing to note is that for generated resources, edged created by Notify and Subscribe metaparameters are never explicitly added to the relationship_graph.

The Problem

Generated resources are created after the relationship_graph is built. This means Subscribe or Notify parameters attached to those resources are never added to the relationship_graph. So, no event is ever generated for something like:

resources { [ "foo", "bar"]: purge => true, notify => Service["baz"] }

A secondary issue pops up in Puppet versions prior to 3.3.0 which refactored a lot of code to enable manifest-ordered application of resources. Prior to 3.3.0, the very first generated resource would be included in relationship_graph because the genesis of this first resource would cause the graph to be built. This leads to the warnings reflected in the linked Redmine ticket:

warning: /Mailalias[bin]: Whit[Completed_class[Postfix]],Whit[Completed_stage[main]],Exec[newaliases] still depend on me -- not purging

Purge seems to have a different idea of what resource dependencies mean compared to the rest of the Puppet codebase. Purge seems to think that a dependency relationship means "the resource must exist" instead of "the resource must be synched first" and thus refuses to destroy things. Purge has acted this way since it sprang into existance.

A Possible Solution

Two changes:

  1. Ensure metaparameters such as Notify and Subscribe are fully evaluated for generated resources and added to the relationship_graph.
  2. Allow resources to be purged even whey they have dependency relationships. Not doing so seems inconsistent with how the rest of Puppet operates.

Change 2 seems like a breaking change that could only go out in Puppet 4.x. However, since dependencies are not properly generated in the first place there may not be any existing behavior to break.

Comment by Charlie Sharpsteen [ 2014/03/24 ]

This needs a decision on whether or not this issue is in the scope of 3.x or 4.x.

Comment by Kylo Ginsberg [ 2014/05/12 ]

Yeah, #2 would be a breaking change so would be nice to get into puppet 4.

Charlie Sharpsteen I haven't looked at the code enough to consider what this would take. Have you spiked the implementation for either #1 or #2?

Basically I'm wondering about LOE, so if you've investigated enough to warrant a guess, that'd be great. If not, we might want to add an "investigating" ticket to the board to spike this. (We also might want to break out #1 and #2 into separate tickets, but we that can be part of the investigation.)

Comment by Charlie Sharpsteen [ 2014/05/13 ]

I haven't tried to implement either of these two suggestions, but from what I recall of the code:

I think #1 could be approached by creating a point during the agent transaction where the relationship graph is explicitly generated rather than doing it lazily the first time the graph is referenced. I can't conceive of any agent transaction that does not include the relationship graph, so it makes sense that this step should be explicit rather than lazy. Once that is done, resolving #1 is a matter of generating the relationship graph after the agent generates additional resources.

#2 is pretty simple and just involves the removal of a logic check. Most of the effort here will be in ensuring there wasn't some edge case or obscure reason that caused this check to be created.

Comment by Matt Palmer [ 2014/08/14 ]

This is trivially fixed by adding the relationship edges to the relationship graph in `Puppet::Transaction::AdditionalResourceGenerator#add_resource`, in exactly the same way that `Puppet::Graph::RelationshipGraph#build_manual_dependencies` does it. I've patched my local implementation in this fashion, and it resolves the bug quite satisfactorily.

Comment by Kylo Ginsberg [ 2014/08/15 ]

Matt Palmer thanks for checking that out! Would you mind submitting the patch as a pull request?

Comment by Matt Palmer [ 2014/08/16 ]

No point; I don't sign CLAs.

Comment by Charlie Sharpsteen [ 2015/02/24 ]

Steve Barlow: This should happen in a major release and not a feature release such as 4.1.0 as it will change the structure of catalogs and behavior of agents processing those catalogs.

Comment by Thomas Mueller [ 2015/06/09 ]

puppetlabs-concat >= 2.0.1 is affacted by this bug (MODULES-2104)

Comment by Kylo Ginsberg [ 2015/06/11 ]

Per Charlie's analysis above, I've broken this into two issues:

  • PUP-1963 (this one), not a breaking change, matches the original summary, targeted at 4.3.0
  • PUP-4748, which is a breaking change, targeted at 5.0.0
Comment by Jonny (JT) Tripathy [ 2015/07/02 ]

This also affect the autorequire of those generated types: i.e. autorequire is never evaluated for those generated types.

Comment by Branan Riley [ 2015/09/14 ]

I've been poking through the transaction a whole bunch today. It looks like we should be able to defer creation of the relationship graph until after all resources have been generated. This will cause metaparams of generated resources to be evaluated, and should also fix the autorequire issue mentioned by Jonathan above (autorequires are evaluated as part of creating the relationship graph)

AFAICT, the only thing that the resource generation process needs the graph for is logic around adding the depth/breadth first edge between the generating and generated resources. Deferring the creation of the relationship graph means this method needs to become a bit more compllicated, checking the various ways that the catalog could potentially have conflicting edges defined (whereas currently it just checks the generated relationship graph).

The alternative is what was proposed by Matt Palmer, which is to copy some logic from the relationship graph creation into resource generation. This adds nearly as much complexity to the resource generation process (analyzing dependency metaparams) without the benefit of also fixing up the autorequire issue.

Since it kills two birds with one stone, and since I've personally been bit by the autorequire thing when playing around with complex resource auto-generation, I'm inclined to go with the first option. It also feels better to me from the perspective of data flow cleanliness - we generate resources, THEN create the graph, as opposed to generating a graph then screwing around with it.

Any strong opinions Kylo Ginsberg Josh Cooper Charlie Sharpsteen?

EDIT: One more point in favor of the delaying creation of the relationship graph: I think it would be super unintuitive for dependency metaparams to work but not autorequries.

Comment by Eric Thompson [ 2015/10/13 ]

risk is independent of test-level. good coverage at unit-level is great. but medium risk implies we should still manually validate against full stack.

Comment by Michael Smith [ 2015/10/15 ]

Branan Riley can you update the release notes?

Comment by Tony Vu [ 2015/10/28 ]

Setting up a dummy type and provider namded hodor, I was able to prove the dependency edge gets created with following snippets code.

Dummy hodor code

Puppet::Type.newtype(:hodor) do
  @doc = "hodortypedoc"
  ensurable
  newparam(:name) do
    desc "namevar"
  end
end
 
Puppet::Type.type(:hodor).provide(:posix) do
  @doc = "hodorhodorhodor"
  def create
    STDOUT.puts "hodorcreate"
  end
  def destroy
    STDOUT.puts "hodordestroy"
  end
  def exists?
    STDOUT.puts "hodorexists?"
  end
  def self.instances
  [new(:name => "hodor")]
  end
end

Puppet code

resources { "hodor": purge => true, notify => Service["ntpd"] }

Puppet run output

Warning: /Stage[main]/Main/Node[centos7]/Hodor[hodor]: Service[ntpd] still depends on me -- not purging

I have written up an associated ticket to change the "me" in this message to the actual conflicting resource, PUP-5477. Otherwise, LGTM!

Comment by Peter Meier [ 2015/11/16 ]

Just for the record, imho this is another example of this bug:

In short: Although resource purging should notify nagios, no notify is sent to the service and hence the service is not restarted.

[root@foo ~]# cat add.pp 
 
nagios_hostextinfo { 'somehost':
  ensure => 'present',
  notes  => 'bla',
}
[root@foo ~]# cat remove.pp 
resources {'nagios_hostextinfo':
  purge => true,
  notify => Service['nagios'],
}
 
service{'nagios':
  ensure => running,
}
[root@foo ~]# cat /etc/nagios/nagios_hostextinfo.cfg
# HEADER: This file was autogenerated at Mon Nov 16 15:40:35 -0500 2015
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
[root@foo ~]# puppet apply add.pp
Notice: Compiled catalog for foo.example.com in environment production in 0.19 seconds
Notice: /Stage[main]/Main/Nagios_hostextinfo[somehost]/ensure: created
Notice: Finished catalog run in 1.63 seconds
[root@foo ~]# cat /etc/nagios/nagios_hostextinfo.cfg
# HEADER: This file was autogenerated at Mon Nov 16 15:41:22 -0500 2015
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
define hostextinfo {
  host_name                      somehost
  notes                          bla
}
[root@foo ~]# puppet apply remove.pp --debug
Debug: Runtime environment: run_mode=user, puppet_version=3.8.3, ruby_version=1.8.7
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Loading external facts from /var/lib/puppet/facts.d
Info: Loading facts
Debug: Loading facts from /var/lib/puppet/lib/facter/ekeyd.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/acpi_available.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/configured_ntp_servers.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/tor_hidden_services.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/keepalived_version.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/mountpoints.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/virtual_guests_count.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/virtual_guests.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/facter_dot_d.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/mysql_version.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/reboot_monkey_services.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/mysql_exists.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/jenkins.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/root_home.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/puppet_vardir.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/pe_version.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/apt_running.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/reboot_monkey.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/interfaces.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/nf_conntrack_max.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/monkeysphere.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/lvm_support.rb
Debug: Loading facts from /var/lib/puppet/lib/facter/concat_basedir.rb
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Notice: Compiled catalog for foo.example.com in environment production in 0.34 seconds
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Puppet::Type::Service::ProviderRunit: file /usr/bin/sv does not exist
Debug: Puppet::Type::Service::ProviderSystemd: file systemctl does not exist
Debug: Puppet::Type::Service::ProviderOpenrc: file /bin/rc-status does not exist
Debug: Puppet::Type::Service::ProviderLaunchd: file /bin/launchctl does not exist
Debug: Puppet::Type::Service::ProviderGentoo: file /sbin/rc-update does not exist
Debug: Puppet::Type::Service::ProviderDaemontools: file /usr/bin/svc does not exist
Debug: Puppet::Type::Service::ProviderDebian: file /usr/sbin/update-rc.d does not exist
Debug: Creating default schedules
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Puppet::Type::User::ProviderUser_role_add: file rolemod does not exist
Debug: Puppet::Type::User::ProviderPw: file pw does not exist
Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dsimport does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]{:loglevel=>:debug, :group=>"puppet", :ensure=>:file, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"640", :path=>"/var/lib/puppet/ssl/private_keys/foo.example.com.pem"}'
Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}'
Debug: Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}'
Debug: Using settings: adding file resource 'classfile': 'File[/var/lib/puppet/state/classes.txt]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :owner=>"root", :backup=>false, :mode=>"640", :path=>"/var/lib/puppet/state/classes.txt"}'
Debug: Using settings: adding file resource 'lastrunreport': 'File[/var/lib/puppet/state/last_run_report.yaml]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :mode=>"640", :path=>"/var/lib/puppet/state/last_run_report.yaml"}'
Debug: Using settings: adding file resource 'clientbucketdir': 'File[/var/lib/puppet/clientbucket]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/clientbucket"}'
Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :mode=>"1755", :path=>"/var/lib/puppet/state"}'
Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:loglevel=>:debug, :group=>"puppet", :ensure=>:file, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/ssl/certs/ca.pem"}'
Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/etc/puppet"}'
Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/ssl/private_keys"}'
Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/facts.d"}'
Debug: Using settings: adding file resource 'graphdir': 'File[/var/lib/puppet/state/graphs]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/state/graphs"}'
Debug: Using settings: adding file resource 'preview_outputdir': 'File[/var/lib/puppet/preview]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/preview"}'
Debug: Using settings: adding file resource 'statefile': 'File[/var/lib/puppet/state/state.yaml]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :mode=>"660", :path=>"/var/lib/puppet/state/state.yaml"}'
Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/run/puppet"}'
Debug: Using settings: adding file resource 'hostcert': 'File[/var/lib/puppet/ssl/certs/foo.example.com.pem]{:loglevel=>:debug, :group=>"puppet", :ensure=>:file, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/ssl/certs/foo.example.com.pem"}'
Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"771", :path=>"/var/lib/puppet/ssl"}'
Debug: Using settings: adding file resource 'client_datadir': 'File[/var/lib/puppet/client_data]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/client_data"}'
Debug: Using settings: adding file resource 'lastrunfile': 'File[/var/lib/puppet/state/last_run_summary.yaml]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/state/last_run_summary.yaml"}'
Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]{:loglevel=>:debug, :group=>"puppet", :ensure=>:file, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/ssl/public_keys/foo.example.com.pem"}'
Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/log/puppet"}'
Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/certificate_requests"}'
Debug: Using settings: adding file resource 'resourcefile': 'File[/var/lib/puppet/state/resources.txt]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :owner=>"root", :backup=>false, :mode=>"640", :path=>"/var/lib/puppet/state/resources.txt"}'
Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :path=>"/var/lib/puppet"}'
Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/ssl/private"}'
Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:loglevel=>:debug, :group=>"puppet", :ensure=>:file, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/ssl/crl.pem"}'
Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/certs"}'
Debug: Using settings: adding file resource 'clientyamldir': 'File[/var/lib/puppet/client_yaml]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/client_yaml"}'
Debug: /File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/private_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/public_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/public_keys
Debug: /File[/var/lib/puppet/ssl/public_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/public_keys
Debug: /File[/var/lib/puppet/ssl/public_keys]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/public_keys
Debug: /File[/var/lib/puppet/ssl/public_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/public_keys
Debug: /File[/var/lib/puppet/lib]/seluser: Found seluser default 'system_u' for /var/lib/puppet/lib
Debug: /File[/var/lib/puppet/lib]/selrole: Found selrole default 'object_r' for /var/lib/puppet/lib
Debug: /File[/var/lib/puppet/lib]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/lib
Debug: /File[/var/lib/puppet/lib]/selrange: Found selrange default 's0' for /var/lib/puppet/lib
Debug: /File[/var/lib/puppet/state/classes.txt]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/classes.txt
Debug: /File[/var/lib/puppet/state/classes.txt]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/classes.txt
Debug: /File[/var/lib/puppet/state/classes.txt]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/classes.txt
Debug: /File[/var/lib/puppet/state/classes.txt]/selrange: Found selrange default 's0' for /var/lib/puppet/state/classes.txt
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/last_run_report.yaml
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/last_run_report.yaml
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/last_run_report.yaml
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/state/last_run_report.yaml
Debug: /File[/var/lib/puppet/clientbucket]/seluser: Found seluser default 'system_u' for /var/lib/puppet/clientbucket
Debug: /File[/var/lib/puppet/clientbucket]/selrole: Found selrole default 'object_r' for /var/lib/puppet/clientbucket
Debug: /File[/var/lib/puppet/clientbucket]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/clientbucket
Debug: /File[/var/lib/puppet/clientbucket]/selrange: Found selrange default 's0' for /var/lib/puppet/clientbucket
Debug: /File[/var/lib/puppet/state]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state
Debug: /File[/var/lib/puppet/state]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state
Debug: /File[/var/lib/puppet/state]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state
Debug: /File[/var/lib/puppet/state]/selrange: Found selrange default 's0' for /var/lib/puppet/state
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/ca.pem
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/ca.pem
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/certs/ca.pem
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/ca.pem
Debug: /File[/etc/puppet]/seluser: Found seluser default 'system_u' for /etc/puppet
Debug: /File[/etc/puppet]/selrole: Found selrole default 'object_r' for /etc/puppet
Debug: /File[/etc/puppet]/seltype: Found seltype default 'puppet_etc_t' for /etc/puppet
Debug: /File[/etc/puppet]/selrange: Found selrange default 's0' for /etc/puppet
Debug: /File[/var/lib/puppet/ssl/private_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys
Debug: /File[/var/lib/puppet/ssl/private_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys
Debug: /File[/var/lib/puppet/ssl/private_keys]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/private_keys
Debug: /File[/var/lib/puppet/ssl/private_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys
Debug: /File[/var/lib/puppet/facts.d]/seluser: Found seluser default 'system_u' for /var/lib/puppet/facts.d
Debug: /File[/var/lib/puppet/facts.d]/selrole: Found selrole default 'object_r' for /var/lib/puppet/facts.d
Debug: /File[/var/lib/puppet/facts.d]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/facts.d
Debug: /File[/var/lib/puppet/facts.d]/selrange: Found selrange default 's0' for /var/lib/puppet/facts.d
Debug: /File[/var/lib/puppet/state/graphs]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/graphs
Debug: /File[/var/lib/puppet/state/graphs]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/graphs
Debug: /File[/var/lib/puppet/state/graphs]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/graphs
Debug: /File[/var/lib/puppet/state/graphs]/selrange: Found selrange default 's0' for /var/lib/puppet/state/graphs
Debug: /File[/var/lib/puppet/preview]/seluser: Found seluser default 'system_u' for /var/lib/puppet/preview
Debug: /File[/var/lib/puppet/preview]/selrole: Found selrole default 'object_r' for /var/lib/puppet/preview
Debug: /File[/var/lib/puppet/preview]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/preview
Debug: /File[/var/lib/puppet/preview]/selrange: Found selrange default 's0' for /var/lib/puppet/preview
Debug: /File[/var/lib/puppet/state/state.yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/state.yaml
Debug: /File[/var/lib/puppet/state/state.yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/state.yaml
Debug: /File[/var/lib/puppet/state/state.yaml]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/state.yaml
Debug: /File[/var/lib/puppet/state/state.yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/state/state.yaml
Debug: /File[/var/run/puppet]/seluser: Found seluser default 'system_u' for /var/run/puppet
Debug: /File[/var/run/puppet]/selrole: Found selrole default 'object_r' for /var/run/puppet
Debug: /File[/var/run/puppet]/seltype: Found seltype default 'puppet_var_run_t' for /var/run/puppet
Debug: /File[/var/run/puppet]/selrange: Found selrange default 's0' for /var/run/puppet
Debug: /File[/var/lib/puppet/ssl/certs/foo.example.com.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/certs/foo.example.com.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/certs/foo.example.com.pem]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/certs/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/certs/foo.example.com.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl
Debug: /File[/var/lib/puppet/ssl]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl
Debug: /File[/var/lib/puppet/ssl]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl
Debug: /File[/var/lib/puppet/ssl]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl
Debug: /File[/var/lib/puppet/client_data]/seluser: Found seluser default 'system_u' for /var/lib/puppet/client_data
Debug: /File[/var/lib/puppet/client_data]/selrole: Found selrole default 'object_r' for /var/lib/puppet/client_data
Debug: /File[/var/lib/puppet/client_data]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/client_data
Debug: /File[/var/lib/puppet/client_data]/selrange: Found selrange default 's0' for /var/lib/puppet/client_data
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/last_run_summary.yaml
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/last_run_summary.yaml
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/last_run_summary.yaml
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/state/last_run_summary.yaml
Debug: /File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/public_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/public_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/public_keys/foo.example.com.pem
Debug: /File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/public_keys/foo.example.com.pem
Debug: /File[/var/log/puppet]/seluser: Found seluser default 'system_u' for /var/log/puppet
Debug: /File[/var/log/puppet]/selrole: Found selrole default 'object_r' for /var/log/puppet
Debug: /File[/var/log/puppet]/seltype: Found seltype default 'puppet_log_t' for /var/log/puppet
Debug: /File[/var/log/puppet]/selrange: Found selrange default 's0' for /var/log/puppet
Debug: /File[/var/lib/puppet/ssl/certificate_requests]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certificate_requests
Debug: /File[/var/lib/puppet/ssl/certificate_requests]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certificate_requests
Debug: /File[/var/lib/puppet/ssl/certificate_requests]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/certificate_requests
Debug: /File[/var/lib/puppet/ssl/certificate_requests]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certificate_requests
Debug: /File[/var/lib/puppet/state/resources.txt]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state/resources.txt
Debug: /File[/var/lib/puppet/state/resources.txt]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state/resources.txt
Debug: /File[/var/lib/puppet/state/resources.txt]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/state/resources.txt
Debug: /File[/var/lib/puppet/state/resources.txt]/selrange: Found selrange default 's0' for /var/lib/puppet/state/resources.txt
Debug: /File[/var/lib/puppet]/seluser: Found seluser default 'system_u' for /var/lib/puppet
Debug: /File[/var/lib/puppet]/selrole: Found selrole default 'object_r' for /var/lib/puppet
Debug: /File[/var/lib/puppet]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet
Debug: /File[/var/lib/puppet]/selrange: Found selrange default 's0' for /var/lib/puppet
Debug: /File[/var/lib/puppet/ssl/private]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private
Debug: /File[/var/lib/puppet/ssl/private]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private
Debug: /File[/var/lib/puppet/ssl/private]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/private
Debug: /File[/var/lib/puppet/ssl/private]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private
Debug: /File[/var/lib/puppet/ssl/crl.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/crl.pem
Debug: /File[/var/lib/puppet/ssl/crl.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/crl.pem
Debug: /File[/var/lib/puppet/ssl/crl.pem]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/crl.pem
Debug: /File[/var/lib/puppet/ssl/crl.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/crl.pem
Debug: /File[/var/lib/puppet/ssl/certs]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs
Debug: /File[/var/lib/puppet/ssl/certs]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs
Debug: /File[/var/lib/puppet/ssl/certs]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/ssl/certs
Debug: /File[/var/lib/puppet/ssl/certs]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs
Debug: /File[/var/lib/puppet/client_yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/client_yaml
Debug: /File[/var/lib/puppet/client_yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/client_yaml
Debug: /File[/var/lib/puppet/client_yaml]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/client_yaml
Debug: /File[/var/lib/puppet/client_yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/client_yaml
Debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/state/classes.txt]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/state/resources.txt]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs]
Debug: /File[/var/lib/puppet/ssl/private_keys/foo.example.com.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys]
Debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/public_keys/foo.example.com.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys]
Debug: /File[/var/lib/puppet/preview]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certs/foo.example.com.pem]: Autorequiring File[/var/lib/puppet/ssl/certs]
Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/state/last_run_report.yaml]/seluser: seluser changed 'unconfined_u' to 'system_u'
Debug: /File[/var/lib/puppet/state/state.yaml]/seluser: seluser changed 'unconfined_u' to 'system_u'
Debug: /File[/var/lib/puppet/state/last_run_summary.yaml]/seluser: seluser changed 'unconfined_u' to 'system_u'
Debug: Finishing transaction 70056953517400
Debug: Loaded state in 0.33 seconds
Debug: Loaded state in 0.33 seconds
Debug: /Stage[main]/Main/Resources[nagios_hostextinfo]/notify: subscribes to Service[nagios]
Info: Applying configuration version '1447706508'
Debug: Executing '/sbin/service nagios status'
Debug: Prefetching naginator resources for nagios_hostextinfo
Notice: /Stage[main]/Main/Nagios_hostextinfo[somehost]/ensure: removed
Debug: Flushing nagios_hostextinfo provider target /etc/nagios/nagios_hostextinfo.cfg
Info: Computing checksum on file /etc/nagios/nagios_hostextinfo.cfg
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Finishing transaction 70056952376520
Debug: Storing state
Debug: Stored state in 1.28 seconds
Notice: Finished catalog run in 1.91 seconds
Debug: Using settings: adding file resource 'rrddir': 'File[/var/lib/puppet/rrd]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/rrd"}'
Debug: /File[/var/lib/puppet/rrd]/seluser: Found seluser default 'system_u' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/selrole: Found selrole default 'object_r' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/selrange: Found selrange default 's0' for /var/lib/puppet/rrd
Debug: Finishing transaction 70056952283880
Debug: Received report to process from foo.example.com
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Processing report from foo.example.com with processor Puppet::Reports::Store
[root@foo ~]# cat /etc/nagios/nagios_hostextinfo.cfg
# HEADER: This file was autogenerated at Mon Nov 16 15:41:50 -0500 2015
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
[root@foo ~]# 

Comment by Curtis Ruck [ 2016/01/12 ]

This introduced a breaking change. If you use stages, and you use resource purging, and you do resource purging in any stage but the last, you now trigger PUP-4748.

Comment by Curtis Ruck [ 2016/01/12 ]

I'm actually exceedingly frustrated by this. We have about 45 different types that we purge on, these purges are sandwiched between a few stages (i.e. a pre, and post stage). I'd look at flattening the post stage into the main stage as a fix, but it contains a refreshonly=true Exec['reboot'], and using stages is the only way we've found to ensure it goes last.

Comment by Alexander Johnson [ 2016/01/12 ]

Hi Curtis Ruck - What version of Puppet are you using? This was fixed in Puppet 4.3.

Comment by Charlie Sharpsteen [ 2016/01/12 ]

Alexander Johnson: I think they're running into PUP-4748 which was a related issue that we split off from this one and deferred for Puppet 5. Basically, purge resources refuse to execute if they have downstream dependencies — that was the original intent of the purge design from the day it was implemented (waaaaaaay back in the past, in a 0.x release). However, purged resources are also generated so, due to this bug, they never received a certain set of dependencies that could trigger that old design decision.

Now that this issue is resolved, PUP-4748 is becoming a problem in a larger set of circumstances.

Comment by Curtis Ruck [ 2016/01/12 ]

We are running puppet 4.3.1. So, by having a bug, for 4-5 years, fixing that, by changing functionality (by not doing #2, PUP-4748), creates a breaking change.

All I'd like is that someone agree that this PUP-1963 created a breaking change (even if it fixes a long standing issue). If we can get PUP-1963 annotated as creating a breaking change, then PUP-4748 can be prioritized for 4.x something since it restores functionality that was broken by PUP-1963.

Without PUP-4748, I have no way to do any resource purging which is one of the bigger reasons we use puppet.

Comment by Branan Riley [ 2016/01/12 ]

Curtis Ruck I'm sorry about the regression with stages. Apparently you've found a gap in our test coverage. If you can send a small reproduction case (ideally attached to a new bug) I'll take a look at it ASAP.

Comment by Branan Riley [ 2016/01/12 ]

Charlie Sharpsteen PUP-4748 shouldn't hit stages, as those are containment edges that are converted to whits (which I explicitly ignored when checking if a purged resource has edges). I'm actually super curious what's going on here, but it does smell like a bug outside the scope of PUP-4748 to me.

Comment by Curtis Ruck [ 2016/01/12 ]

Put some hostfile entries in your /etc/hosts and run puppet apply --modulepath . -v bug.pp

bug/manifests/reboot.pp

class bug::reboot {
  exec{'reboot':
    path=>'/sbin',
    refreshonly=>true,
  }
}

bug/manifests/stages.pp

class bug::stages {
  stage{'reboot':
    require=>Stage['main'],
  }
}

bug.pp

include 'bug::stages'
 
resources{'host':
  purge=>true,
}
 
class{'bug::reboot':
  stage=>'reboot',
}

My results in debug form are

Info: Applying configuration version '1452620593'
Debug: /Stage[reboot]/require: requires Stage[main]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: /Stage[main]/Main/Resources[host]/before: requires Host[sanitized.localhost]
Debug: Prefetching parsed resources for host
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Warning: /Stage[main]/Main/Host[sanitized.localhost] still depends on me -- not purging
Debug: Finishing transaction 22547580
Debug: Storing state
Debug: Stored state in 0.12 seconds
Notice: Applied catalog in 0.30 seconds
Debug: Applying settings catalog for sections reporting, metrics
Debug: /File[/etc/puppetlabs/code/environments/production]/seluser: Found seluser default 'system_u' for /etc/puppetlabs/code/environments/production
Debug: /File[/etc/puppetlabs/code/environments/production]/selrole: Found selrole default 'object_r' for /etc/puppetlabs/code/environments/production
Debug: /File[/etc/puppetlabs/code/environments/production]/seltype: Found seltype default 'etc_t' for /etc/puppetlabs/code/environments/production
Debug: /File[/etc/puppetlabs/code/environments/production]/selrange: Found selrange default 's0' for /etc/puppetlabs/code/environments/production
Debug: Finishing transaction 22273500
Debug: Received report to process from sanitized.localhost
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Processing report from sanitized.localhost with processor Puppet::Reports::Store

Comment by Thomas Mueller [ 2016/01/13 ]

Curtis Ruck maybe you could work around the following way:

create a script to check a trigger file and use it as your postrun_command:

puppet config set postrun_command="my-reboot-checker-script"

instead of using the exec to reboot, create a trigger file to signal the postrun-command:

exec{'trigger-reboot-with-postrun-command':
    command =>'touch /var/run/puppet-reboot-required',
    refreshonly=>true,
  }

Comment by Eric Sorenson [ 2016/01/20 ]

I've just read through the updates from Curtis Ruck and others and am trying to figure out what to do about this; if it's true that we were wrong about the initial impact of fixing this but not implementing PUP-4748 (i.e. only doing this ticket's scope caused a semver-major breaking change), then I would support pulling the PUP-4748 work into a Puppet 4.x release and not waiting for another major release cycle (or alternately, calling whatever release that goes out in "Puppet 5.0" and documenting the two things together as being a break). In any event since this change caused problems both for concat and purge I would like to complete the proposed changes soon-ish if possible.

OTOH if the behaviour Curtis is seeing is a bug as-implemented and not part of PUP-4748 we should address that - Branan Riley any idea what it would take to ascertain whether or not that's the case? Does Curtis' repro case help with that?

Comment by Branan Riley [ 2016/01/21 ]

I'm going to start with a simple example, and work up to what happens with stages. Previously, the following catalog:

resources{ 'host':
  purge => true
} ->
 
exec { 'reboot':
  path => '/sbin'
}

Would create the following graph, with the evaluation order below it. Note that the graph doesn't actually contain the edges for the Exec dependency, even though the purged hosts are guaranteed to evaluate immediately after the Resources resource, and thus the edge can be said to be implied.

Whit[Admissible_stage[main]] --> Resources[host] --------------------> Whit[Completed_stage[main]]
                                                 |-> Exec[reboot] -/
                                                 |-> Host[foo.bar]
                                                 \-> Host[baz.qux]
 
Whit[Admissible...]
Resources[host]
Host[foo.bar]
Host[baz.qux]
Exec[reboot]
Whit[Completed...]

With the new behavior, the graph looks as follows. The implied edge before the Exec (and the Whit for the end of the stage) is now explicit. The evaluation order for this graph is identical to that of the first one, but the purged resources WILL NOT RUN because the dependent resource check now sees the edge to the Exec

Whit[Admissible_stage[main]] --> Resources[host] ---------------------------------------> Whit[Completed_stage[main]]
                                                 |--------------------> Exec[reboot] -|
                                                 |-> Host[foo.bar] -|-----------------|
                                                 \-> Host[baz.qux] -/-----------------/

Arguably, this is a "bug fix", since previously the transaction would order the resource application such that a purge operation could have "dependent" resources, even though the edge itself was never generated. Now that the edge is in place, the "no dependent resources" check is actually effective.

(Aside: because this edge was implied,things would be ordered but events from notifies would not be fired, which is the initial bug report here).

Now, we get to stages. A simplified form of the example catalog looks something like this when converted to a graph:

Whit[Admissible_stage[main] --> Resources[host] ---------------------> Whit[Completed_stage[main]] -> Whit[Admissible_stage[reboot]] -> Exec[reboot] -> Whit[Completed_stage[reboot]]
                                                |-> Host[foo.bar] -|
                                                \-> Host[baz.qux] -/

The generated resources only have a direct relationship with a Whit, but the indirect relationship to the Exec resource still counts as a dependent.

All that being said, I think there are three options to fix this:

  • Change the dependent check to look only one level deep. This will allow any dependency behind a Whit (across class boundaries is a great example) to be expressed, which is pretty clearly not the intent of the dependency check, but it'll be fast to implement.
  • Change the dependent check to stop walking the tree when it hits a Whit from a stage. This is the most complicated option of the three, but I think best represents the current "intended" behavior regarding dependants of purged resources.
  • Just go ahead and do PUP-4748

Option 1 really isn't an option, IMO. It's a total hack, and this part of the codebase is gross enough already. Option 2 is pretty reasonable. It is more work than just doing PUP-4647, although not tremendously so (a day instead of an afternoon). It maintains the current documented behavior in a Z release, which is also fairly important. Option 3 speaks for itself.

Comment by Branan Riley [ 2016/01/21 ]

Given that our previous behavior was to allow dependencies and silently ignore notifies, I'm kind of inclined to just do PUP-4748. Even though option 2 is more "correct" to our current documented behavior, it might still cause brekages for other users who have managed to accidentally order a resource purge while the check was unenforced.

Since we want to eventually get rid of the check anyway, I think going from "ordering allowed but notifies don't work" to "all ordering metaparameters work as expected" is a way better move than "fixing" the documented purges-can't-be-ordered behavior.

Comment by Jonny (JT) Tripathy [ 2016/02/10 ]

Folks, what was the status of this ticket?

We are currently seeing an issue in the firewall module, where purging doesn't work properly (use the resources metatype), if the iptables package isn't installed. The firewall class is supposed to install the iptables package, however regardless of the before/require metaparam includes, the generated resources are always executed first.

Thanks

Comment by Branan Riley [ 2016/02/10 ]

Jonny (JT) Tripathy A version of this was released in puppet-agent 1.3.0 (Puppet 4.3) that tried to dance between allowing relationships for generated resources and still match the "documented" behavior of purged resources. It should work for that particular case, where you want to have purged resources depend on another resource. If you've upgraded to that version and it's not working for you (or working worse than it did with older Puppet versions) please let me know.

In the next couple weeks I'll be tackling PUP-4748 to completely remove the checks around relationships of purged resources, and that should go out to the world in puppet-agent 1.4.0 with Puppet 4.4

EDIT: More generally: Anything not working right with purged resources on puppet-agent 1.3 is a bug and I'd like to tackle it in the next two weeks before we lock down Puppet 4.4

Comment by Josh Cooper [ 2016/02/22 ]

This also causes a regression in tidy (due to adding resources to the catalog in the reverse order) filed as PUP-5963:

$ mkdir -p /tmp/tidy/foo/bar/baz
$ git checkout fd3732b3328134f145e68ddfda2a5273abe9dd12
...
$ bundle exec puppet apply -e 'tidy { "/tmp/tidy/foo": matches => "*", recurse => true, rmdirs => true}' --graph --trace
Notice: Compiled catalog for XXX in environment production in 0.07 seconds
Notice: /Stage[main]/Main/Tidy[/tmp/tidy/foo]: Tidying 3 files
Error: Failed to apply catalog: Could not find dependency File[/tmp/tidy/foo/bar] for File[/tmp/tidy/foo]
/Users/josh/work/puppet/lib/puppet/util/errors.rb:106:in `fail'
/Users/josh/work/puppet/lib/puppet/type.rb:1491:in `block in validate_relationship'
/Users/josh/work/puppet/lib/puppet/type.rb:1488:in `each'
/Users/josh/work/puppet/lib/puppet/type.rb:1488:in `validate_relationship'
/Users/josh/work/puppet/lib/puppet/type.rb:2468:in `block in finish'
/Users/josh/work/puppet/lib/puppet/type.rb:2466:in `collect'
/Users/josh/work/puppet/lib/puppet/type.rb:2466:in `finish'
/Users/josh/work/puppet/lib/puppet/type/file.rb:456:in `finish'
/Users/josh/work/puppet/lib/puppet/transaction/additional_resource_generator.rb:131:in `add_resource'
/Users/josh/work/puppet/lib/puppet/transaction/additional_resource_generator.rb:37:in `block in generate_additional_resources'
/Users/josh/work/puppet/lib/puppet/transaction/additional_resource_generator.rb:27:in `reverse_each'
/Users/josh/work/puppet/lib/puppet/transaction/additional_resource_generator.rb:27:in `generate_additional_resources'
/Users/josh/work/puppet/lib/puppet/transaction.rb:83:in `block in evaluate'
1 ~/work/puppet (fd3732b...)  $ git checkout HEAD^
Previous HEAD position was fd3732b... Merge pull request #4258 from branan/bug/master/pup-1963-generate-resource-before-graph
HEAD is now at c2a46bc... Merge pull request #4324 from royosherove/ticket/master/pup-3953-fix-single-quotes-to-manifest
0 ~/work/puppet (c2a46bc...)  $ bundle exec puppet apply -e 'tidy { "/tmp/tidy/foo": matches => "*", recurse => true, rmdirs => true}' --graph --trace
Notice: Compiled catalog for arcturus.delivery.puppetlabs.net in environment production in 0.05 seconds
Notice: /Stage[main]/Main/Tidy[/tmp/tidy/foo]: Tidying 3 files
Notice: /Stage[main]/Main/File[/tmp/tidy/foo/bar/baz]/ensure: removed
Notice: /Stage[main]/Main/File[/tmp/tidy/foo/bar]/ensure: removed
Notice: /Stage[main]/Main/File[/tmp/tidy/foo]/ensure: removed
Notice: Applied catalog in 0.03 seconds

Generated at Wed Apr 01 14:37:56 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.