Details
-
Bug
-
Status: Closed
-
Normal
-
Resolution: Fixed
-
PUP 3.6.2
-
None
-
SmartOS, Puppet 3.6.2
-
3
-
2014-08-20
Description
Our Puppet instance has been pretty slow lately so we traced a puppet agent run to see what it was doing. In the trace we see a bunch of lines like this:
stat("/opt/local/lib/ruby/site_ruby/2.0.0/puppet/vendor/semantic/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/modules/admin/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/modules/percona/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/shared/modules/apt/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/shared/modules/puppetlabs-stdlib/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/external/modules/python/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/etc/puppet/external/modules/yum_puppet_plugin/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/var/lib/puppet/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/opt/local/lib/ruby/gems/2.0.0/gems/bigdecimal-1.2.0/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/opt/local/lib/ruby/gems/2.0.0/gems/bundler-1.5.3/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/opt/local/lib/ruby/gems/2.0.0/gems/byebug-3.1.2/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/opt/local/lib/ruby/gems/2.0.0/gems/columnize-0.8.9/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
stat("/opt/local/lib/ruby/gems/2.0.0/gems/daemons-1.1.9/lib/puppet/type/completed_stage.rb", 0xFFFFFD7FFFDFE4A0) Err#2 ENOENT
Which means that Puppet is attempting to find a file in a bunch of different locations, which we believe is part of a process Puppet goes through when trying to find and load a class or function. In total, we see around 20,000 attempts to open files that fit the completed_ or admissible_ pattern.
I dug in to what it was trying to find and found this: https://github.com/puppetlabs/puppet/blob/2350ed0db1264040864ad4dfc9908429d6298eb1/lib/puppet/type/whit.rb. My understanding is that a Whit is essentially a virtual graph node which appears before or after a regular graph node. I don't find any cases of us calling these in our Puppet repo, so I'm assuming that its internal Puppet code of some kind that is asking for these.
So it appears to me that Puppet is defining a virtual class type then searching disk for it through some set of directories including every gem it sees, every time it is accessed, on every run of Puppet. Am I totally misunderstanding this or Is there something we as users could be doing to create this situation? Is this the default behavior? If so, it seems broken or insane to me. Please talk me off this ledge!