Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
PUP 6.17.0
-
Coremunity
-
Platform Core KANBAN
-
Needs Assessment
-
Bug Fix
-
Fixes a performance regression in 5.5.x and up which affected hiera lookups in large hashes.
-
Needs Assessment
Description
Puppet Version: starting from 5.0.0
**
After upgrading our Puppet installation from 4.10 to 6.x I found large catalog compile time increasing (over 300 seconds) for some nodes.
Profiling shows problem with large hashes in Hiera yamls.
Â
Good case:
Â
$ sudo yum downgrade -y http://yum.puppetlabs.com/el/6x/PC1/x86_64/puppet-agent-1.10.14-1.el6.x86_64.rpm |
$ puppet --version
|
4.10.12 |
$ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
|
--- value1
|
...
|
real 0m2.195s
|
user 0m2.010s
|
sys 0m0.181s
|
Bad cases:
Â
Â
$ sudo yum update -y http://yum.puppetlabs.com/puppet5/el/6/x86_64/puppet-agent-5.0.0-1.el6.x86_64.rpm |
$ puppet --version
|
5.0.0 |
$ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
|
--- value1
|
...
|
real 0m9.837s
|
user 0m9.623s
|
sys 0m0.207s
|
$ sudo yum update -y puppet-agent
|
$ puppet --version
|
6.17.0 |
$ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
|
--- value1
|
real 0m9.788s
|
user 0m9.494s
|
sys 0m0.296s
|
My research ended on follow debug points:
Â
$ patch /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/data_provider.rb <<EOF
|
--- data_provider.rb
|
+++ data_provider.rb
|
@@ -80,11 +80,14 @@ |
# @param data_provider [DataProvider] The data provider that produced the hash |
# @return [Object] The data value |
def validate_data_value(value, &block)
|
+require 'time' |
+print "data_provider.rb::validate_data_value::1 ", ::Time.now.to_f, "\n" |
# The DataProvider.value_type is self recursive so further recursive check of collections is needed here
|
unless value_is_validated? || DataProvider.value_type.instance?(value)
|
actual_type = Types::TypeCalculator.singleton.infer(value)
|
raise Types::TypeAssertionError.new("#{yield} has wrong type, expects Puppet::LookupValue, got #{actual_type}", DataProvider.value_type, actual_type) |
end
|
+print "data_provider.rb::validate_data_value::2 ", ::Time.now.to_f, "\n" |
value
|
end
|
end
|
EOF
|
Run with debug timestamps on 6.17.0:
$ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
|
data_provider.rb::validate_data_value::1 1597847682.16178 |
data_provider.rb::validate_data_value::2 1597847689.6889844 |
--- value1
|
real 0m9.819s
|
user 0m9.557s
|
sys 0m0.261s
|
Run with debug timestamps on 4.10.12:
$ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
|
data_provider.rb::validate_data_value::1 1597847876.923722 |
data_provider.rb::validate_data_value::2 1597847876.9429817 |
--- value1
|
...
|
real 0m2.206s
|
user 0m2.050s
|
sys 0m0.151s
|
Slowdown from 19ms to 7.5s (390 times) on 2000 items.
Â
Â
For 4000 items - slowndown from 38ms to 31s (800 times)
Â
Attachments
Issue Links
- relates to
-
PUP-10650 Remove TypeSetType from RichData
-
- Closed
-
