Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10628

Performance regression with large hashes using lookup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PUP 6.17.0
    • Fix Version/s: PUP 5.5.22, PUP 6.19.0
    • Component/s: Hiera & Lookup
    • Labels:
    • Environment:
    • Template:
      PUP Bug Template
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Fixes a performance regression in 5.5.x and up which affected hiera lookups in large hashes.
    • QA Risk Assessment:
      Needs Assessment

      Description

      Puppet Version: starting from 5.0.0
      **

      After upgrading our Puppet installation from 4.10 to 6.x I found large catalog compile time increasing (over 300 seconds) for some nodes.

      Profiling shows problem with large hashes in Hiera yamls.

       

      Good case:

       

      $ sudo yum downgrade -y http://yum.puppetlabs.com/el/6x/PC1/x86_64/puppet-agent-1.10.14-1.el6.x86_64.rpm
      $ puppet --version
      4.10.12
      $ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
      --- value1
      ...
      real    0m2.195s
      user    0m2.010s
      sys     0m0.181s

      Bad cases:

       

       

      $ sudo yum update -y http://yum.puppetlabs.com/puppet5/el/6/x86_64/puppet-agent-5.0.0-1.el6.x86_64.rpm
      $ puppet --version
      5.0.0
      $ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
      --- value1
      ...
      real    0m9.837s
      user    0m9.623s
      sys     0m0.207s

      $ sudo yum update -y puppet-agent
      $ puppet --version
      6.17.0
      $ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
      --- value1
      real    0m9.788s
      user    0m9.494s
      sys     0m0.296s

      My research ended on follow debug points:

       

      $ patch /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/data_provider.rb <<EOF
      --- data_provider.rb
      +++ data_provider.rb
      @@ -80,11 +80,14 @@
         # @param data_provider [DataProvider] The data provider that produced the hash
         # @return [Object] The data value
         def validate_data_value(value, &block)
      +require 'time'
      +print "data_provider.rb::validate_data_value::1 ", ::Time.now.to_f, "\n"
           # The DataProvider.value_type is self recursive so further recursive check of collections is needed here
           unless value_is_validated? || DataProvider.value_type.instance?(value)
             actual_type = Types::TypeCalculator.singleton.infer(value)
             raise Types::TypeAssertionError.new("#{yield} has wrong type, expects Puppet::LookupValue, got #{actual_type}", DataProvider.value_type, actual_type)
           end
      +print "data_provider.rb::validate_data_value::2 ", ::Time.now.to_f, "\n"
           value
         end
       end
      EOF

      Run with debug timestamps on 6.17.0:

      $ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
      data_provider.rb::validate_data_value::1 1597847682.16178
      data_provider.rb::validate_data_value::2 1597847689.6889844
      --- value1
      real    0m9.819s
      user    0m9.557s
      sys     0m0.261s

      Run with debug timestamps on 4.10.12:

      $ time puppet lookup --hiera_config hiera.yaml hash1.hash2.hash3.key1
      data_provider.rb::validate_data_value::1 1597847876.923722
      data_provider.rb::validate_data_value::2 1597847876.9429817
      --- value1
      ...
      real    0m2.206s
      user    0m2.050s
      sys     0m0.151s
      

      Slowdown from 19ms to 7.5s (390 times) on 2000 items.

       

       

      For 4000 items - slowndown from 38ms to 31s (800 times)

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              josh Josh Cooper
              Reporter:
              oleksandr.shtepa Oleksandr Shtepa
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support