Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-6622

Puppet function_loading benchmark performance degredation

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PUP 4.6.0
    • Fix Version/s: PUP 4.6.1
    • Component/s: None
    • Labels:
    • Template:
    • Acceptance Criteria:
      Hide

      The benchmark:function_loading should run in the same amount of time that it did in 4.5.x

      Show
      The benchmark:function_loading should run in the same amount of time that it did in 4.5.x
    • Story Points:
      1
    • Sprint:
      Language 2016-08-24
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Function loading and calling was hit by a performance regression that caused a 300% degradation in performance as observed by benchmarks. This is now fixed.

      Description

      On Windows 2008R2 / Ruby 2.1, commit https://github.com/puppetlabs/puppet/commit/01db61e24178a3baebbeaa1c58250fcbf447d8bc introduced a 10x performance degredation for the benchmark:function_loading micro-benchmark in the Puppet suite. I used git bisect to identify that particular commit.

      For 2211c64 (parent of 01db61e)

      C:\source\puppet [(2211c64...) +68 ~0 -0 !]> bundle exec rake benchmark:function_loading
      Document generation not available without yard. cannot load such file -- yard
      mkdir -p C:/Users/ADMINI~1/AppData/Local/Temp/function_loading20160816-3900-13bswjg
      DL is deprecated, please use Fiddle
                       user     system      total        real
      Run 1       10.296000   2.125000  12.421000 ( 13.608853)
      Run 2        6.766000   1.031000   7.797000 (  8.749664)
      Run 3        8.078000   0.797000   8.875000 (  9.692987)
      Run 4        8.578000   0.891000   9.469000 ( 10.468348)
      Run 5        9.094000   0.844000   9.938000 ( 10.937080)
      Run 6        9.984000   0.844000  10.828000 ( 12.093286)
      Run 7       10.610000   0.844000  11.454000 ( 12.629397)
      Run 8       12.109000   0.843000  12.952000 ( 13.953529)
      Run 9       12.250000   1.032000  13.282000 ( 14.311859)
      Run 10      13.438000   0.828000  14.266000 ( 15.218068)
      > total:   101.203000  10.079000 111.282000 (121.663071)
      > avg:      10.120300   1.007900  11.128200 ( 12.166307)
      

      For 01db61e

      C:\source\puppet [(01db61e...) +68 ~0 -0 !]> bundle exec rake benchmark:function_loading
      Document generation not available without yard. cannot load such file -- yard
      mkdir -p C:/Users/ADMINI~1/AppData/Local/Temp/function_loading20160816-49440-1miin4u
      DL is deprecated, please use Fiddle
                       user     system      total        real
      Run 1      114.015000  14.797000 128.812000 (142.752140)
      Run 2      156.344000  14.547000 170.891000 (184.167353)
      Run 3      201.969000  13.188000 215.157000 (228.894914)
      Run 4      245.016000  12.828000 257.844000 (271.543842)
      Run 5      288.734000  13.312000 302.046000 (315.255168)
      

      NOTE: The above was terminated prematurely given the obvious trend of a further decrease of performance on each run.

      Current stable HEAD (13d208b2e935151c6b8cf77e8bf4ad1b45fc72c5) as of 8/16/2016

      Some corrective efforts appear to have already been taken to address this performance problem (and it's compounding effect), but the current performance of latest code is still 3x slower than it was on versions of Puppet prior to 4.6.0.

      C:\source\puppet [stable...origin/stable +68 ~0 -0 !]> bundle exec rake benchmark:function_loading
      Document generation not available without yard. cannot load such file -- yard
      mkdir -p C:/Users/ADMINI~1/AppData/Local/Temp/function_loading20160816-10824-x58pt6
      DL is deprecated, please use Fiddle
                       user     system      total        real
      Run 1       29.266000   5.266000  34.532000 ( 35.654881)
      Run 2       24.453000   3.969000  28.422000 ( 29.694252)
      Run 3       25.563000   3.890000  29.453000 ( 30.420901)
      Run 4       26.453000   3.797000  30.250000 ( 31.485412)
      Run 5       27.219000   3.563000  30.782000 ( 31.846841)
      Run 6       30.390000   3.328000  33.718000 ( 34.311228)
      Run 7       29.047000   4.000000  33.047000 ( 33.956486)
      Run 8       28.938000   4.219000  33.157000 ( 34.445907)
      Run 9       36.890000   3.828000  40.718000 ( 42.357819)
      Run 10      59.438000   3.484000  62.922000 ( 64.077050)
      > total:   317.657000  39.344000 357.001000 (368.250777)
      > avg:      31.765700   3.934400  35.700100 ( 36.825078)
      

      Current master HEAD (50a20b2 as of 8/16/2016

      Master is also exhibiting the slow-down

      :\source\puppet [master...origin/master +68 ~0 -0 !]> bundle exec rake benchmark:function_loading
      Document generation not available without yard. cannot load such file -- yard
      mkdir -p C:/Users/ADMINI~1/AppData/Local/Temp/function_loading20160816-26064-1n6bwhd
      DL is deprecated, please use Fiddle
                       user     system      total        real
      Run 1       29.563000   4.938000  34.501000 ( 35.829908)
      Run 2       24.343000   3.890000  28.233000 ( 29.436511)
      Run 3       25.782000   3.860000  29.642000 ( 30.963654)
      Run 4       25.609000   4.109000  29.718000 ( 30.983185)
      Run 5       26.328000   3.703000  30.031000 ( 31.282978)
      Run 6       28.610000   3.875000  32.485000 ( 33.677326)
      Run 7       26.296000   3.797000  30.093000 ( 31.264224)
      Run 8       28.157000   3.812000  31.969000 ( 33.066140)
      Run 9       53.203000   3.547000  56.750000 ( 58.190413)
      Run 10      53.531000   4.188000  57.719000 ( 58.727435)
      > total:   321.422000  39.719000 361.141000 (373.421774)
      > avg:      32.142200   3.971900  36.114100 ( 37.342177)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ethan Ethan Brown
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support