Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-4703

Optimize future_parser? checks for faster catalog production

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PUP 3.7.5, PUP 3.8.1
    • Fix Version/s: PUP 3.8.2
    • Component/s: None
    • Environment:

      Puppetlabs's puppet RPM
      RHEL6
      Ruby1.8.7 or 2.2.0
      Agent + Master on Passenger setup

    • Template:
    • Story Points:
      1
    • Sprint:
      Language 2015-06-10, Language 2015-06-24
    • Release Notes:
      New Feature

      Description

      UPDATE

      A suboptimal implementation was changed to reduce the number of calls the catalog production process makes to check if running with parser=future in the current environment. The older logic checked this once per copied resource attribute. The optimized logic only once per resource. This affects all users irrespective of if running with parser = future or not.

      The amount of speed up depends on the number of resources in the catalog, and the ratio of attributes per resource. As noted in comments a speed up (measured slightly differently) saved 7 seconds on a 17 second run (a reduction by 42%).

      ORIGINAL

      While launching a one-time run I've seen that upgrading from 3.6.2 to 3.7.5 almost doubles the config retrieval time of the same catalog (~850 Ressources).
      Some code instrumentation with puppet's Profiler shows that the compile terminus launches no less than 13000 Puppet.future_parser? in the ~12sec compilation time, and that's quite a number...

      Most of them comes from the conversion of the catalog to resources

      Puppet::Resource::Catalog.to_catalog

      It doesn't seems to affect 4.x.

      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): AGGREGATE PROFILING RESULTS:
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): ----------------------------
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler: 37.337394 ms (196 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> compile: 12.525465 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> compile -> production: 12.525465 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> compile -> production -> puppet64.xxxxxxx: 12.525465 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_generators: 5.927503 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> iterate_on_generators: 5.926884 ms (4 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_definitions: 5.925818 ms (4 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_resource: 5.884373 ms (176 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_main: 0.623718 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> finish_catalog: 0.308283 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> find_node: 0.135445 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> find_facts: 0.037252 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> init_server_facts: 0.020561 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> create_settings_scope: 0.015438 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_ast_node: 0.004912 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> set_node_params: 0.000756 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_node_classes: 0.000714 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): compiler -> evaluate_collections: 0.000272 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions: 19.882293 ms (159 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> include: 19.799214 ms (125 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> regsubst: 0.079596 ms (19 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> defined: 0.001082 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> realize: 0.001005 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> hiera: 0.000879 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): functions -> versioncmp: 0.000517 ms (12 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http: 18.64083 ms (3 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http -> POST: 18.225792 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http -> POST -> /production/catalog/puppet64.xxxxxxx: 18.225792 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http -> v1_render: 0.414972 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http -> v1_render -> Puppet::Network::Format[pson]: 0.414972 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): http -> v1_response: 6.6e-05 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): future_parser: 5.80201499999996 ms (13926 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): indirector: 5.081889 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): indirector -> filter: 5.081889 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): indirector -> filter -> catalog: 5.081889 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): indirector -> filter -> catalog -> puppet64.xxxxxx: 5.081889 ms (1 calls)
      Fri Jun 05 11:29:18 -0400 2015 Puppet (debug): ----------------------------
      

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  aimnor Romain Fihue
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Zendesk Support