Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10958

Load everything in an environment upfront before first compile



    • New Feature
    • Status: Closed
    • Normal
    • Resolution: Won't Do
    • None
    • None
    • None
    • Needs Assessment


      Synchronizing the methods on loaders in multithreaded mode results in deadlocks when environment caching is enabled. In this case, the same set of loaders is being used in multiple threads, and while resolving a value, loaders call into other loaders to see if they have the value. If two mutually-connected loaders are trying to load something, and wind up asking each other at the same time if the other has their value, they can deadlock. See the stack traces here.

      Analyzing what would need to be locked to allow safe loading, it doesn't appear that it is possible to avoid this situation. So instead, we want to look into doing loading everything up front when an environment is first used for a compile. We can synchronize this process, and then subsequent compiles against the environment won't need to do any loading, so it's safe to not synchronize the loaders at all.

      Ideally, this preloading would be fairly contained, so that we could easily opt into it, and use it only when multithreaded and environment caching are both in use. It's a fairly expensive operation that we wouldn't want to run all the time.

      Investigation doc: https://docs.google.com/document/d/1xVxv4zDcT0eT44dN39Bqr85QwU_I9IsGfBXRnSMdyUc/edit?usp=sharing

      The most promising thing I've found so far is that loaders currently implement a discover method, that can search for and load all examples of a certain class of Puppet object: functions, types, tasks, and plans. Calling these before first compile gets us a good bit of the way, but they don't work for some things, notably resource types. So part of this work might be extending that system, if possible, to support other classes of loadable things.

      This effort needs more breaking down and better scoping, and might end up needing its own epic.


        Issue Links



              Unassigned Unassigned
              maggie Maggie Dreyer
              0 Vote for this issue
              9 Start watching this issue



                Zendesk Support