[PUP-9092] External process termination handling rate (including "exec" resource) is slow Created: 2018/08/30  Updated: 2018/10/09  Resolved: 2018/10/08

Status: Closed
Project: Puppet
Component/s: Performance, Windows
Affects Version/s: PUP 4.10.5
Fix Version/s: PUP 5.5.7, PUP 6.0.2

Type: Bug Priority: Major
Reporter: Aristarkh Zagorodnikov Assignee: Josh Cooper
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Team: Coremunity
Sprint: Platform Core KANBAN
Method Found: Needs Assessment
Release Notes: Bug Fix
Release Notes Summary: Previously puppet took at least one second to execute external processes, even if the process completed more quickly than that. This fix significantly decreases the time it takes the puppet agent to apply a catalog if the catalog contains a large number of exec resources and each child process completes in less than 1 second.
QA Risk Assessment: Needs Assessment

 Description   

Puppet Version: 4.10.5+
Puppet Server Version: irrelevant
OS Name/Version: Any supported Windows OS

Executing external processes by puppet on Windows can be very slow, compared to the execution time of the process itself, if the execution time is below 1 second. When applying manifests that contain many "exec" resources, it may take many minutes to apply. When analyzing logs, I took note that even simple "reg query" process took 1.02/1.03 second to run. Windows doesn't spawn processes that slow, I wrote simple example in C# which showed that my machine is capable of executing about 140 of such commands per second (no parallelism, waiting for exit of each process before spawning another).
I did a little bit of digging and found this piece of code. This code waits for at least one second, unless the process terminated immediately (which is rarely feasible).
Replacing 'sleep(1)' with 'sleep(0)' is the simplest fix, but is not too efficient from a CPU consumption standpoint, so I propose a different fix in a PR#7033.

I did a little more digging and found out that this 'sleep(1)' was added to properly handle signals, I did additional testing ("exec"-ed a slow process and tried to Ctrl-C an interactive puppet run), and it worked.

Desired Behavior: Process termination is handled immediately

Actual Behavior: Processes are considered to be running at least for 1 second, even if they exit immediately

 It would be great if this gets into 4.10.z release train, since at least our company is not yet ready for 5.x.



 Comments   
Comment by Josh Cooper [ 2018/10/02 ]

Merged to 5.5.x in https://github.com/puppetlabs/puppet/commit/3acaa21d499d356b427aeeaa4e0c0a1d4b41403f

Comment by Josh Cooper [ 2018/10/08 ]

passed CI for 5.5.x and 6.0.x

Generated at Sat Oct 20 20:20:20 PDT 2018 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.