[SERVER-1697] Manual configuration of per-environment pools for isolation Created: 2016/12/19 Updated: 2018/09/26
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
|Epic Name:||Isolate 1 environment per JRuby|
|QA Risk Assessment:||Needs Assessment|
This "sub-epic" is an offshoot from one of the comments on
This "sub-epic" would expand our JRuby pooling to allow a user to configure a pooling "strategy". The default strategy would, for some period of time, continue to be the current "single-pool" strategy under which the same JRuby instance can be exercised by multiple environments.
This "sub-epic" would introduce a new manual/explicit pooling "strategy". In this approach, the user would be responsible for setting up their configuration to list what the most common (and performance-sensitive) environments were, and tell us how many JRubies to keep around dedicated to those environments. Then, there'd be a sort of 'overflow' pool for uncommonly used environments. The user could also configure the size of this pool. Whenever we got an agent request from an environment that isn't on the 'dedicated' list, we'd use this overflow pool, which we'll flush instances out of using an LRU strategy. We can keep a couple of JRubies warm to make the process of cycling things in and out of this overflow pool faster.
The config might look something like this (credit to Nan Liu as this was his idea, probably like 3 years ago now):
The upside of this approach is that the implementation should be fairly simple and easy to debug, and the configurability should be sufficient for tuning for most users needs. The downside is that it will require some knowledge and configuration on the part of the user. But, since it should be faster to get out the door, and should serve as a valuable stepping stone towards any other strategies we want to implement.
|Comment by Thomas Mueller [ 2016/12/23 ]|
|Comment by Jeremy Barlow [ 2017/01/04 ]|
Thomas Mueller Thanks much for the feedback!
We've had similar discussions about this and agree with the suggestion. I created a separate ticket under this epic,
The way we had envisioned this would work is that for the "dedicated-environments", we would allocate fixed capacity for the named environments at startup. We had not planned on doing any specific discovery on disk as to whether the specified environments already exist at the time the service is being started up - or if they are created after the server is already up and running.
If you had somewhat short-lived environments mapped to test_* or feature_* branches, instances for those could be spun up out of the overflow-pool as needed but rotated out as they are no longer being actively used. The maximum number of these could still be capped per the size of the overflow-pool.
Do you think this would be sufficient for your needs?
/CC Eric Sorenson for awareness and any feedback you might want to provide around this use case.
|Comment by Thomas Mueller [ 2017/01/09 ]|
Jeremy Barlow I think I got it now. I intepreted overflow-pool-size to be used by environments not specified dedicated-environments AND by requests that can not satisfied by configured pool-size in dedicated-environments ("the overflow").
If I configure a dedicated jruby pool for a environment only this pool will serve requests for this environment. Is this correct? If this is true, wouldn't the term fallback-pool-size (or other-environments-pool-size) be more descriptive?
Regarding the recycling of the overflow pool (or fallback pool) I could imaging unclear situations like it works on one agent (served by already recycled process since last environment update) vs. it works not (not recycled since last environment update). I'd like to see Puppet Server: Admin API: JRuby Pool to support recycling the overflow-pool too. So I can call the API after updating a overflow-pool environment.
|Comment by Jeremy Barlow [ 2017/01/10 ]|
Yes, that's right. Considering the example from the ticket description...
... any requests that need to use the "production" environment would draw instances from the "subpool" created for "dedicated-environments.production". If the "dedicated-environments.production" subpool were empty at the point the request were made, the request would block until an instance is returned to the "dedicated-environments.production" pool - and would not draw from the subpool created for overflow/fallback. Likewise, if a request were made for the "dev" environment - which is not one of the configured "dedicated-environments" - the request would only draw instances from the overflow/fallback "subpool".
I think the names you mentioned - fallback-pool-size or other-environments-pool-size - may be a better fit, thanks for the suggestion. I expect we may tweak these names a bit before the implementation is complete. We'll be working with our UX team here on the final presentation of this configuration. /CC Lori Landesman for visibility on this feedback.
To keep the comments organized, I think it would be best to discuss the feature to provide more surgical JRuby pool flushing on
Thanks again for the feedback!
|Comment by Trevor Vaughan [ 2018/02/06 ]|
I would like to propose that the production environment get its own dedicated-environment by default.
I also think that the 'globbing' proposed by Thomas Mueller is a good idea.
It might be worth taking a look at how Passenger solved this issue as well and drag that into the discussion thread for analysis.