[SERVER-1697] Manual configuration of per-environment pools for isolation Created: 2016/12/19  Updated: 2018/09/26

Status: Open
Project: Puppet Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Normal
Reporter: Jeremy Barlow Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to SERVER-94 Environment Isolation Closed
relates to SERVER-1789 Characterize JRuby performance problems Closed
Epic Name: Isolate 1 environment per JRuby
Template:
Team/s:
Server
Epic Status: Done
QA Risk Assessment: Needs Assessment

 Description   

This "sub-epic" is an offshoot from one of the comments on SERVER-94, from here: https://tickets.puppetlabs.com/browse/SERVER-94?focusedCommentId=329083&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-329083. Text below is copied pretty liberally from that comment:

This "sub-epic" would expand our JRuby pooling to allow a user to configure a pooling "strategy". The default strategy would, for some period of time, continue to be the current "single-pool" strategy under which the same JRuby instance can be exercised by multiple environments.

This "sub-epic" would introduce a new manual/explicit pooling "strategy". In this approach, the user would be responsible for setting up their configuration to list what the most common (and performance-sensitive) environments were, and tell us how many JRubies to keep around dedicated to those environments. Then, there'd be a sort of 'overflow' pool for uncommonly used environments. The user could also configure the size of this pool. Whenever we got an agent request from an environment that isn't on the 'dedicated' list, we'd use this overflow pool, which we'll flush instances out of using an LRU strategy. We can keep a couple of JRubies warm to make the process of cycling things in and out of this overflow pool faster.

The config might look something like this (credit to Nan Liu as this was his idea, probably like 3 years ago now):

{
  jruby-puppet:
    {pooling-strategy: configured-environments
     dedicated-environments: {
        production: {pool-size: 4}
        test: {pool-size: 4}
        staging: {pool-size: 4}
     }
     overflow-pool-size: 4
     pre-warmed-pool-size: 2
}

The upside of this approach is that the implementation should be fairly simple and easy to debug, and the configurability should be sufficient for tuning for most users needs. The downside is that it will require some knowledge and configuration on the part of the user. But, since it should be faster to get out the door, and should serve as a valuable stepping stone towards any other strategies we want to implement.



 Comments   
Comment by Thomas Mueller [ 2016/12/23 ]
  • I would like to be able to specify name globbing (*, ?) characters to match environment names. Normally all my dev/test branches have a common prefix (eg. test_*, feature_*) which would maybe enable me to limit the usage on these.
  • If I can configure dedicated jruby pools it could make sense to be able to restart a named jruby pool with the Puppet Server: Admin API: JRuby Pool API. For example the test environment could require to be restarted way more frequently than production.
Comment by Jeremy Barlow [ 2017/01/04 ]

Thomas Mueller Thanks much for the feedback!

If I can configure dedicated jruby pools it could make sense to be able to restart a named jruby pool with the Puppet Server: Admin API: JRuby Pool API. For example the test environment could require to be restarted way more frequently than production.

We've had similar discussions about this and agree with the suggestion. I created a separate ticket under this epic, SERVER-1705, to capture this work specifically.

I would like to be able to specify name globbing (, ?) characters to match environment names. Normally all my dev/test branches have a common prefix (eg. test_, feature_*) which would maybe enable me to limit the usage on these.

The way we had envisioned this would work is that for the "dedicated-environments", we would allocate fixed capacity for the named environments at startup. We had not planned on doing any specific discovery on disk as to whether the specified environments already exist at the time the service is being started up - or if they are created after the server is already up and running.

If you had somewhat short-lived environments mapped to test_* or feature_* branches, instances for those could be spun up out of the overflow-pool as needed but rotated out as they are no longer being actively used. The maximum number of these could still be capped per the size of the overflow-pool.

Do you think this would be sufficient for your needs?

/CC Eric Sorenson for awareness and any feedback you might want to provide around this use case.

Comment by Thomas Mueller [ 2017/01/09 ]

Jeremy Barlow I think I got it now. I intepreted overflow-pool-size to be used by environments not specified dedicated-environments AND by requests that can not satisfied by configured pool-size in dedicated-environments ("the overflow").

If I configure a dedicated jruby pool for a environment only this pool will serve requests for this environment. Is this correct? If this is true, wouldn't the term fallback-pool-size (or other-environments-pool-size) be more descriptive?

Regarding the recycling of the overflow pool (or fallback pool) I could imaging unclear situations like it works on one agent (served by already recycled process since last environment update) vs. it works not (not recycled since last environment update). I'd like to see Puppet Server: Admin API: JRuby Pool to support recycling the overflow-pool too. So I can call the API after updating a overflow-pool environment.

Comment by Jeremy Barlow [ 2017/01/10 ]

I think I got it now. I intepreted overflow-pool-size to be used by environments not specified dedicated-environments AND by requests that can not satisfied by configured pool-size in dedicated-environments ("the overflow").

If I configure a dedicated jruby pool for a environment only this pool will serve requests for this environment. Is this correct? If this is true, wouldn't the term fallback-pool-size (or other-environments-pool-size) be more descriptive?

Yes, that's right. Considering the example from the ticket description...

{
  jruby-puppet:
    {pooling-strategy: configured-environments
     dedicated-environments: {
        production: {pool-size: 4}
        test: {pool-size: 4}
        staging: {pool-size: 4}
     }
     overflow-pool-size: 4
     pre-warmed-pool-size: 2
}

... any requests that need to use the "production" environment would draw instances from the "subpool" created for "dedicated-environments.production". If the "dedicated-environments.production" subpool were empty at the point the request were made, the request would block until an instance is returned to the "dedicated-environments.production" pool - and would not draw from the subpool created for overflow/fallback. Likewise, if a request were made for the "dev" environment - which is not one of the configured "dedicated-environments" - the request would only draw instances from the overflow/fallback "subpool".

I think the names you mentioned - fallback-pool-size or other-environments-pool-size - may be a better fit, thanks for the suggestion. I expect we may tweak these names a bit before the implementation is complete. We'll be working with our UX team here on the final presentation of this configuration. /CC Lori Landesman for visibility on this feedback.

I'd like to see Puppet Server: Admin API: JRuby Pool to support recycling the overflow-pool too. So I can call the API after updating a overflow-pool environment.

To keep the comments organized, I think it would be best to discuss the feature to provide more surgical JRuby pool flushing on SERVER-1705. I'll copy your previous comment over to that ticket and respond there.

Thanks again for the feedback!

Comment by Trevor Vaughan [ 2018/02/06 ]

I would like to propose that the production environment get its own dedicated-environment by default.

I also think that the 'globbing' proposed by Thomas Mueller is a good idea.

It might be worth taking a look at how Passenger solved this issue as well and drag that into the discussion thread for analysis.

Generated at Mon Dec 16 01:41:08 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.