[MCO-736] Reduce number of permanent subscriptions using the ActiveMQ connector Created: 2015/11/18 Updated: 2017/02/02 Resolved: 2017/02/02
|Fix Version/s:||MCO 2.10.0|
|Reporter:||Alberto Rodriguez Peon||Assignee:||Unassigned|
|Labels:||community, docs_reviewed, maintenance|
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
|Release Notes:||New Feature|
|Release Notes Summary:||Add an activemq.agents_multiplex option to enable using a single destination /topic/<collection>.agents for all agents as opposed to creating a topic subscription for each agent. This is an optimization that may make sense when running collectives with several thousands of nodes in order to reduce the number of subscriptions in the message broker. Using the option is a trade-off between increasing network traffic by delivering messages to all nodes - and letting them select messages they care about - versus increasing work in the message broker to handle large numbers of subscriptions.|
First of all, in case Jira is not the right place to report this, let me know where should I put it.
In our MCollective setup, we have around 20k hosts and we plan to scale it up to at least 30k.
If we understand correctly the internals of MCollective, each host creates one queue subscription (to /queue/<collective>.nodes) but also one topic subscription per agent (to /topic/<collective>.<agent>.agent). We are using 10 agents at the moment (discovery, filemgr, mgrep, nrpe, package, process, puppet, rpcutil, service and shell) so we have a total of 11 subscriptions per host. With 30k hosts, this would be 330k subscriptions, which is very high.
What we propose to drastically reduce the number of subscriptions: use a single destination (/topic/<collective>.agents) for all the agents and use a header key (like "mc_agent") to identify which agent should get the message. In fact, this is similar to what is already done with the "nodes" queue, using an "mc_identity" header for targeted messages.
This simple change would divide the number of topics used by 10 (in our case, from 1500 to 150) and the number of permanent subscriptions by 5.5 (from 330k to 60k). This would simplify a lot what ActiveMQ has to do for MCollective.
Add an activemq.agents_multiplex option to enable using a single destination /topic/<collection>.agents for all agents as opposed to creating a topic subscription for each agent.
|Comment by Richard Clamp [ 2015/11/18 ]|
Do you have evidence that links a high number of subscriptions to any performance impact before we go and consider doing this behavioural change?
Without supporting data or guiding documentation this seems to be a specification for a solution (work towards reducing number of subscriptions) without a quantified and clearly stated problem.
|Comment by R.I.Pienaar [ 2015/11/18 ]|
I'd def agree the plugin as is isnt optimised for 30k hosts, but anyway - seems inconceivable such a large single collective/cluster would be of any realistic use or function on a basic level.
The current design is optimised for cases where you have an uneven distribution of plugins - you have some mail servers, some db server, some other servers and groups would have different sets of plugins. The current optimise for that.
In a large collective you'd almost certainly end up with other sources of discovery truth and you can almost completely forego the topics all together since they wouldn't scale at all but what would work is very dependant on the use case, traffic patterns, collective layout and clustering choices.
Almost certainly would be worth experimenting and trying a custom flavour of the activemq plugin with your own tweaks. It would be really easy to do this iirc you can inherit from the activemq plugin and override the make_target and headers_for methods for whatever scheme you wish - I think that would be sufficient from a experimentation stage. Would be pretty hard to come up with reliable comparative benchmarks I think though
|Comment by Michael Smith [ 2016/05/18 ]|
Nicholas Fagerlund this looks to have a docs component. I can probably do that myself, as it should follow a similar pattern to other options, and appears to be maintained in the repo. /cc Garrett Guillotte
|Comment by Nicholas Fagerlund [ 2016/05/19 ]|
Michael Smith Awesome. Yeah, what you see in the repo is what goes up on the site; we just publish whatever's in the master branch when the site builds.
|Comment by Michael Smith [ 2016/05/19 ]|
This could potentially be interesting in the RabbitMQ connector as well. Since it's not done here, I'll update the ticket to reflect that it only applies to ActiveMQ.
|Comment by Steve Traylen [ 2016/05/26 ]|
|Comment by Kenn Hussey [ 2016/08/04 ]|
|Comment by Michael Smith [ 2016/08/04 ]|
|Comment by Kenn Hussey [ 2016/12/27 ]|
Michael Smith did this make it through CI yet?
|Comment by Kenn Hussey [ 2017/01/13 ]|
Michael Smith please provide release notes for this issue.