[MCO-58] Document approaches to more scalable deploys Created: 2013/12/10  Updated: 2017/03/07  Resolved: 2017/03/07

Status: Closed
Project: MCollective
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Normal
Reporter: redmine.exporter Assignee: Unassigned
Resolution: Done Votes: 0
Labels: community, redmine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:

 Description   

Currently the state of art regarding complex or large mcollective sites is just to mesh all the activemqs together. This does not work well and is not a good idea given the design of ActiveMQ clustering however it seems the obvious thing so everyone does it.

We need to test, document, benchmark etc a scalable mcollective deploy strategy based on ActiveMQ



 Comments   
Comment by Doug Forster [ 2016/07/27 ]

I recently submitted https://github.com/puppetlabs/marionette-collective/pull/387 to update to include some things that helped me to get a larger deployment more stable.

Comment by Ben Ford [ 2016/08/02 ]

Reid Vandewiele some time ago, you did a lot of research into mco scaling. Was this one of the options you tried?

Comment by Reid Vandewiele [ 2016/08/05 ]

Ben Ford pendingMessageLimitStrategy is not something I've done anything with.

Doug Forster, how large is your deployment? Ballpark is fine. I'd like to make sure we know more specifically what we're talking about when we suggest that this is something that can be used to "get a larger deployment more stable." In that vein it would also be useful to know what the ActiveMQ architecture you're using looks like, how many brokers, configured as hub-spoke or other, etc. Thanks in advance!

Comment by Doug Forster [ 2016/08/05 ]

Reid Vandewiele My deployment is about 1200 servers we expect this to grow north of 3k. We have them pointed to an f5 loadbalancer with 6 ActiveMQ servers in 2 datacenters. Each datacenter is pointed to their own ActiveMQ server. The information I got from a mailing list post helped solve my issues of multiple ActiveMQ servers having trouble replicating traffic.

I got that advice from a post on the mailing list. http://grokbase.com/t/gg/mcollective-users/163xya3hgq/activemq-topology

Comment by Reid Vandewiele [ 2016/08/09 ]

Doug Forster your responded 1,200 servers, but based on the mailing list comments I think you meant 12,000? Please confirm. Thanks!

Comment by Doug Forster [ 2016/08/09 ]

Reid Vandewiele That is correct. My cluster is only 1,200 but looking to grow north of 3,000. I was mentioning you might want to reach out to the guy on the mailing list or archives about the modifications.

Comment by Michael Smith [ 2017/03/07 ]

This has had a number of improvements. We also provide built-in hub-and-spoke configuration in PE. https://docs.puppet.com/mcollective/deploy/middleware/activemq.html#settings-for-networks-of-brokers documents a hub-and-spoke configuration. I'm going to close this unless anyone has other specific suggestions on improving documentation.

http://choria.io also documents and provides tools for a simpler deployment using https://nats.io.

Generated at Sun Aug 18 00:37:06 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.