[PDB-1227] Test new table structure and fact path/value GC at scale Created: 2015/02/10 Updated: 2015/04/14 Resolved: 2015/03/31
|Fix Version/s:||PDB 2.3.1|
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
|Sprint:||PuppetDB 2015-02-25, PuppetDB 2015-03-11, PuppetDB 2015-03-25|
Once we have
We should be able to use Ken Barber's setup for this
|Comment by Wyatt Alt [ 2015/03/27 ]|
This patch looks great based on my performance testing.
On stable we can reproduce the fact_paths problem with 30,000 simulated nodes checking in every 30 minutes with a random percentage of 10. The problem does not occur on stable at 20,000 nodes, so the breaking point is somewhere between 20k and 30k.
The CPUs on the postgres server are pegged on deletes more or less as soon as commands switch from being processed with add-facts! to update-facts!, i.e once the database stores 30k certnames. The gc issues at this point will cause the queue size to explode and all kinds of problems to occur.
This patch on the other hand can handle 30k and 50k nodes without breaking a sweat – in fact, using the -i and -n settings we have not been able to submit factsets fast enough to back up the queue. When we set -n to 100000, only 17 commands per second were being submitted, which means the benchmark tool was underperforming (or our metrics are wrong). Queue size remains at or around 0 during both additions and updates for any of those numbers.
The -n and -N flags of the benchmark tool allow you to submit commands unthrottled. Doing this will back up the queue both here and on stable, but is not a realistic scenario.
Migration of 30,000 nodes of data from stable to this patch took two and a half minutes, which is fine. All in all this seems to completely solve our problem.