need your help in understanding and tuning puppet masters. We have 6 puppet masters (in two datacenters, using f5 to load balance those (least conn):
puppet version: puppet-agent-1.10.10-1.el7.x86_64
Centos7: CentOS Linux release 7.4.1708 (Core)
5000 nodes, 90 modules, runinterval is 2 hours.
max-active-instances: 7, heap size is set to 20GB
CPU spikes max to 40% (which i think is OK), however ram is constantly running at 96%.
I am concerned about memory utilization as it is constantly high. Is there any other tuning i missed? I looked at max-requests-per-instance but do not think that is useful in my case as I do not see OOM. when puppet agent runs on nodes (client), cpu on that hosts spikes, but catalog is applied within 40 sec. And even though total number of modules is 90, only 10 modules are used/applied.
Would appreciate any help on this.