When Puppet Server handles an API request for a catalog, an instance of Puppet::Parser::Compiler is created to generate the catalog from inputs supplied by the client. This Compiler instance is discarded after the API request finishes and should be destroyed by the Java garbage collector as it contains a lot of state that is not useful once the request is completed.
Puppet Server is retaining a handful of these Compiler instances after their requests finish which bloats the memory footprint of the service and can lead to out of memory errors if the retained Compilers happen to have built particularly large catalogs.
Reproduction Case
- Install PE 2019.8 on CentOS 7 with the following debug configuration in pe.conf:
"puppet_enterprise::master::puppetserver::jruby_max_active_instances": 1
|
"puppet_enterprise::profile::master::java_args": {
|
"Djruby.reify.classes": "=true"
|
}
|
- Install the development tools for pe-java:
yum install -y pe-java11-devel
|
- Run puppet a couple of times:
puppet agent -t
|
puppet agent -t
|
- Check the number of Puppet::Parser::Compiler instances retained in Puppet Server memory:
sudo -u pe-puppet $(find /opt/puppetlabs/server/apps -name jmap) -histo:live $(systemctl show -p MainPID pe-puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'
|
Outcome
After running the agent twice, the puppetserver service is retaining two Compiler instances in memory:
# sudo -u pe-puppet $(find /opt/puppetlabs/server/apps -name jmap) -histo:live $(systemctl show -p MainPID pe-puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'
|
|
6172: 2 64 rubyobj.Puppet.Parser.Compiler
|
Expected Outcome
No compiler instances should be retained as the compiler is discarded at the end of the request and the -histo:live flag forces a full garbage collection.
Bug
Major
