Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2319

puppetserver 6.0.0 dies with java.lang.OutOfMemoryError: Metaspace

    Details

    • Template:
      PUP Bug Template
    • Team:
      Server
    • Method Found:
      Needs Assessment
    • Release Notes:
      Known Issue
    • Release Notes Summary:
      Hide
      When upgrading from Puppet Server 5 using JRuby 1.7 (9k was optional in those releases), Server-side gems (those installed manually with `puppetserver gem` command or using the puppetserver_gem package provider) may need to be updated to work with the newer JRuby. In most cases gems do not have APIs that break when upgrading from the Ruby versions implemented between JRuby 1.7 and JRuby 9k, so there may be no necessary updates, but users should be aware. Two notable exceptions are that the autosign gem should be 0.1.3 or later and yarddoc must be 0.9 or later.

      Users outside of lab environments should increase ReservedCodeCache to 512m under normal load. Those users with 6-12 JRuby instances (or a max-requests-per-instance value significantly less than 100k) should run with a ReservedCodeCache of 1G, 12+ JRuby instances in a single server may require 2G or more.

      Similar caveats regarding scaling ReservedCodeCache may apply if users are managing MaxMetaspace.
      Show
      When upgrading from Puppet Server 5 using JRuby 1.7 (9k was optional in those releases), Server-side gems (those installed manually with `puppetserver gem` command or using the puppetserver_gem package provider) may need to be updated to work with the newer JRuby. In most cases gems do not have APIs that break when upgrading from the Ruby versions implemented between JRuby 1.7 and JRuby 9k, so there may be no necessary updates, but users should be aware. Two notable exceptions are that the autosign gem should be 0.1.3 or later and yarddoc must be 0.9 or later. Users outside of lab environments should increase ReservedCodeCache to 512m under normal load. Those users with 6-12 JRuby instances (or a max-requests-per-instance value significantly less than 100k) should run with a ReservedCodeCache of 1G, 12+ JRuby instances in a single server may require 2G or more. Similar caveats regarding scaling ReservedCodeCache may apply if users are managing MaxMetaspace.
    • QA Risk Assessment:
      Needs Assessment

      Description

      We have a Puppet server with max-active-instances set to 2, running on a VM with 4GiB of memory and 4 CPUs. We set JAVA_ARGS in /etc/sysconfig/puppserver to:

      JAVA_ARGS="-Xms1560m -Xmx1560m -XX:MaxMetaspaceSize=256m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"
      

      This is a beta Puppet server and always runs the latest puppetserver package available. It serves 37 clients that want to live on the bleeding edge of Puppet.

      This morning, the server automatically upgraded to puppetserver 6.0.0. After it did so, the server refused to start, logging:

      Sep 19 12:33:57 server.example.org puppetserver: # java.lang.OutOfMemoryError: Metaspace
      

      Upping MaxMetaspaceSize from 256m to 512m resolved this issue, permitting the server to start successfully.

      Almost certainly, this change in resource requirements was due to the move to JRuby 9k. Although the move to JRuby 9k is duly noted in the Puppet Server 6.0.0 release notes, there is no warning whatsoever that Java-related memory tuning parameters may need to be adjusted as a consequence of this change, in either the release notes or the known issues:

      https://puppet.com/docs/puppetserver/6.0/release_notes.html
      https://puppet.com/docs/puppet/6.0/known_issues.html

      The implications of the move to JRuby 9k need to be documented.

        Attachments

          Activity

            jsd-sla-details-panel

              People

              • Assignee:
                michelle.fredette Michelle Fredette
                Reporter:
                ralston James Ralston
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: