Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2021

Cannot allocate memory for 'gpg' with JRuby 9k and "large" JVM heap size

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Normal
    • Resolution: Done
    • SERVER 5.0.0, SERVER 5.1.0, SERVER 5.1.3
    • None
    • Puppet Server

    • Ubuntu 16.04, Oracle Java 1.8.0_144 and OpenJDK 1.8.0_131

    • Froyo
    • Needs Assessment
    • Needs Assessment

    Description

      We use hiera-eyaml and hiera-eyaml-gpg along with the ruby_gpg Gem (since gpgme cannot work within JRuby). Secret data in Hiera is encrypted with eyaml using GPG (our team consists of 15 people so data is encrypted with 15 public keys).

      When using JRuby 9k and setting the JVM heap size for Puppet Server to more than about 2GB I can reliably reproduce the following error on every Puppet agent node whose catalog will contain (decrypted) secret data:

      2017-10-31 18:34:34,933 ERROR [qtp865617117-104] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Cannot allocate memory - gpg at /etc/puppetlabs/code/environments/production/site/role/manifests/loadbalancer.pp:3:3 on node lb01.example.at
      		 
      2017-10-31 18:34:34,934 ERROR [qtp865617117-104] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Cannot allocate memory - gpg at /etc/puppetlabs/code/environments/production/site/role/manifests/loadbalancer.pp:3:3 on node lb01.example.at
      		 
      org/jruby/RubyProcess.java:1566:in `spawn'
      		 
      org/jruby/RubyKernel.java:1517:in `spawn'
      		 
      uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:206:in `popen_run'
      		 
      uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:102:in `popen3'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/ruby_gpg-0.3.2/lib/ruby_gpg.rb:59:in `run_command'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/ruby_gpg-0.3.2/lib/ruby_gpg.rb:52:in `decrypt_string'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-gpg-0.6/lib/hiera/backend/eyaml/encryptors/gpg.rb:167:in `decrypt'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:15:in `encrypted_value'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:80:in `create_enc_token'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:100:in `create_token'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:90:in `create_token'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/parser.rb:71:in `parse_scanner'
      		 
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/parser.rb:36:in `parse'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:85:in `decrypt'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:65:in `decrypt_value'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:40:in `eyaml_lookup_key'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:43:in `block in dispatch'
      		 
      org/jruby/RubyKernel.java:1120:in `catch'

      This happens with both Oracle Java 1.8.0_144 and OpenJDK 1.8.0_131.

      The contents of /etc/default/puppetserver, where the heap size is set and JRuby 9k is enabled:

      JAVA_BIN="/usr/bin/java"
      		 
      JAVA_ARGS="-Xms4G -Xmx4G -XX:MaxMetaspaceSize=512m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"
      		 
      JAVA_ARGS_CLI=""
      		 
      USER="puppet"
      		 
      GROUP="puppet"
      		 
      INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver"
      		 
      CONFIG="/etc/puppetlabs/puppetserver/conf.d"
      		 
      JRUBY_JAR="/opt/puppetlabs/server/apps/puppetserver/jruby-9k.jar"
      		 
      BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/"
      		 
      SERVICE_STOP_RETRIES=60
      		 
      START_TIMEOUT=300
      		 
      RELOAD_TIMEOUT=120

      As soon as I comment the JRUBY_JAR line and restart Puppet Server, so that it uses JRuby 1.7, this error is no longer reproducible.

      This is concerning because a decently powerful Puppet Server machine with quite a few CPUs/cores will definitely require a larger JVM heap size than 2GB (see https://puppet.com/docs/puppetserver/5.1/tuning_guide.html#jvm-heap-size), otherwise the JVM gets bogged down garbage collecting very quickly. In this particular case we have a standalone Puppet Server with 4 vCPUs and 8GB RAM. According to the tuning guide the JVM's heap size should be least 2560MB. But even such a moderate heap size results in the above error when JRuby 9k is enabled. We also have Puppet Server instances with 12 vCPUs and 16GB RAM (not yet productive) and they would need even more heap space, probably around 7GB.

      I suppose this is not really a bug in Puppet Server but in JRuby, and it probably has to do with the way JRuby 9k changed the ways it does process and IO handling. From what little I could gather researching this issue the problem may be that the gpg processes used to decrypt the secret data produce "too much" output for JRuby to handle properly, but why this is related to the JVM's heap size I don't know.

      I have yet to try reducing the number of public keys with which the secret data is encrypted, in an effort to decrease the size of the ciphertexts. Maybe that really makes a difference.

      I am reporting this problem here because JRuby 9k will probably be enabled by default in future Puppet releases and then this will probably hit more people than just us (I don't believe we are the only organisation using hiera-eyaml and hiera-eyaml-gpg with Puppet Server). Maybe there is something that can be done to mitigate this issue in Puppet Server?

      Some more info:

      # /opt/puppetlabs/server/bin/puppetserver gem list
      		 
       
      		 
      *** LOCAL GEMS ***
      		 
       
      		 
      deep_merge (1.1.1)
      		 
      fast_gettext (1.1.0)
      		 
      gettext (3.2.2)
      		 
      gettext-setup (0.28)
      		 
      hiera-eyaml (2.1.0)
      		 
      hiera-eyaml-gpg (0.6)
      		 
      highline (1.6.21)
      		 
      hocon (1.2.5)
      		 
      jar-dependencies (0.2.6)
      		 
      jruby-openssl (0.9.19 java)
      		 
      json (1.8.0 java)
      		 
      locale (2.1.2)
      		 
      rake (10.1.0)
      		 
      rdoc (4.1.2)
      		 
      ruby_gpg (0.3.2)
      		 
      semantic_puppet (0.1.3)
      		 
      text (1.3.1)
      		 
      trollop (2.1.2)

      hiera.yaml:

      ---
      		 
      version: 5
      		 
      defaults:
      		 
        datadir: hieradata
      		 
        data_hash: yaml_data
      		 
       
      		 
      hierarchy:
      		 
        - name: Eyaml-encrypted data
      		 
          datadir: hieradata
      		 
          lookup_key: eyaml_lookup_key
      		 
          options:
      		 
            extension: yaml
      		 
            encrypt_method: gpg
      		 
            gpg_gnupghome: "/etc/puppetlabs/puppet/keys/gpg"
      		 
            gpg_always_trust: true
      		 
          paths:
      		 
            - "nodes/%{trusted.domain}/%{trusted.hostname}.yaml"
      		 
            - "services/%{facts.stage}/%{facts.service}.yaml"
      		 
            - "services/%{facts.service}.yaml"
      		 
            - "domains/%{trusted.domain}.yaml"
      		 
            - "stages/%{facts.stage}.yaml"
      		 
            - common.yaml
      		 
            - users.yaml
      		 
            - repos.yaml
      		 
            - sshdata.yaml
      		 
            - ssldata/common.yaml
      		 
       
      		 
        - name: Per-node data
      		 
          path: "nodes/%{trusted.domain}/%{trusted.hostname}.yaml"
      		 
       
      		 
        - name: Service data per stage
      		 
          path: "services/%{facts.stage}/%{facts.service}.yaml"
      		 
       
      		 
        - name: Service data
      		 
          path: "services/%{facts.service}.yaml"
      		 
       
      		 
        - name: Domain data
      		 
          path: "domains/%{trusted.domain}.yaml"
      		 
       
      		 
        - name: Stage data
      		 
          path: "stages/%{facts.stage}.yaml"
      		 
       
      		 
        - name: OS data per OS release (codename)
      		 
          path: "os/%{facts.os.name}/%{facts.os.distro.codename}.yaml"
      		 
       
      		 
        - name: OS data
      		 
          path: "os/%{facts.os.name}.yaml"
      		 
       
      		 
        - name: Common data
      		 
          path: common.yaml
      		 
       
      		 
        - name: User account and group data
      		 
          path: users.yaml
      		 
       
      		 
        - name: Package repositories
      		 
          path: repos.yaml
      		 
       
      		 
        - name: SSH key data
      		 
          path: sshdata.yaml
      		 
       
      		 
        - name: TLS/SSL certificate, key and other related data
      		 
          path: ssldata/common.yaml
      		 
       
      		 
        - name: File data, potentially Eyaml-encrypted
      		 
          datadir: hierafiles
      		 
          lookup_key: file_eyaml_lookup_key
      		 
          options:
      		 
            interpolate: false
      		 
            extension: enc
      		 
            encrypt_method: gpg
      		 
            gpg_gnupghome: "/etc/puppetlabs/puppet/keys/gpg"
      		 
            gpg_always_trust: true
      		 
          paths:
      		 
            - "nodes/%{trusted.domain}/%{trusted.hostname}.d"
      		 
            - "services/%{facts.stage}/%{facts.service}.d"
      		 
            - "services/%{facts.service}.d"
      		 
            - "domains/%{trusted.domain}.d"
      		 
            - "stages/%{facts.stage}.d"
      		 
            - common.d
      		 
            - users.d
      		 
            - repos.d
      		 
            - sshdata.d
      		 
            - ssldata/common.d

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-2167 Work with community to use Puppet::Util::Execution.execute in hiera-eyaml[-gpg]

          • Normal - Regular issue, some loss of functionality under specific circumstances.
          • Closed

          CI Blocker - An issue that is preventing (a) CI job(s) from running successfully SERVER-2164 "Cannot allocate memory" in travis

          • Normal - Regular issue, some loss of functionality under specific circumstances.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              daff Andreas Ntaflos
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support