Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2021

Cannot allocate memory for 'gpg' with JRuby 9k and "large" JVM heap size



    • Bug
    • Status: Closed
    • Normal
    • Resolution: Done
    • SERVER 5.0.0, SERVER 5.1.0, SERVER 5.1.3
    • None
    • Puppet Server
    • Ubuntu 16.04, Oracle Java 1.8.0_144 and OpenJDK 1.8.0_131

    • Froyo
    • Needs Assessment
    • Needs Assessment


      We use hiera-eyaml and hiera-eyaml-gpg along with the ruby_gpg Gem (since gpgme cannot work within JRuby). Secret data in Hiera is encrypted with eyaml using GPG (our team consists of 15 people so data is encrypted with 15 public keys).

      When using JRuby 9k and setting the JVM heap size for Puppet Server to more than about 2GB I can reliably reproduce the following error on every Puppet agent node whose catalog will contain (decrypted) secret data:

      2017-10-31 18:34:34,933 ERROR [qtp865617117-104] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Cannot allocate memory - gpg at /etc/puppetlabs/code/environments/production/site/role/manifests/loadbalancer.pp:3:3 on node lb01.example.at
      2017-10-31 18:34:34,934 ERROR [qtp865617117-104] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Cannot allocate memory - gpg at /etc/puppetlabs/code/environments/production/site/role/manifests/loadbalancer.pp:3:3 on node lb01.example.at
      org/jruby/RubyProcess.java:1566:in `spawn'
      org/jruby/RubyKernel.java:1517:in `spawn'
      uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:206:in `popen_run'
      uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:102:in `popen3'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/ruby_gpg-0.3.2/lib/ruby_gpg.rb:59:in `run_command'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/ruby_gpg-0.3.2/lib/ruby_gpg.rb:52:in `decrypt_string'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-gpg-0.6/lib/hiera/backend/eyaml/encryptors/gpg.rb:167:in `decrypt'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:15:in `encrypted_value'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:80:in `create_enc_token'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:100:in `create_token'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/encrypted_tokens.rb:90:in `create_token'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/parser.rb:71:in `parse_scanner'
      /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml/parser/parser.rb:36:in `parse'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:85:in `decrypt'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:65:in `decrypt_value'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/eyaml_lookup_key.rb:40:in `eyaml_lookup_key'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:43:in `block in dispatch'
      org/jruby/RubyKernel.java:1120:in `catch'

      This happens with both Oracle Java 1.8.0_144 and OpenJDK 1.8.0_131.

      The contents of /etc/default/puppetserver, where the heap size is set and JRuby 9k is enabled:

      JAVA_ARGS="-Xms4G -Xmx4G -XX:MaxMetaspaceSize=512m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"

      As soon as I comment the JRUBY_JAR line and restart Puppet Server, so that it uses JRuby 1.7, this error is no longer reproducible.

      This is concerning because a decently powerful Puppet Server machine with quite a few CPUs/cores will definitely require a larger JVM heap size than 2GB (see https://puppet.com/docs/puppetserver/5.1/tuning_guide.html#jvm-heap-size), otherwise the JVM gets bogged down garbage collecting very quickly. In this particular case we have a standalone Puppet Server with 4 vCPUs and 8GB RAM. According to the tuning guide the JVM's heap size should be least 2560MB. But even such a moderate heap size results in the above error when JRuby 9k is enabled. We also have Puppet Server instances with 12 vCPUs and 16GB RAM (not yet productive) and they would need even more heap space, probably around 7GB.

      I suppose this is not really a bug in Puppet Server but in JRuby, and it probably has to do with the way JRuby 9k changed the ways it does process and IO handling. From what little I could gather researching this issue the problem may be that the gpg processes used to decrypt the secret data produce "too much" output for JRuby to handle properly, but why this is related to the JVM's heap size I don't know.

      I have yet to try reducing the number of public keys with which the secret data is encrypted, in an effort to decrease the size of the ciphertexts. Maybe that really makes a difference.

      I am reporting this problem here because JRuby 9k will probably be enabled by default in future Puppet releases and then this will probably hit more people than just us (I don't believe we are the only organisation using hiera-eyaml and hiera-eyaml-gpg with Puppet Server). Maybe there is something that can be done to mitigate this issue in Puppet Server?

      Some more info:

      # /opt/puppetlabs/server/bin/puppetserver gem list
      *** LOCAL GEMS ***
      deep_merge (1.1.1)
      fast_gettext (1.1.0)
      gettext (3.2.2)
      gettext-setup (0.28)
      hiera-eyaml (2.1.0)
      hiera-eyaml-gpg (0.6)
      highline (1.6.21)
      hocon (1.2.5)
      jar-dependencies (0.2.6)
      jruby-openssl (0.9.19 java)
      json (1.8.0 java)
      locale (2.1.2)
      rake (10.1.0)
      rdoc (4.1.2)
      ruby_gpg (0.3.2)
      semantic_puppet (0.1.3)
      text (1.3.1)
      trollop (2.1.2)


      version: 5
        datadir: hieradata
        data_hash: yaml_data
        - name: Eyaml-encrypted data
          datadir: hieradata
          lookup_key: eyaml_lookup_key
            extension: yaml
            encrypt_method: gpg
            gpg_gnupghome: "/etc/puppetlabs/puppet/keys/gpg"
            gpg_always_trust: true
            - "nodes/%{trusted.domain}/%{trusted.hostname}.yaml"
            - "services/%{facts.stage}/%{facts.service}.yaml"
            - "services/%{facts.service}.yaml"
            - "domains/%{trusted.domain}.yaml"
            - "stages/%{facts.stage}.yaml"
            - common.yaml
            - users.yaml
            - repos.yaml
            - sshdata.yaml
            - ssldata/common.yaml
        - name: Per-node data
          path: "nodes/%{trusted.domain}/%{trusted.hostname}.yaml"
        - name: Service data per stage
          path: "services/%{facts.stage}/%{facts.service}.yaml"
        - name: Service data
          path: "services/%{facts.service}.yaml"
        - name: Domain data
          path: "domains/%{trusted.domain}.yaml"
        - name: Stage data
          path: "stages/%{facts.stage}.yaml"
        - name: OS data per OS release (codename)
          path: "os/%{facts.os.name}/%{facts.os.distro.codename}.yaml"
        - name: OS data
          path: "os/%{facts.os.name}.yaml"
        - name: Common data
          path: common.yaml
        - name: User account and group data
          path: users.yaml
        - name: Package repositories
          path: repos.yaml
        - name: SSH key data
          path: sshdata.yaml
        - name: TLS/SSL certificate, key and other related data
          path: ssldata/common.yaml
        - name: File data, potentially Eyaml-encrypted
          datadir: hierafiles
          lookup_key: file_eyaml_lookup_key
            interpolate: false
            extension: enc
            encrypt_method: gpg
            gpg_gnupghome: "/etc/puppetlabs/puppet/keys/gpg"
            gpg_always_trust: true
            - "nodes/%{trusted.domain}/%{trusted.hostname}.d"
            - "services/%{facts.stage}/%{facts.service}.d"
            - "services/%{facts.service}.d"
            - "domains/%{trusted.domain}.d"
            - "stages/%{facts.stage}.d"
            - common.d
            - users.d
            - repos.d
            - sshdata.d
            - ssldata/common.d


        Issue Links



              Unassigned Unassigned
              daff Andreas Ntaflos
              0 Vote for this issue
              9 Start watching this issue



                Zendesk Support