Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-1723

Puppetserver gem uses all the entropy and then hangs

    Details

    • Template:
    • Team:
      Systems Engineering
    • Story Points:
      1
    • Sprint:
      SE 2017-02-08, SE 2017-02-22
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      By default the Puppetserver CLI subcommands used /dev/random for entropy. On systems that have limited sources of entropy, such as VMs, these subcommands could drain the entropy pool rapidly and then would block while the pool gradually refilled. For all practical purposes /dev/urandom provides enough entropy for our needs, so the Puppetserver CLI subcommands have been updated to use /dev/urandom over /dev/random.
      Show
      By default the Puppetserver CLI subcommands used /dev/random for entropy. On systems that have limited sources of entropy, such as VMs, these subcommands could drain the entropy pool rapidly and then would block while the pool gradually refilled. For all practical purposes /dev/urandom provides enough entropy for our needs, so the Puppetserver CLI subcommands have been updated to use /dev/urandom over /dev/random.
    • QA Risk Assessment:
      Needs Assessment

      Description

      Installing gems in the puppetserver is incredibly, amazingly slow. When deploying new masters for my personal use, runs take many minutes upon provisioning, the majority of which is the installation of two gems. I can't see why it should take this long. Here are some examples:

      These were run with machines in SLICE so network speeds should be a non-issue

      [root@master centos]# time /opt/puppetlabs/server/bin/puppetserver gem install cri
      Fetching: colored-1.2.gem (100%)
      Successfully installed colored-1.2
      Fetching: cri-2.7.1.gem (100%)
      Successfully installed cri-2.7.1
      2 gems installed
       
      real    3m22.958s
      user    0m38.800s
      sys     0m0.701s
       
      [root@master centos]# time /opt/puppetlabs/puppet/bin/gem install cri
      Fetching: cri-2.7.1.gem (100%)
      Successfully installed cri-2.7.1
      Parsing documentation for cri-2.7.1
      Installing ri documentation for cri-2.7.1
      Done installing documentation for cri after 0 seconds
      1 gem installed
       
      real    0m3.152s
      user    0m1.623s
      sys     0m0.105s
      

      In this case installing the 2 gems was 67x slower than it was using puppet's gem command. Even if it involved a restart of the Puppetserver it shouldn't be anywhere near that slow:

      [root@master centos]# time systemctl restart pe-puppetserver
       
      real    0m35.280s
      user    0m0.009s
      sys     0m0.014s
      

      After a bit of digging we can see a heap of calls like this:

      0.050128 futex(0x7ff9a80ccc28, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000016>
      [pid 30514]      0.000118 futex(0x7ff9a80ccc54, FUTEX_WAIT_BITSET_PRIVATE, 1, {614513, 99881915}, ffffffff) = -1 ETIMEDOUT (Connection timed out) <0.050022>
      

      The beginning of these calls happening coincides with the available entropy in /dev/random being completely depleted. It then keeps waiting until there is about ~60 entropy available and depletes it again, this happens two or three times and the everything continues from where it left off.

      While this is a pain in the ass for installing gems it makes you wonder; what else is hanging because java is depleting /dev/random instead of using /dev/urandom

      After installing haveged to generate more entropy we have a phenomenal increase ins speed:

      [root@master centos]# time /opt/puppetlabs/server/bin/puppetserver gem install cri
      Fetching: colored-1.2.gem (100%)
      Successfully installed colored-1.2
      Fetching: cri-2.7.1.gem (100%)
      Successfully installed cri-2.7.1
      2 gems installed
       
      real    0m11.302s
      user    0m38.741s
      sys     0m0.606s
      

      Then turning off the

      {haveged}

      daemon and running again (It doesn't even have to install this time, it's already there):

      [root@master centos]# time /opt/puppetlabs/server/bin/puppetserver gem install cri
      Successfully installed cri-2.7.1
      1 gem installed
       
      real    0m54.294s
      user    0m39.811s
      sys     0m0.578s
      

        Attachments

          Activity

            People

            • Assignee:
              jeremy.barlow Jeremy Barlow
              Reporter:
              dylan.ratcliffe Dylan Ratcliffe
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support