Uploaded image for project: 'Facter'
  1. Facter
  2. FACT-3088

EC2 token hangs for minutes on non-AWS machines

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • FACT 3.14.20
    • None
    • Facter 3
    • None
    • Needs Assessment
    • Needs Assessment

    Description

      Who found the bug?

      Open Source Puppet user

      Where was the bug found?

      • System: KVM virtual machines not running in AWS
      • Version: N/A
      • Operating system(s): CentOS 7.9, Rocky Linux 8.4
      • Puppet version: puppet-agent-6.25.0-1.el8.x86_64

      What is malfunctioning?

      Puppet agent 6.24.0 uses facter 3.14.19, Puppet agent 6.25.0 uses facter 3.14.20

      Diff between facter versions: https://github.com/puppetlabs/facter/compare/3.14.19...3.14.20

      Facter added the ability to use a IMDSv2 token when querying ec2 metadata.  However, this implementation does not include an explicit timeout and non-AWS KVM virtual machines it will hang for minutes.

      For comparison, running "time puppet facts --debug" using puppet-agent 6.24.0 takes 4.6 seconds, and the same command on puppet-agent 6.25.0 takes 131.7 seconds - over 2 minutes to gather facts!

       

       

      # puppet-agent 6.24.0
      $ puppet facts --debug
       
      Debug: Facter: executing command: /opt/puppetlabs/puppet/bin/virt-what
      Debug: Facter: kvm
      Debug: Facter: completed processing output: closing child pipes.
      Debug: Facter: process exited with status code 0.
      Debug: Facter: fact "is_virtual" has resolved to true.
      Debug: Facter: fact "virtual" has resolved to "kvm".
      Debug: Facter: not running under a Azure instance.
      Debug: Facter: resolving EC2 facts.
      Debug: Facter: querying EC2 instance metadata at http://169.254.169.254/latest/meta-data/.
      Debug: Facter: requesting http://169.254.169.254/latest/meta-data/.
      Debug: Facter: Trying 169.254.169.254:80...
      Debug: Facter: Connection timed out after 600 milliseconds
      Debug: Facter: Closing connection 0
      Debug: Facter: EC2 facts are unavailable: not running under an EC2 instance or EC2 is not responding in a timely manner.
      Debug: Facter: resolving cloud facts.
      Debug: Facter: resolving cloud fact
      Debug: Facter: resolving GCE facts.
      Debug: Facter: not running under a GCE instance.

      # puppet-agent 6.25.0
      $ puppet facts --debug
       
      Debug: Facter: executing command: /opt/puppetlabs/puppet/bin/virt-what
      Debug: Facter: kvm
      Debug: Facter: completed processing output: closing child pipes.
      Debug: Facter: process exited with status code 0.
      Debug: Facter: fact "is_virtual" has resolved to true.
      Debug: Facter: fact "virtual" has resolved to "kvm".
      Debug: Facter: not running under a Azure instance.
      Debug: Facter: resolving EC2 facts.
      Debug: Facter: requesting IMDSv2 token at http://169.254.169.254/latest/api/token.
      Debug: Facter: requesting http://169.254.169.254/latest/api/token.
      Debug: Facter: Trying 169.254.169.254:80...
      Debug: Facter: connect to 169.254.169.254 port 80 failed: Connection timed out
      Debug: Facter: Failed to connect to 169.254.169.254 port 80: Connection timed out
      Debug: Facter: Closing connection 0
      Debug: Facter: EC2 IMDSv2 endpoint is unavailable
      Debug: Facter: querying EC2 instance metadata at http://169.254.169.254/latest/meta-data/.
      Debug: Facter: requesting http://169.254.169.254/latest/meta-data/.
      Debug: Facter: Trying 169.254.169.254:80...
      Debug: Facter: Connection timed out after 600 milliseconds
      Debug: Facter: Closing connection 1
      Debug: Facter: EC2 facts are unavailable: not running under an EC2 instance or EC2 is not responding in a timely manner.
      Debug: Facter: resolving cloud facts.
      Debug: Facter: resolving cloud fact
      Debug: Facter: resolving GCE facts.
      Debug: Facter: not running under a GCE instance.

       

       

      What does success look like?

      Fix the regression introduced in facter 3.14.20 by using the same 600ms timeout when requesting a token.

      How will success be validated?

      On a KVM virtual machine outside of AWS running puppet-agent 6.25, executing the command "puppet facts" should take less than 5 seconds.

      Should anyone be contacted after this is fixed?

      Open Source Puppet 6.x needs to be updated with this fix.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              knack-marc Marc Schweikert
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Zendesk Support