Uploaded image for project: 'Puppet Communications Protocol'
  1. Puppet Communications Protocol
  2. PCP-585

pxp-agent may fail to connect to broker following reboot of Fedora host

    XMLWordPrintable

    Details

    • Template:
    • Team:
      Dumpling
    • Story Points:
      3
    • Release Notes:
      Not Needed

      Description

      https://jenkins.puppetlabs.com/view/puppet-agent%20suite%20pipelines/job/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/35/SLAVE_LABEL=beaker,TEST_TARGET=fedora24-32a/testReport/junit/(root)/tests/restart_host_run_puppet_rb/

      Begin tests/restart_host_run_puppet.rb
       
      C94777 - Ensure pxp-agent functions after agent host restart
        
        * Ensure each agent host has pxp-agent service running and enabled
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:30:50$ puppet resource service pxp-agent ensure=stopped
            Notice: /Service[pxp-agent]/ensure: ensure changed 'running' to 'stopped'
            service { 'pxp-agent':
              ensure => 'stopped',
            }
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) executed in 1.35 seconds
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:30:51$ puppet config print ssldir
            /etc/puppetlabs/puppet/ssl
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) executed in 0.94 seconds
          localhost $ scp /tmp/beaker20160906-10733-lkoyzw fedora24-32-1:/etc/puppetlabs/pxp-agent/pxp-agent.conf {:ignore => }
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:30:53$ puppet resource service pxp-agent ensure=running enable=true
            Notice: /Service[pxp-agent]/ensure: ensure changed 'stopped' to 'running'
            service { 'pxp-agent':
              ensure => 'running',
              enable => 'true',
            }
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) executed in 1.39 seconds
        
        * restart each agent
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:30:56$ /sbin/shutdown -r now
            Warning: ssh channel on fedora24-32-1 received exception post command execution IOError - closed stream
            Warning: ssh.close: connection is already closed, no action needed
            ssh connection to fedora24-32-1 has been terminated
          
          vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) executed in 0.24 seconds
          
          * wait until pxp-agent is back up and associated on vzeypakzrc9l48b.delivery.puppetlabs.net
            
            vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:31:06$ #<Beaker::Command:0x00000004bab950>
              Trying command 30 times.
            .        Attempting ssh connection to vzeypakzrc9l48b.delivery.puppetlabs.net, user: root, opts: {:config=>false, :paranoid=>false, :auth_methods=>["publickey"], :port=>22, :forward_agent=>true, :keys=>["~/.ssh/id_rsa-acceptance"], :user_known_hosts_file=>"/var/lib/jenkins/.ssh/known_hosts", :keepalive=>true, :user=>"root"}
              Warning: Try 1 -- Host vzeypakzrc9l48b.delivery.puppetlabs.net unreachable: Errno::ETIMEDOUT - Connection timed out - connect(2) for "vzeypakzrc9l48b.delivery.puppetlabs.net" port 22
              Warning: Trying again in 3 seconds
              Attempting ssh connection to vzeypakzrc9l48b.delivery.puppetlabs.net, user: root, opts: {:config=>false, :paranoid=>false, :auth_methods=>["publickey"], :port=>22, :forward_agent=>true, :keys=>["~/.ssh/id_rsa-acceptance"], :user_known_hosts_file=>"/var/lib/jenkins/.ssh/known_hosts", :keepalive=>true, :user=>"root"}
            
            vzeypakzrc9l48b.delivery.puppetlabs.net (fedora24-32-1) 17:32:14$ #<Beaker::Command:0x00000004bab950> ostensibly successful.
            Minitest::Assertion: Agent vzeypakzrc9l48b.delivery.puppetlabs.net should be associated with pcp-broker following host reboot
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:53
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:47
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:33
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:33
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:32
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/tests/restart_host_run_puppet.rb:3
            /var/lib/jenkins/workspace/platform_puppet-agent_intn-van-sys_suite-daily-pxp-agent-master/SLAVE_LABEL/beaker/TEST_TARGET/fedora24-32a/pxp-agent/acceptance/.bundle/gems/gems/beaker-2.50.0/bin/beaker:6
            Begin teardown
            End teardown
            tests/restart_host_run_puppet.rb failed in 169.39 seconds

      Initially I just had this as a comment on QENG-4371 with the most likely cause being a slow VM; but on another look at the restart_host_run_puppet.rb test case, it is quite patient and allows up to 60 seconds (starting from when the pxp-agent service is confirmed to be running again) for pxp-agent to be associated with the broker again. Either the pxp-agent startup is very slow, or it is faulty.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              james.stocks James Stocks
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support