Uploaded image for project: 'Modules'
  1. Modules
  2. MODULES-8398

puppet_agent: puppet services are not re-started if a Windows upgrade fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: puppet_agent
    • Labels:
    • CS Priority:
      Needs Priority
    • QA Risk Assessment:
      Needs Assessment

      Description

      When used to automate puppet-agent upgrades on Windows nodes, the puppet_agent module stops Puppet services before running msiexec to perform the upgrade and assumes the MSI will re-start the services. If msiexec fails for some reason, the node will be left with all Puppet management de-activated. This means an alternate management method must be used to restore Puppet services (hopefully WinRM is enabled...).

      Reproduction Case

      • Install PE 2018.1.5 and Bolt 1.x on a master node.
      • Configure the master with Windows packages for PE 2018.1.4:

      # Fix a hard-coded PE version in the template used to generate install.ps1
      sed -i'' "s/current/<%= scope['pe_version'] %>/" /opt/puppetlabs/puppet/modules/pe_repo/templates/install.ps1.erb
      mkdir -p /opt/puppetlabs/server/data/packages/public/2018.1.4
      puppet apply -e 'pe_repo::windows { "windows-x86_64": arch => "x64", agent_version => "5.5.6", pe_version => "2018.1.4"}'
      

      • Install the 2018.1.4 agent on a Windows node:

      # NOTE: Change to the hostname of your windows node
      win_node=tbzxn8pmyrvdir7.delivery.puppetlabs.net
       
      bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "[Net.ServicePointManager]::ServerCertificateValidationCallback = {\$true}
        \$webClient = New-Object System.Net.WebClient
        \$webClient.DownloadFile('https://$(hostname -f):8140/packages/2018.1.4/install.ps1', \$env:temp + '/install.ps1')
        powershell -File (\$env:temp + '/install.ps1')" \
        --user Administrator --password
       
      puppet cert sign -a
      bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "puppet agent -t" \
        --user Administrator --password
      

      • Install the puppet_agent module:

      puppet module install puppetlabs-puppet_agent --version 1.7.0
      

      • Apply the puppet_agent class to upgrade the Windows node, but specify an invalid package source to fail msiexec:

      puppet apply <<'EOF'
      pe_node_group { 'Agent Upgrade':
        parent          => 'PE Agent',
        refresh_classes => true,
        pinned          => puppetdb_query('inventory[certname] { facts.osfamily ~ "windows" }').map |$row| {
          $row['certname']
        },
        classes         => {
          'puppet_agent' => {
            'package_version' => $facts['aio_agent_version'],
            'source' => 'http://bad-url.invalid',
          }
        }
      }
      EOF
      

      • Check puppet service status and run the upgrade:

      bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "Get-Service puppet,pxp-agent
        puppet agent --onetime --no-daemonize --verbose" \
        --user Administrator --password
      

      Outcome

      Prior to the upgrade attempt, Puppet services are running. The attempt its self signals success:

      # bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "Get-Service puppet,pxp-agent
        puppet agent --onetime --no-daemonize --verbose" \
        --user Administrator --password
      Please enter your password:
      Started on tbzxn8pmyrvdir7.delivery.puppetlabs.net...
      Finished on tbzxn8pmyrvdir7.delivery.puppetlabs.net:
        STDOUT:
       
          Status   Name               DisplayName
          ------   ----               -----------
          Running  puppet             Puppet Agent
          Running  pxp-agent          Puppet PXP Agent
          Info: Using configured environment 'production'
          Info: Retrieving pluginfacts
          Info: Retrieving plugin
          Info: Retrieving locales
          Info: Loading facts
          Info: Caching catalog for tbzxn8pmyrvdir7.delivery.puppetlabs.net
          Info: Applying configuration version '1545331851'
          Notice: /Stage[main]/Puppet_agent::Windows::Install/File[C:\Users\ADMINI~1\AppData\Local\Temp\install_puppet.bat]/content: content changed '{md5}5c2f67ffd40c171186eab48e1e5f109a' to '{md5}c44f817f71083e7d4926409ae31cbf36'
          Notice: /Stage[main]/Puppet_agent::Windows::Install/Exec[install_puppet.bat]/returns: executed successfully
          Notice: /Stage[main]/Puppet_agent::Windows::Install/Exec[fix inheritable SYSTEM perms]/returns: executed successfully
          Notice: Applied catalog in 3.17 seconds
       
       
      Successful on 1 node: winrm://tbzxn8pmyrvdir7.delivery.puppetlabs.net
      Ran on 1 node in 20.20 seconds
      

      However, services are stopped afterwards and the puppet-agent package has not been upgraded:

      # bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "Get-Service puppet,pxp-agent
        puppet resource package 'Puppet Agent (64-bit)'" \
        --user Administrator --password
      Please enter your password:
      Started on tbzxn8pmyrvdir7.delivery.puppetlabs.net...
      Finished on tbzxn8pmyrvdir7.delivery.puppetlabs.net:
        STDOUT:
       
          Status   Name               DisplayName
          ------   ----               -----------
          Stopped  puppet             Puppet Agent
          Stopped  pxp-agent          Puppet PXP Agent
          package { 'Puppet Agent (64-bit)':
            ensure => '5.5.6',
          }
       
      Successful on 1 node: winrm://tbzxn8pmyrvdir7.delivery.puppetlabs.net
      Ran on 1 node in 5.29 seconds
      

      The upgrade log shows that msiexec failed:

      # bolt command run --nodes "winrm://${win_node}" --no-ssl \
        "Get-Content (\$env:temp + '/puppet*installer.log')" \
        --user Administrator --password
      Please enter your password:
      Started on tbzxn8pmyrvdir7.delivery.puppetlabs.net...
      Finished on tbzxn8pmyrvdir7.delivery.puppetlabs.net:
        STDOUT:
          === Verbose logging started: 12/20/2018  18:51:04  Build type: SHIP UNICODE 5.00.10011.00  Calling process: C:\Windows\system32\msiexec.exe ===
          MSI (c) (B4:44) [18:51:04:710]: Resetting cached policy values
          MSI (c) (B4:44) [18:51:04:710]: Machine policy value 'Debug' is 0
          MSI (c) (B4:44) [18:51:04:710]: ******* RunEngine:
                     ******* Product: http://bad-url.invalid
                     ******* Action:
                     ******* CommandLine: **********
          MSI (c) (B4:44) [18:51:04:710]: Client-side and UI is none or basic: Running entire install on the server.
          MSI (c) (B4:44) [18:51:04:710]: Grabbed execution mutex.
          MSI (c) (B4:44) [18:51:04:742]: Cloaking enabled.
          MSI (c) (B4:44) [18:51:04:742]: Attempting to enable all disabled privileges before calling Install on Server
          MSI (c) (B4:44) [18:51:04:742]: Incrementing counter to disable shutdown. Counter after increment: 0
          MSI (s) (9C:FC) [18:51:04:742]: Running installation inside multi-package transaction http://bad-url.invalid
          MSI (s) (9C:FC) [18:51:04:742]: Grabbed execution mutex.
          MSI (s) (9C:D0) [18:51:04:757]: Resetting cached policy values
          MSI (s) (9C:D0) [18:51:04:757]: Machine policy value 'Debug' is 0
          MSI (s) (9C:D0) [18:51:04:757]: ******* RunEngine:
                     ******* Product: http://bad-url.invalid
                     ******* Action:
                     ******* CommandLine: **********
          MSI (s) (9C:D0) [18:51:04:757]: Using WinHttp to perform URL download
          MSI (s) (9C:D0) [18:51:04:757]: File path is a URL. Downloading file. . .
          MSI (s) (9C:D0) [18:51:04:757]: Msi WinHttp: Performing auto proxy detection
          MSI (s) (9C:D0) [18:51:04:757]: MSI WinHttp: Proxy Settings Proxy: (none) | Bypass: (none) | AccessType: 0
          MSI (s) (9C:D0) [18:51:04:757]: Download of URL resource http://bad-url.invalid/ failed with last error 12007
          MSI (s) (9C:D0) [18:51:04:757]: MainEngineThread is returning 2
          MSI (s) (9C:FC) [18:51:04:757]: User policy value 'DisableRollback' is 0
          MSI (s) (9C:FC) [18:51:04:757]: Machine policy value 'DisableRollback' is 0
          MSI (s) (9C:FC) [18:51:04:757]: Incrementing counter to disable shutdown. Counter after increment: 0
          MSI (s) (9C:FC) [18:51:04:757]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
          MSI (s) (9C:FC) [18:51:04:757]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
          MSI (s) (9C:FC) [18:51:04:757]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1
          MSI (c) (B4:44) [18:51:04:757]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1
          MSI (c) (B4:44) [18:51:04:757]: MainEngineThread is returning 2
          === Verbose logging stopped: 12/20/2018  18:51:04 ===
       
      Successful on 1 node: winrm://tbzxn8pmyrvdir7.delivery.puppetlabs.net
      Ran on 1 node in 0.60 seconds
      

      Expected Outcome

      • "Exec[install_puppet.bat]/returns: executed successfully" should not be displayed if msiexec fails to upgrade the package.
      • If msiexec fails to upgrade the installation, the puppet_agent module should try to re-start any services that were running prior to the upgrade attempt.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sean.mcdonald Sean McDonald
                Reporter:
                chuck Charlie Sharpsteen
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support