Puppet Windows - Upgrading from 3.7.5 causes Ruby.exe/Rubyw.exe to go missing

When upgrading Puppet using Puppet on Windows (which isn't really recommended, see MODULES-2040), it can cause file locking.

This is due to an incorrect version of Ruby being released with Puppet 3.7.5 (it was Ruby 2.1.5, caused by 8c73637a1e2e, which is known to have compatibility issues with Puppet < 4). This was fixed later in 1173b42101da8.

Note: This doesn't affect PE versions of 3.x agents for Windows as they were using a different process for identifying dependencies to pull in.

This was discovered on 7/17/2015 after a message was sent about upgrade issues. We subsequently yanked the Windows versions of Puppet 3.7.5 and notified puppet-dev/puppet-users indicating actions users should take.

We recommended users do one of the following to upgrade Puppet:

• Use a means other than Puppet to upgrade Puppet
• Look at implementing parts of MODULES-2040 to perform the upgrades
• Possibly use the puppetversion module to perform the upgrade.

We found out on 7/24 that using the puppetversion module to perform the upgrades also caused this behavior to occur. We are looking at other possibilities, like using psexec and/or winexe to perform an in-place reinstall for an affected 3.8.1 (or future) upgrade.

Mitigation Steps

We're updating the defaults for 3.x to use tags to mitigate this from future issues and will be examining changes to the build_defaults.yaml more closely. Puppet v4+ is unaffected since we switched to using puppet-agent to build the MSIs (which uses tags).

We may evaluate more mitigation steps such as returning the versions of all components as part of smoke testing.

Reproduction steps

1. Install Puppet 3.7.5 (I used x86 and noticed others with this issue were upgrading the x86 version).
2. Use a puppet script to apply the upgrade (I used puppet apply but run as an agent would produce the same outcome):

   package { 'Puppet':

   ensure => "3.8.1",

   source => "https://downloads.puppetlabs.com/windows/puppet-3.8.1.msi",

   provider => windows,

   }

3. Try running puppet, you should get the following message:

   C:\Users\Administrator>puppet

   'ruby' is not recognized as an internal or external command,

   operable program or batch file.

Workaround for non-upgraded systems

To upgrade from 3.7.5, you should employ one of the following means:

• Use a means other than Puppet to upgrade Puppet
  • You may need to run the upgrade twice unless manually performing the upgrade.
• Look at implementing parts of MODULES-2040 to perform the upgrades

Workaround [Possible Solution] for affected systems (no ruby.exe)

Use PsExec or Winexe to rerun the install for Puppet MSI. The command should be msiexec /qn /i https://downloads.puppetlabs.com/windows/puppet-3.8.1.msi /l*v "c:\PuppetInstall.log" along with any MSI properties you may need to pass - see http://docs.puppetlabs.com/guides/install_puppet/install_windows.html#msi-properties

Note: Both PsExec and Winexe are known to send passwords in the clear. This is important to know if you are planning to run over an insecure network.

For PSExec the command must be run from a Windows machine and it should look something like:

psexec \\%i -u <domain account> -p <password> msiexec.exe /qn /l*v C:\Windows\temp\puppet.log /i https://downloads.puppetlabs.com/windows/puppet-3.7.4.msi

liamjbennett has mentioned in the comments how you can loop the command.

For winexe (running from Linux), the command should look similar to:

winexe -U 'DOMAIN\\USERNAME%password' --uninstall --interactive=1 //hostIP 'msiexec /qn /i https://downloads.puppetlabs.com/windows/puppet-3.8.1.msi /l*v "c:\PuppetInstall.log"'

Tested with winexe -U 'robz' --uninstall --interactive=0 //169.169.169.12 'msiexec /qn /i https://downloads.puppetlabs.com/windows/puppet-3.8.1.msi' and it fixed the affected upgrade that occurred after the repro steps.

It's recommended to use PSExec as it can run against multiple hosts as part of the command, where with winexe you may need to batch the node names and call the command each time. Plus PSExec is known to clean up after itself better than winexe.

Note: I attempted to downgrade to 3.7.4 using this means and it failed the first time (no puppet install at all), but worked the second time I ran the same command.