[PA-1850] PXP fails to start after a Puppet Agent upgrade with SSL library errors Created: 2018/02/06  Updated: 2018/12/12  Resolved: 2018/03/20

Status: Closed
Project: Puppet Agent
Component/s: Windows
Affects Version/s: None
Fix Version/s: puppet-agent 1.10.12, puppet-agent 5.3.6, puppet-agent 5.5.0

Type: Bug Priority: Major
Reporter: Glenn Sarti Assignee: Enis Inan
Resolution: Fixed Votes: 0
Labels: customer-escalation, manage-service
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

TEST_TARGET=windows2008r2-64a
SUITE_VERSION=1.6.2.135.g8a48f21
SUITE_COMMIT=8a48f210f6901be0edba16eb49c1bfbfb07d338b


Attachments: PNG File screenshot-1.png    
Issue Links:
Relates
relates to PUP-6702 Windows service in Paused state canno... Closed
Template:
Epic Link: Windows service management improvements
Team: Platform OS
Sprint: Platform OS Kanban
Method Found: Customer Feedback
CS Priority: Major
CS Frequency: 3 - 25-50% of Customers
CS Severity: 4 - Major
CS Business Value: 5 - $$$$$$
CS Impact: If openssl libraries(of a different version than ours) are installed in the Windows system folders then the PXP agent will fail to start. The workaround is to is to reorder the path or remove or replace those library files in other locations.

This has come up with 5-10x in the last few weeks, a very high rate of occurrence.
Release Notes: Bug Fix
Release Notes Summary: All of the .dll files that the pxp-agent binary depends on are now copied over to the executable's directory (the <pxp-agent-install-root>/bin directory).

 Description   

 

This issue was created from PUP-6702

 Steps to reproduce;
Unknown at this stage.  Looks like it could be made to happen after the puppet_agent module does an agent upgrade.



 Comments   
Comment by Adam Buxton [ 2018/02/06 ]

Nathanael Cole I sooke to a customer yesterday who reckognised it is an issue with having anlternative version of Libraries PCP relies upon in path.

It dawned on me that the pxp-agent uses openssl and we also have windows modules which use openssl libraries.
Once I removed these from PATH the pxp-agent ran up ok.
Namely, the libeay32.dll, libss32.dll and ssleay32.dll

Comment by Josh Cooper [ 2018/02/06 ]

The pxp-agent service should be configuring its PATH so that our vendored openssl takes precedence over any directory including system32. Please file a pxp-agent ticket if there isn't one already. /cc Michael Smith, Alex Dreyer

Comment by Michael Smith [ 2018/02/06 ]

I'll have to look into that. Last I checked, there wasn't a way to make PATH override system32; I think the only thing that works is to put the binaries in the same directory.

Comment by Ryan Murphy [ 2018/02/06 ]

I just confirmed that copying DLL's + openssl and curl from the puppet/bin directory to pxp-agent/bin and copying zlib1.dll from one of the ruby directories will allow the Puppet PXP Agent to start and I can remotely run Puppet from the PE console.

Comment by Michael Smith [ 2018/02/06 ]

The most straight-forward short-term fix may be to change where we're installing pxp-agent binaries.

Comment by Ryan Murphy [ 2018/02/06 ]

zlib1.dll copied from C:\program files\Puppet Labs\Puppet\sys\ruby\bin

Comment by Ryan Murphy [ 2018/02/06 ]

Copying zlib1.dll is unnecessary if actually starting the service it appears, it was needed when I was trying to run the pxp-agent binary from command prompt for testing.

Comment by Ethan Brown [ 2018/02/06 ]

With respect to loading libraries on Windows, the LoadLibrary docs cover the ordering of the search paths. You can introduce a search path before system32...

The first directory searched is the directory containing the image file used to create the calling process (for more information, see the CreateProcess function). Doing this allows private dynamic-link library (DLL) files associated with a process to be found without adding the process's installed directory to the PATH environment variable. If a relative path is specified, the entire relative path is appended to every token in the DLL search path list. To load a module from a relative path without searching any other path, use GetFullPathName to get a nonrelative path and call LoadLibrary with the nonrelative path. For more information on the DLL search order, see Dynamic-Link Library Search Order.

The search path can be altered using the SetDllDirectory function. This solution is recommended instead of using SetCurrentDirectory or hard-coding the full path to the DLL.

If a path is specified and there is a redirection file for the application, the function searches for the module in the application's directory. If the module exists in the application's directory, LoadLibrary ignores the specified path and loads the module from the application's directory. If the module does not exist in the application's directory, LoadLibrary loads the module from the specified directory. For more information, see Dynamic Link Library Redirection.

I don't believe we can change the LoadLibrary call itself here, right? We can, however, change the ordering by calling SetDllDirectory to insert an app-specific load path. After the call, the modified ordering becomes:

The SetDllDirectory function affects all subsequent calls to the LoadLibrary and LoadLibraryEx functions. It also effectively disables safe DLL search mode while the specified directory is in the search path.

After calling SetDllDirectory, the standard DLL search path is:

1. The directory from which the application loaded.
2. The directory specified by the lpPathName parameter.
3. The system directory. Use the GetSystemDirectory function to get the path of this directory. The name of this directory is System32.
4. The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched. The name of this directory is System.
5. The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
6. The directories that are listed in the PATH environment variable.

Comment by Michael Smith [ 2018/02/06 ]

Josh Cooper I believe what you suggested is already done at https://github.com/puppetlabs/puppet-agent/blob/master/resources/windows/wix/service.pxp-agent.wxs.erb#L37. It looks like Puppet and sys/ruby were added as of puppet-agent 1.6.0. It's not clear to me why other openssl versions would be loaded ahead of our packaged version simply by being on PATH, since we ensure our binaries are at the head of PATH when starting the service.

Comment by Ethan Brown [ 2018/02/06 ]

As a follow-up, it looks like SetDllDirectory is not viable solution to our problem given the load order of statically linked libraries like openssl. Michael Smith found some good detail on the process bootstrapping in https://stackoverflow.com/questions/41392014/what-does-windows-do-before-main-is-called

That said, I think we can use application manifests to alter load behavior, but I think the XML manifest has to be a resource compiled into the library. Will have to do a bit more research on that, but if feasible, that seems better than re-working the installation layout right now.

I believe the answer to your question about PATH Michael Smith is because PATH is searched after the system directory per the default LoadLibrary ordering described in my last comment.

Comment by Michael Smith [ 2018/02/06 ]

Yes, the problem description wasn't specific about where OpenSSL was located. If they were installed in C:\Windows or C:\Windows\system32, then PATH is insufficient.

Comment by Ryan Murphy [ 2018/02/06 ]

In my case, OpenSSL was installed to C:\Program Files\HEAT Software\HEAT\ThirdParty\OpenSSL

One installer I saw for OpenSSL on windows defaults to C:\OpenSSL-Win32

I think installing OpenSSL to C:\Windows or C:\Windows\system32 would be bad practice. Most vendors should know better not to actually install things in those locations.

Comment by Michael Smith [ 2018/02/06 ]

I'm not sure how to look into HEAT, but one of the things I notice with the installer from https://slproweb.com/products/Win32OpenSSL.html (which generally installs to C:\OpenSSL-Win32 or 64) is that it has a default option to Copy OpenSSL DLLs to the Windows system directory.

Comment by Owen Rodabaugh [ 2018/02/06 ]

Does this ticket need to be split into one about the SSL issue and another about the paused state?

Comment by Ethan Brown [ 2018/02/06 ]

Yes Owen Rodabaugh, we have multiple issues grouped together here because the error message presented is the same. The puppet_agent upgrade issue where pxp-agent fails to restart due to OpenSSL libraries present on the system should be spun off into a new issue. This ticket was originally about something completely different.

Comment by Glenn Sarti [ 2018/02/06 ]

Interesting - https://community.ivanti.com/docs/DOC-61035

Comment by Glenn Sarti [ 2018/02/06 ]

Repro'd the issue.  Some steps may not be required but it's how I did it.

  • Spin up a Puppet Master and Windows 2012R2 client.  Fully connect them and ensure the "Run Puppet" functionality is working (Use your own tools for this.  For my repro I used the Learning VM from SLICE and a Windows 2012R2 client with the Puppet 5.3.1 x64 Agent
  • Stop the PXP-Agent Service
  • copy the C:\Windows\System32\lltdapi.dll to C:\Windows\System32\libeay32.dll  (This is basically using a valid DLL file but it does not have the same functions exported etc. so it should break if used)
  • Start the PXP-Agent Service
  • It Should error and then NSSM will put the service into a Pause state
  • Delete the "bad" C:\Windows\System32\libeay32.dll file and restart the PXP agent service and it will run correctly.

 

Comment by Glenn Sarti [ 2018/02/06 ]

I don't think https://puppetlabs.zendesk.com/agent/tickets/26386 is relevant for this ticket

Comment by Ethan Brown [ 2018/02/13 ]

So I think we have 3 options for addressing this problem.

1. Copy the OpenSSL DLL files into the same directory as pxp-agent.exe, by making changes to the MSI installer. This is the most direct solution and carries a small penalty of including extra files in the installer - but is the least complicated solution (presuming no additional libraries must be copied there).

2. Modify how pxp-agent.exe and the OpenSSL dlls are compiled, so that the win32 loader uses the SxS mechanism to find the libraries by relative path.

https://stackoverflow.com/a/1973332 has a good explanation of using resource manifests for precisely this situation, but note that it requires multiple steps from what I can tell:

  • Compile manifests into each of the OpenSSL dlls (requires using mt.exe) that specify the library versions
  • Compile a manifest into pxp-agent.exe that references the OpenSSL dlls
  • Add a pxp-agent.exe.config that lives alongside

This requires making some changes to CMake build scripts to include embedded resources. CMake 3.4 added support for manifest resources, but our current CMake 3.2 doesn't understand them natively, though they're straightforward to add https://stackoverflow.com/a/9611396

3. Redo the internals of the installer to move around files so that all binaries are colocated. I'm reluctant to go down this path considering it may cause breaking changes for end users.

Comment by Michael Smith [ 2018/02/13 ]

We already do the 1st with Ruby, I think that sounds fine until we restructure.

If we wanted to do 3 on a small scale, we could just move pxp-agent files, which would be unlikely to break anything people use.

Comment by Geoff Nichols [ 2018/02/21 ]

Enis Inan, when you have a chance, could you please take a look at this?

Generated at Tue Feb 25 17:15:34 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.