[PA-2628] gem update --system breaks vendored gems Created: 2019/04/09  Updated: 2019/07/03  Resolved: 2019/05/27

Status: Resolved
Project: Puppet Agent
Component/s: puppet-runtime
Affects Version/s: puppet-agent 5.5.12, puppet-agent 6.4.0
Fix Version/s: puppet-agent 5.5.16, puppet-agent 6.0.10, puppet-agent 6.4.3

Type: Bug Priority: Major
Reporter: Charlie Sharpsteen Assignee: Oana Tanasoiu
Resolution: Fixed Votes: 0
Labels: resolved-issue-added
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Relates
relates to PA-1894 Add new gem path to vendored ruby for... Closed
Template:
Team: Night's Watch
Story Points: 3
Sprint: PR - 2019-05-02, PR - 2019-05-15, PR - 2019-05-29
Method Found: Customer Feedback
Release Notes: Bug Fix
Release Notes Summary: Bug: 'gem update --system' command for ruby that is in the puppet agent/PE project breaks other projects that depends on gems that are in vendor_ruby directory (ex. r10k).
Fix: created a file operating_system.rb in order for gem paths to always contain the path for vendor_ruby directory, even after update.
QA Risk Assessment: Needs Assessment

 Description   

As part of the build process for the puppet-agent package, we patch the RubyGems library to add an additional library location to the default GEM_PATH:

https://github.com/puppetlabs/puppet-runtime/blob/master/resources/patches/ruby_245/rubygems_add_puppet_vendor_dir_r2.4.patch
https://github.com/puppetlabs/puppet-runtime/blob/master/resources/patches/ruby_253/rubygems_add_puppet_vendor_dir_r2.5.patch

This directory holds gems that are shared between the Puppet agent and Puppet server. However, RubyGems is also designed to update its self to a newer version without requiring an upgrade of the core Ruby package. If the gem update --system command is used to perform this sort of upgrade, the newly installed RubyGems library will not have the patch that adds vendor_dir to the GEM_PATH.

Reproduction Case

  • Install PE 2018.1.7 on CentOS 7
  • Run /opt/puppetlabs/bin/puppet gem env to print out the GEM_PATH
  • Upgrade RubyGems: /opt/puppetlabs/bin/puppet gem update --system

Outcome

Prior to the upgrade gem env includes the vendor_gems directory in the GEM PATHS section:

[root@pe-201817-master puppetserver]# /opt/puppetlabs/puppet/bin/gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 2.6.14.3
  - RUBY VERSION: 2.4.5 (2018-10-18 patchlevel 335) [x86_64-linux]
  - INSTALLATION DIRECTORY: /opt/puppetlabs/puppet/lib/ruby/gems/2.4.0
  - USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.4.0
  - RUBY EXECUTABLE: /opt/puppetlabs/puppet/bin/ruby
  - EXECUTABLE DIRECTORY: /opt/puppetlabs/puppet/bin
  - SPEC CACHE DIRECTORY: /root/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/puppetlabs/puppet/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /opt/puppetlabs/puppet/lib/ruby/gems/2.4.0
     - /root/.gem/ruby/2.4.0
     - /opt/puppetlabs/puppet/lib/ruby/vendor_gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /sbin
     - /bin
     - /usr/sbin
     - /usr/bin
     - /root/bin
     - /usr/local/bin
     - /opt/puppetlabs/bin
     - /root/bin

After running the upgrade, gem env no longer includes the vendor_gems directory:

[root@pe-201817-master puppetserver]# /opt/puppetlabs/puppet/bin/gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.0.3
  - RUBY VERSION: 2.4.5 (2018-10-18 patchlevel 335) [x86_64-linux]
  - INSTALLATION DIRECTORY: /opt/puppetlabs/puppet/lib/ruby/gems/2.4.0
  - USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.4.0
  - RUBY EXECUTABLE: /opt/puppetlabs/puppet/bin/ruby
  - GIT EXECUTABLE:
  - EXECUTABLE DIRECTORY: /opt/puppetlabs/puppet/bin
  - SPEC CACHE DIRECTORY: /root/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/puppetlabs/puppet/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /opt/puppetlabs/puppet/lib/ruby/gems/2.4.0
     - /root/.gem/ruby/2.4.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /sbin
     - /bin
     - /usr/sbin
     - /usr/bin
     - /root/bin
     - /usr/local/bin
     - /opt/puppetlabs/bin
     - /root/bin

PE's r10k binary stops working as the fast_gettext gem is inside the missing vendor_gems directory:

[root@pe-201817-master puppetserver]# /opt/puppetlabs/puppet/bin/r10k --help
/opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/dependency.rb:311:in `to_specs': Could not find 'fast_gettext' (~> 1.1.0) among 40 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=/root/.gem/ruby/2.4.0:/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0', execute `gem env` for more information
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1449:in `block in activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `each'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1420:in `activate'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1452:in `block in activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `each'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1420:in `activate'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1452:in `block in activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `each'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1420:in `activate'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1452:in `block in activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `each'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1438:in `activate_dependencies'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1420:in `activate'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems.rb:304:in `block in activate_bin_path'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems.rb:303:in `synchronize'
	from /opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0/rubygems.rb:303:in `activate_bin_path'
	from /opt/puppetlabs/puppet/bin/r10k:23:in `<main>'

Expected Outcome

Runing gem update --system does not break everything.



 Comments   
Comment by Charlie Sharpsteen [ 2019/04/09 ]

Instead of patching defaults.rb it looks like we should be doing this sort of path customization by creating a rubygems/defaults/operating_system.rb file, for example at:

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb

This is a documented location to modify values set in defaults.rb:

http://ruby-doc.org/stdlib-2.4.5/libdoc/rubygems/rdoc/Gem.html#module-Gem-label-RubyGems+Defaults-2C+Packaging

The RubyGems library loads this file when it starts up:

https://github.com/rubygems/rubygems/blob/v3.0.3/lib/rubygems.rb#L1370-L1387

Putting the defaults here would allow an upgraded version of rubygems to automatically incorporate the GEM_PATH that Puppet needs.

Generated at Tue Jan 28 07:50:43 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.