[PUP-7738] "bundle exec puppet module install" fails on Puppet 5.0.0 Created: 2017/06/28  Updated: 2017/07/19  Resolved: 2017/07/18

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: PUP 5.0.0
Fix Version/s: PUP 5.0.1

Type: Bug Priority: Normal
Reporter: Robert Heinzmann Assignee: Unassigned
Resolution: Fixed Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Puppet 5.0 gem on Centos7 with librarian puppet running with "bundle exec"


Issue Links:
Duplicate
is duplicated by MODULES-5155 Puppetlabs_spec_helper fails to insta... Closed
Template:
Team: Agent
Story Points: 2
Sprint: Agent 2017-07-12, Agent 2017-07-26
Method Found: Customer Feedback
Release Notes: Bug Fix
Release Notes Summary: Puppet 5.0.0 introduced a regression when running "puppet module install" via bundler. This issue only occurred if the Gemfile directly or indirectly expressed a dependency on the "semantic_puppet" gem. A common way for this to happen was if the module's Gemfile relied on the "metadata-json-lint" gem, which depends on "semantic_puppet". This fix ensures puppet works correctly when using either the external "semantic_puppet" gem or the vendored version in Puppet.
QA Risk Assessment: Automate
QA Risk Assessment Reason: partially covered by unit tests. should still test in acceptance with semantic_puppet gem installed

 Description   

We have a Puppet 4.10.x pipeline running librarian puppet to install modules from Puppetfile.

This now failes on Puppet 5.0 with:

10:51:05 10:51:05 Error executing puppet module install. Check that this command succeeds:
10:51:05 10:51:05 puppet module install --version 4.17.1 --target-dir /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --module_repository https://forgeapi.puppetlabs.com --modulepath /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --module_working_dir /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --ignore-dependencies puppetlabs-stdlib
10:51:05 10:51:05 Error:
10:51:05 10:51:05 Warning: Support for ruby version 2.0.0 is deprecated and will be removed in a future release. See https://docs.puppet.com/puppet/latest/system_requirements.html#ruby for a list of supported ruby versions.
10:51:05 10:51:05    (at /yyyyy/gems/ruby/gems/puppet-5.0.0/lib/puppet.rb:181:in `<module:Puppet>')
10:51:05 10:51:05 Error: wrong number of arguments (2 for 1)
10:51:05 10:51:05 Error: Try 'puppet help module install' for usage



 Comments   
Comment by Kenn Hussey [ 2017/06/28 ]

Geoff Nichols FYI

Comment by Ben Ford [ 2017/06/28 ]

Robert Heinzmann Can you check your quoting, escaping, etc? What does your pipeline look like? Does your command work when run by hand?

It appears to work here on CentOS 7,

root@master:~ # /usr/local/bin/puppet --version
5.0.0
root@master:~ # /usr/local/bin/puppet module install --version 4.17.1 --target-dir /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --module_repository https://forgeapi.puppetlabs.com --modulepath /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --module_working_dir /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 --ignore-dependencies puppetlabs-stdlib
Warning: Support for ruby version 2.0.0 is deprecated and will be removed in a future release. See https://docs.puppet.com/puppet/latest/system_requirements.html#ruby for a list of supported ruby versions.
   (at /usr/local/share/gems/gems/puppet-5.0.0/lib/puppet.rb:181:in `<module:Puppet>')
Notice: Preparing to install into /xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1 ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/xxxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/puppetlabs-stdlib/4.17.1
└── puppetlabs-stdlib (v4.17.1)

Comment by Josh Cooper [ 2017/06/28 ]

Robert Heinzmann if you can reproduce outside of librarian puppet, can you add --trace to the command line, and add the backtrace to this ticket?

Comment by Robert Heinzmann [ 2017/06/28 ]

Hello,
it seems to be only a problem when run with "bundle exec". This worked for puppet 4.10.4.

In our pipeline librarian is called with "bundle exec librarian-puppet install". This failes if

  • librarian-puppet is in the Gemfile: Error: wrong number of arguments (2 for 1)
  • json is missin in Gemfile: cannot load such file – json

OK:

gem 'json_pure'
gem 'puppet', '~> 5.0'

Failes with Error: wrong number of arguments (2 for 1):

gem 'librarian-puppet'
gem 'json_pure'
gem 'puppet', '~> 5.0'

Failes with Error: json missing:

gem 'puppet', '~> 5.0'

Debugging (bundle exec gem list) shows that the failing Gem is puppet_forge:

Failes with Error: wrong number of arguments (2 for 1)

gem 'puppet_forge'
gem 'json_pure'
gem 'puppet', '~> 5.0'

Notice: Preparing to install into /xxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/AlexCline-dirtree/0.2.1 ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Error: wrong number of arguments (2 for 1)
Error: Try 'puppet help module install' for usage

Trace:

Notice: Preparing to install into /xxx/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/AlexCline-dirtree/0.2.1 ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Error: wrong number of arguments (2 for 1)
/usr/local/share/gems/gems/semantic_puppet-0.1.4/lib/semantic_puppet/version_range.rb:26:in `parse'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/module_tool/applications/installer.rb:209:in `build_single_module_graph'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/module_tool/applications/installer.rb:86:in `run'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/module_tool/applications/application.rb:11:in `run'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/face/module/install.rb:131:in `block (3 levels) in <top (required)>'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/interface/action.rb+eval[wrapper]:242:in `install'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/application/face_base.rb:247:in `main'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/application.rb:366:in `run_command'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/application.rb:358:in `block in run'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/util.rb:666:in `exit_on_fail'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/application.rb:358:in `run'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/util/command_line.rb:132:in `run'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/util/command_line.rb:72:in `execute'
/home/xxxxx/.gem/ruby/gems/puppet-5.0.0/bin/puppet:5:in `<top (required)>'
/home/xxxxx/bin/puppet:23:in `load'
/home/xxxxx/bin/puppet:23:in `<top (required)>'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:74:in `load'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:27:in `run'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli.rb:360:in `exec'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli.rb:20:in `dispatch'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/cli.rb:10:in `start'
/usr/local/share/gems/gems/bundler-1.15.0/exe/bundle:35:in `block in <top (required)>'
/usr/local/share/gems/gems/bundler-1.15.0/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/usr/local/share/gems/gems/bundler-1.15.0/exe/bundle:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Error: Try 'puppet help module install' for usage

Gemspec https://github.com/puppetlabs/forge-ruby/blob/master/puppet_forge.gemspec

Comment by Robert Heinzmann [ 2017/06/28 ]

Debugged some more and it seems puppet 5.0 now comes with a bundled SemanticPuppet gem that has a new API "/home/xxxx/.gem/ruby/gems/puppet-5.0.0/lib/puppet/vendor/semantic_puppet/lib/semantic_puppet.rb

So if semantic_puppet is used as a gem by some other module (e.g. puppet_forge) this breaks load path.

Why not fix / release and use the external gem ? This would allow tool developers to follow puppet internal semver versioning ?

Comment by Thomas Hallgren [ 2017/06/29 ]

Robert Heinzmann, the external gem is both fixed and released. According to the trace, you're using version 0.1.4. The current version of semantic_puppet is 1.0.0.

That said, upgrading will not help you in this case, since the module tool relies on the vendored version of semantic_puppet where we've added code to cater for backward compatibility issues. As part of that, an additional (but optional) argument was added to the VersionRange#parse method. The released gem does not expect that argument.

Comment by Robert Heinzmann [ 2017/06/29 ]

Agreed

...where we've added code to cater for backward compatibility issues...

Shouldn't this also be backported to semantic_puppet then, so that puppet_forge module can pick up the changes and then librarian will work again ?

Comment by Thomas Hallgren [ 2017/06/29 ]

That could be a solution but it's debatable since it introduces broken semver behavior that is undesirable for all other use-cases. Would it be possible to change librarian to instead use the vendored version?

Comment by Robert Heinzmann [ 2017/06/29 ]

It seems that librarian-puppet (https://github.com/voxpupuli/librarian-puppet/blob/master/librarian-puppet.gemspec) uses puppet_forge (https://github.com/puppetlabs/forge-ruby/blob/master/puppet_forge.gemspec) a gem from puppetlabs and that one uses semantic_puppet.

So fixing puppet_forge (on puppet side) would solve the issue

In general https://github.com/puppetlabs/forge-ruby still uses 0.x version, while 1.0 is out "fixing" things, so toughing this code may valuable anyway.

Comment by Kenn Hussey [ 2017/06/29 ]

Craig Gomes ^^^

Comment by Hunter (Hunner) Haugen [ 2017/06/29 ]

In retrospect, to "vendor" something means to include a released codebase in your own codebase, but the codebase present in https://github.com/puppetlabs/puppet/tree/5.0.0/lib/puppet/vendor/semantic_puppet/lib/semantic_puppet is not a pre-existing codebase.

Comment by Josh Cooper [ 2017/06/29 ]

So it's complicated. Originally it was semver and was implemented in puppet, see https://projects.puppetlabs.com/issues/7204. Then we used that as the basis for constructing an external semantic library (since it was needed by puppet, r10k, etc). But we renamed it to semantic_puppet because there was already a semantic gem on the forge. Up until recently, puppet still vendored the old crusty version of semantic_puppet 0.1.4, and we updated that to 1.0.0 for puppet 5. The intention has always been to unvendor semantic_puppet from puppet, but we ran into puppetserver packaging issues.

Comment by Josh Cooper [ 2017/06/29 ]

This issue is occurring because Puppet's module tool calls:

        range = SemanticPuppet::VersionRange.parse(version, @strict_semver)

Normally, that invokes the vendored semantic_puppet library in puppet whose parse method accepts 2 arguments. However, when running in bundler and semantic_puppet is a gem dependency, then bundler will put the external gem ahead in the $LOAD_PATH. So then the PMT tries to call the wrong version of the method, and it fails with:

Error: wrong number of arguments (2 for 1)

The immediate fix is going to be to fix the external semantic_puppet gem to accept an optional argument and release a new version of the gem. So closing this as a dup of MODULES-5155.

Comment by Thomas Hallgren [ 2017/06/30 ]

Reopening this since we decided that the puppet module tool needs to check the arity of the VersionRange#parse method, or it will still fail when using other versions of the gem.

Although this will fix almost all use-cases, there is still one that will remain. And that is if the semantic_puppet version 1.0.x is used. Its method has an arity of 1 and when called, it will always enforce strict semver semantics. We can only fix this partially by providing a 1.0.1 version of the gem where the patch is applied but the problem will still remain for the 1.0.0. version.

Comment by Robert Heinzmann [ 2017/06/30 ]

Sorry to interfere from the outside (more or less), but I still do not get (as more of a ops then a dev guy) why to bundle semantic_puppet gem within the puppet gem, while semantic_puppet has initially been created to avoid code duplicates. Shouldn't you just depend on if in gemspec and bundle it in Puppet AIO in a tested version ?

Comment by Thomas Hallgren [ 2017/06/30 ]

Robert Heinzmann, I understand your concern and we are going to remove the bundled version. We did however decide to postpone that for Puppet 6 (see PUP-7115). The reason is partly because we want to retain a somewhat broken behavior that was present in 0.1.4 but has been corrected in 1.0.0. Incidentally, this is also what's causing the current problems with the extra argument. We didn't really anticipate that users were using the gem and that doing so actually overrides the vendored gem.

Another reason for not removing it right now is that we're concerned that users might depend on the gem being vendored and thus be left with broken configurations if it's no longer present.

Comment by Markus Frosch [ 2017/06/30 ]

Thanks for dealing with that!

Comment by David Hollinger [ 2017/06/30 ]

Josh Cooper,

Any idea when such a fix might be pushed into the external gem and release. Right now this issue is preventing users of the VP metadata-json-lint tool from using Puppet 5 as it breaks puppet module install with the external gem and the metadata-json-lint gem is broken without the external gem.

We're looking into workarounds on our end, but it'd be preferential to not have to hack together some crazy logic depending on Puppet version.

Comment by Josh Cooper [ 2017/06/30 ]

Hey David Hollinger, I'm reviewing the changes now, hope to get that merged today. But next Monday and Tues are US/company holidays, so it's going to be next week before we can get a fix published.

To repro, create a Gemfile containing:

source "https://rubygems.org"
gem "puppet"
gem "semantic_puppet"

$ bundle install --path .bundle
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Installing CFPropertyList 2.2.8
Installing fast_gettext 1.1.0
Installing locale 2.1.2
Installing text 1.3.1
Installing hiera 3.4.0
Using bundler 1.14.6
Installing facter 2.4.6 (universal-darwin)
Installing gettext 3.2.3
Installing gettext-setup 0.25
Installing puppet 5.0.0 (universal-darwin)
Installing semantic_puppet 1.0.0
Bundle complete! 2 Gemfile dependencies, 11 gems now installed.
Bundled gems are installed into ./.bundle.

Trying to puppet module install fails with cryptic error:

$ bundle exec puppet module install puppetlabs-apache
Notice: Preparing to install into /Users/josh/.puppetlabs/etc/code/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: Could not install module 'puppetlabs-apache' (???)
  No version of 'puppetlabs-apache' can satisfy all dependencies
    Use `puppet module install --ignore-dependencies` to install only this module

Retry with --ignore-dependencies, fails with wrong number of arguments error.

$ puppet module install puppetlabs-apache --ignore-dependencies --trace
Notice: Preparing to install into /Users/josh/.puppetlabs/etc/code/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: wrong number of arguments (given 2, expected 1)
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/semantic_puppet-1.0.0/lib/semantic_puppet/version_range.rb:64:in `parse'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/module_tool/applications/installer.rb:209:in `build_single_module_graph'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/module_tool/applications/installer.rb:86:in `run'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/module_tool/applications/application.rb:11:in `run'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/face/module/install.rb:131:in `block (3 levels) in <top (required)>'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/interface/action.rb+eval[wrapper]:242:in `install'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/application/face_base.rb:247:in `main'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/application.rb:366:in `run_command'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/application.rb:358:in `block in run'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/util.rb:666:in `exit_on_fail'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/application.rb:358:in `run'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/util/command_line.rb:132:in `run'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/lib/puppet/util/command_line.rb:72:in `execute'
/Users/josh/work/test_semantic/.bundle/ruby/2.3.0/gems/puppet-5.0.0-universal-darwin/bin/puppet:5:in `<top (required)>'

Comment by Josh Cooper [ 2017/07/10 ]

The issue James Pogran mentioned is due to our semantic_puppet implementation (in both the external gem and in our vendored implementation) is MODULES-5159.

Comment by Josh Cooper [ 2017/07/14 ]

Merged to 5.0.x in https://github.com/puppetlabs/puppet/commit/7f53ea252a1f25189b9264e0f8dd5c5089f96753

Comment by Josh Cooper [ 2017/07/14 ]

To watchers on the ticket, can you try modifying your Gemfile as:

gem "puppet", :git => "https://github.com/puppetlabs/puppet", :branch => "5.0.x"

And make sure this fix resolves the issue you're seeing?

Comment by Taylan Develioglu [ 2017/07/17 ]

Josh Cooper Can confirm this fixes the issue.

Comment by Kenn Hussey [ 2017/07/17 ]

Josh Cooper please provide release notes for this issue, if needed.

Comment by Josh Cooper [ 2017/07/17 ]

Awesome, thanks Taylan Develioglu!

Comment by Eric Thompson [ 2017/07/17 ]

Erik Dasher, this looks like a prime first suspect for automation.

bundle install with the Gemfile above (possibly JUST with semantic_puppet?) and bundle exec puppet module install with a version specified and strict_semver turned on.

looks like this needs a testrail entry

this is partially covered by unit tests. but needs to test external gems don't mess with internal function namespacing.

Comment by Kenn Hussey [ 2017/07/18 ]

Eric Thompson Erik Dasher please create a separate ticket to track addition of automated tests per Eric Thompson's comment.

Comment by Eric Thompson [ 2017/07/18 ]

validated using a bundle install of semantic-puppet and puppet as above on ubuntu1404 (puppet gem from source ( 5.0.x@bcda912)):

root@yr8hnm5adwkpdpg:~# bundle exec puppet module install puppetlabs-apache
Warning: Support for ruby version 1.9.3 is deprecated and will be removed in a future release. See https://docs.puppet.com/puppet/latest/system_requirements.html#ruby for a list of supported ruby versions.
   (at /root/.bundle/ruby/1.9.1/gems/puppet-5.0.0/lib/puppet.rb:181:in `<module:Puppet>')
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: Could not install module 'puppetlabs-apache' (???)
  No version of 'puppetlabs-apache' can satisfy all dependencies
    Use `puppet module install --ignore-dependencies` to install only this module
 
root@yr8hnm5adwkpdpg:~# vi Gemfile
 
root@yr8hnm5adwkpdpg:~# bundle update
Fetching https://github.com/puppetlabs/puppet
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Rubygems 1.8.23 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.
Fetching CFPropertyList 2.3.5
Installing CFPropertyList 2.3.5
Using bundler 1.15.2
Using facter 2.5.0
Using fast_gettext 1.1.0
Using locale 2.1.2
Using text 1.3.1
Using hiera 3.4.0
Fetching hocon 1.2.5
Installing hocon 1.2.5
Using gettext 3.2.3
Using gettext-setup 0.26
Using puppet 5.0.1 from https://github.com/puppetlabs/puppet (at 5.0.x@bcda912)
Using semantic_puppet 1.0.1
Bundle updated!
root@yr8hnm5adwkpdpg:~# bundle exec puppet module install puppetlabs-apache
Warning: Support for ruby version 1.9.3 is deprecated and will be removed in a future release. See https://docs.puppet.com/puppet/latest/system_requirements.html#ruby for a list of supported ruby versions.
   (at /root/.bundle/ruby/1.9.1/bundler/gems/puppet-bcda9125e43a/lib/puppet.rb:181:in `<module:Puppet>')
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: VersionRanges will always be strict when using non-vendored SemanticPuppet gem, version 1.0.1
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppetlabs-apache (v1.11.0)
  ├── puppetlabs-concat (v2.2.1)
  └── puppetlabs-stdlib (v4.17.1)

Generated at Tue Jan 28 08:04:59 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.