[PDK-985] `pdk validate` on large set of files: "Argument list too long" or "stack level too deep (SystemStackError)" Created: 2018/05/11  Updated: 2018/06/22  Resolved: 2018/06/14

Status: Closed
Project: Puppet Development Kit
Component/s: None
Affects Version/s: None
Fix Version/s: PDK 1.6.0

Type: Bug Priority: Normal
Reporter: David Schmitt Assignee: Jesse Scott
Resolution: Fixed Votes: 0
Labels: resolved-issue-added
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relates to PDK-1032 Fix multi-spinner display when doing ... Accepted
relates to PDK-1045 `pdk validate` fails with `The filena... Closed
Epic Link: PDK Maintenance
Method Found: Needs Assessment
Release Notes: Bug Fix
Release Notes Summary: With a large number of files to validate against, PDK will fail with a ruby error. This fix will run the validation in batches so that this doesn't fail.
QA Risk Assessment: Needs Assessment


When running pdk validate against a repo with more than 50k files in lib/puppet, it aborts with

david@davids:~/tmp/blubb3$ pdk validate ruby
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
[|] Checking Ruby code style (**/**.rb).pdk (FATAL): Failed to execute '/home/david/tmp/blubb3/bin/rubocop --format json [50k file names]': Argument list too long - /home/david/tmp/blubb3/bin/rubocop
[/] Checking Ruby code style (**/**.rb).david@davids:~/tmp/blubb3$ 

With 100k files, ruby chokes completely:

david@davids:~/tmp/blubb3$ find lib/ | wc -l
david@davids:~/tmp/blubb3$ pdk validate ruby
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
/opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:125:in `new': stack level too deep (SystemStackError)
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:125:in `block in invoke'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:121:in `each'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:121:in `invoke'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:21:in `block in invoke'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:20:in `each'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:20:in `invoke'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:106:in `block (3 levels) in <module:CLI>'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:105:in `each'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:105:in `block (2 levels) in <module:CLI>'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:329:in `run_this'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:269:in `run'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:287:in `run'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli.rb:18:in `run'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/exe/pdk:6:in `<top (required)>'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/bin/pdk:23:in `load'
	from /opt/puppetlabs/pdk/private/ruby/2.4.4/bin/pdk:23:in `<main>'

This was originally raised by a customer running the PDK against their control repo.

Comment by David Schmitt [ 2018/05/11 ]

Create 100k files with this easy trick:

for i in $(seq 1 10000) ; do touch lib/puppet/{a,b,c,d,e}{f,g}$i.rb; done

Comment by Jarret Lavallee [ 2018/06/22 ]

We are seeing this with a customer that has a deep path on Windows. The result is The filename or extension is too long. (206). There are only 1000 files in the command, but with the deep path it may be too much for the command line to execute. Can we decrease this from 1000 to a lower number, or make it a command line argument?

Comment by David Schmitt [ 2018/06/22 ]

Jarret Lavallee if you are still seeing this or a similar issue with 1.6.0, please open a new issue, so that it can be tracked separately, but link it here. Commenting on a closed issue is a good chance to get missed in the daily grind.

Comment by Jarret Lavallee [ 2018/06/22 ]

David Schmitt Sorry about that. A new ticket has been created for the issue. 

Generated at Wed Nov 20 22:15:19 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.