Uploaded image for project: 'Puppet Development Kit'
  1. Puppet Development Kit
  2. PDK-985

`pdk validate` on large set of files: "Argument list too long" or "stack level too deep (SystemStackError)"

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PDK 1.6.0
    • Component/s: None
    • Template:
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      With a large number of files to validate against, PDK will fail with a ruby error. This fix will run the validation in batches so that this doesn't fail.
    • QA Risk Assessment:
      Needs Assessment

      Description

      When running pdk validate against a repo with more than 50k files in lib/puppet, it aborts with

      david@davids:~/tmp/blubb3$ pdk validate ruby
      pdk (INFO): Using Ruby 2.4.4
      pdk (INFO): Using Puppet 5.5.1
      [|] Checking Ruby code style (**/**.rb).pdk (FATAL): Failed to execute '/home/david/tmp/blubb3/bin/rubocop --format json [50k file names]': Argument list too long - /home/david/tmp/blubb3/bin/rubocop
      [/] Checking Ruby code style (**/**.rb).david@davids:~/tmp/blubb3$ 
      david@davids:~/tmp/blubb3$ 
      

      With 100k files, ruby chokes completely:

      david@davids:~/tmp/blubb3$ find lib/ | wc -l
      100002
      david@davids:~/tmp/blubb3$ pdk validate ruby
      pdk (INFO): Using Ruby 2.4.4
      pdk (INFO): Using Puppet 5.5.1
      /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:125:in `new': stack level too deep (SystemStackError)
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:125:in `block in invoke'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:121:in `each'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/base_validator.rb:121:in `invoke'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:21:in `block in invoke'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:20:in `each'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/validate/ruby_validator.rb:20:in `invoke'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:106:in `block (3 levels) in <module:CLI>'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:105:in `each'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli/validate.rb:105:in `block (2 levels) in <module:CLI>'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:329:in `run_this'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:269:in `run'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/cri-2.10.1/lib/cri/command.rb:287:in `run'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/lib/pdk/cli.rb:18:in `run'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/lib/ruby/gems/2.4.0/gems/pdk-1.6.0.pre/exe/pdk:6:in `<top (required)>'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/bin/pdk:23:in `load'
      	from /opt/puppetlabs/pdk/private/ruby/2.4.4/bin/pdk:23:in `<main>'
      david@davids:~/tmp/blubb3$ 
      

      This was originally raised by a customer running the PDK against their control repo.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  jesse Jesse Scott
                  Reporter:
                  david.schmitt David Schmitt
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: