Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7014

Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000


    • Type: Task
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Template:
    • Team:
    • Story Points:
    • Sprint:
      AP 2017-01-11
    • QA Risk Assessment:
      No Action
    • QA Risk Assessment Reason:
      [Bulk update] Tickets are for build, infra, or docs and so not risk assessed


      It was determined that the new CI.next pipelines were not always running the full spec suite on Windows - at job https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/

      Up through job 52 on 12/12/2016, we were getting false positive passing runs because only about 1900 tests were running out of the 20,000+ tests.

      The pipeline started failing as of job 53 when the full spec suite started running properly (the actual test bugs were fixed in https://github.com/puppetlabs/puppet/pull/5431).

      It is currently unknown why the full suite has not been running, and this ticket is for:

      • determining why the suites ran a reduced set of specs (its possible this is related to Cygwin git being used on these new CI spec boxes? Note that ticket https://tickets.puppetlabs.com/browse/IMAGES-431 exists for updaiting vmpooler / acceptance nodes to not use Cygwin git, but doesn't address spec boxes)
      • are Windows spec jobs the only affected ones
      • should other matrices in CI.next be audited to confirm they have been migrated properly?

      /cc Josh Cooper Samuel Beaulieu




            • Assignee:
              ethan Ethan Brown
              ethan Ethan Brown
            • Votes:
              0 Vote for this issue
              5 Start watching this issue


              • Created:

                Zendesk Support