Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7014

Audit CI.next pipelines to ensure all tests are being run / RCA the Windows passing runs that ran 1900 tests instead of 20000

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Template:
    • Team:
      Agent
    • Story Points:
      1
    • Sprint:
      AP 2017-01-11
    • QA Risk Assessment:
      No Action
    • QA Risk Assessment Reason:
      [Bulk update] Tickets are for build, infra, or docs and so not risk assessed

      Description

      It was determined that the new CI.next pipelines were not always running the full spec suite on Windows - at job https://jenkins-master-prod-1.delivery.puppetlabs.net/job/platform_puppet_unit-ruby-win_master/

      Up through job 52 on 12/12/2016, we were getting false positive passing runs because only about 1900 tests were running out of the 20,000+ tests.

      The pipeline started failing as of job 53 when the full spec suite started running properly (the actual test bugs were fixed in https://github.com/puppetlabs/puppet/pull/5431).

      It is currently unknown why the full suite has not been running, and this ticket is for:

      • determining why the suites ran a reduced set of specs (its possible this is related to Cygwin git being used on these new CI spec boxes? Note that ticket https://tickets.puppetlabs.com/browse/IMAGES-431 exists for updaiting vmpooler / acceptance nodes to not use Cygwin git, but doesn't address spec boxes)
      • are Windows spec jobs the only affected ones
      • should other matrices in CI.next be audited to confirm they have been migrated properly?

      /cc Josh Cooper Samuel Beaulieu

        Attachments

          Activity

            People

            • Assignee:
              ethan Ethan Brown
              Reporter:
              ethan Ethan Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support