In running Puppet acceptance tests locally with Beaker, I encountered a couple of problems with the ticket_3360_allow_duplicate_csr_with_option_set.rb test:
1) assert_not_equal not found.
Here's a snippet of the failed output from one run:
I don't see a method called assert_not_equal in the version of minitest that we're pulling in from the Gemfile, 5.8.3. I saw that some other gems alias refute_equal, which does exist in minitest to assert_not_equal - e.g., nokogiri. This makes me think that this failure may not occur in full Jenkins CI runs because code that implicitly sources the alias is being run first. This seems dangerous, though, and that it might be better to just use refute_equals directly from this test. AFAICT, no other Puppet acceptance test uses assert_not_equal.
2) The test appears to have been written with the expectation that it will iterate through all of the agents setup for the test run, submitting duplicate csrs for each. It doesn't appear to take into account the pre-existence of other csrs in the ssldir, however, and could fail when those other csrs don't change.
3) The test always skips over the master node. So if a node is configured with both the master and agent roles, it will skip that node entirely - meaning that the test essentially does nothing for a pipeline configured to do just the master and agent on a single node.
4) The regex that the test uses to locate certs in the cert list doesn't seem precise enough, leaving some csrs to be skipped over completely.