[PDB-1034] Ezbake: switch source based tests to use ezbake somehow and retire that old code Created: 2014/11/17  Updated: 2015/11/20  Resolved: 2015/02/11

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: None
Fix Version/s: PDB 3.0.0

Type: Task Priority: Normal
Reporter: Ken Barber Assignee: Ken Barber
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relates to SERVER-318 track / assist with Ken's ezbake PR f... Closed
Epic Link: QA/QEng Phase 1
Story Points: 3
Sprint: PuppetDB 2014-12-03, PuppetDB 2014-12-17, PuppetDB 2015-01-14, PuppetDB 2015-01-28, PuppetDB 2015-02-11


The ezbake work in PDB-663 introduced ezbake but we're not able to shed the old build code until we have source based installations working. This ticket tracks the work necessary to make that happen.

Two ideas for now:

a) Use the ezbake helpers already in beaker to build and install locally on a jenkins slave & ship the tarball to an ec2 instance so we can run make on it.
b) We do PR testing with packages, but use Roger's pipelining idea to do this.

Either way our goals should be: PR tests still work, and that old code is dead.

Comment by Ken Barber [ 2014/12/11 ]

Sent out this email to the PuppetDB team, and to Wayne Warren, Past Haus, Chris Price, Roger Ignazio and Alice Nodelman for review and visibility:

TL;DR source based builds for PuppetDB + ezbake patches up, please review

So I've managed to get some patches raised now for the ezbake & beaker changes required to support source based installations for PR testing and one off runs. This is the second phase of the PuppetDB ezbake cutover, currently for PR testing and source based testing it is using the old packaging methodology to do this, but that has to go, since it no longer reflects proper 'shipped' reality.

This is a 3-way patch, and is reasonably complex to follow, I do apoligise:

I would appreciate some review when people have the time. These are WIP for now, as I expect some more changes to come after discussion, and I anticipate some testing failures potentially, but for now they are good to review at least. Without a doubt, the beaker patch includes the largest set of changes FWIW, the rest are pretty minor in LoC but potentially big in breaking compatibility potential.

Here are the tests I've been running, in particular the latest one just before I raised the PR's:


And the existing production PR testing is here:


FYI: there is a transient failure in that second test, due to a maven download error, which can be fixed in the future with a proper maven mirror.

FYI: There is also currently a bug on Beaker 2.0.0 and RHEL 7 that kills off SSH connections (I think something changed in the way we manage sshd_config) so please ignore that failure for now (its raised here: https://tickets.puppetlabs.com/browse/QENG-1594).

Primarily the things to note:

  • Source based installations need to support the same installation methodology as packages, so I've moved the package build tasks into the Makefile. Some of the actions were already moved over, this moves almost everything over now, so the Makefile is authoritative. I expect this to be something worth discussing.
  • The ezbake helpers are now much dumber, and simpler for the singleton case, so that less work is required on the consumer. For example, we now do the lein install and ezbake runs in a local m2 repo for safety to ensure we don't wreck a hosts ~/.m2/repository (which is bad in jenkins). We also can autodetect the project name and version using pprint. This is probably going to need to change in the future for a composite ezbake project case (like the various multi-projects in PE).
  • I have no idea what I might have broken for other projects, if other projects use this code today. I've tried to keep compatibility to some extent, but various actions (like the new lein install handling) have probably broken the expected internal behaviour. Please advise .
  • I haven't yet retired the old PuppetDB code on purpose. I figure the patch would be very hard to read if I did an en-masse retirement of the old packaging code this time around, so I'm punting it for later. It is however totally a candidate once we merge this code in. Now in regards to the beaker smoke tests, they should still work for these patches, but I'm recommending we switch over to this new method earlier rather than later (note: Alice - the script ext/jenkins/beaker-tests-source.sh contains this methodology - I'll talk to you more about this).

... I've also left more details in the git comment on each patch also ...

I guess the order these things need to be merged in eventually are:

  • ezbake
  • beaker (and release or pinning)
  • puppetdb
Generated at Sun Sep 22 06:55:59 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.