[SERVER-1772] Enhance ezbake to support extra uberjars for JRuby 1.7 vs 9k Created: 2017/04/12  Updated: 2017/06/28  Resolved: 2017/06/28

Status: Closed
Project: Puppet Server
Component/s: None
Affects Version/s: None
Fix Version/s: SERVER 5.0.0

Type: Task Priority: Normal
Reporter: Jeremy Barlow Assignee: Joe Pinsonault
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
blocks SERVER-1775 Setup Jenkins CI job for Puppet Serve... Closed
relates to EZ-109 ezbake.manifest has duplicate headers... Resolved
relates to SERVER-1630 Support user configuration of JRuby 1... Closed
Epic Link: Upgrade Puppet Server to JRuby 9k
Team: Systems Engineering
Sub-team: Server
Story Points: 5
Sprint: Server 2017-04-19, Server 2017-05-03, Server 2017-05-31
Release Notes: New Feature
Release Notes Summary: * Added ezbake option to specify additional uberjars which should be built and installed along-side the main project
* Allows puppetserver to include 2 jars for jruby 1.7 and jruby 9k dependencies
QA Risk Assessment: Needs Assessment


In looking into the work to support the configuration of JRuby 1.7 vs. 9k uberjars for SERVER-1630, we realized that we're going to need some changes made to ezbake. The ezbake work will likely consist of:

1) Provide an optional mechanism in a project's ezbake profile config to specify:

  • A set of Maven coordinates (or maybe GitHub repo + treeish) for a set of "additional jar dependencies" to retrieve.
  • A default set of files to add to the Java classpath for the start, foreground, ruby, irb, and gem scripts.

2) Provide a way for users to configure a new environment variable in the sysconfig/default script which, when set, is appended to the Java classpath in place of any default configured in the project ezbake configuration.

3) For any "additional jar dependencies" set in the ezbake configuration, modify ezbake to:

  • Retrieve project source or Maven artifact (immediate jar + pom.xml/project.clj dependencies).
  • Build an uberjar (immediate jar + any upstream dependencies) of each retrieved project.
  • Package up each additional uberjar alongside the main project's uberjar.

4) Modify the ezbake.manifest to include information for any additional uberjars which might be built into the package.

Comment by Jeremy Barlow [ 2017/04/19 ]

For a prototype implementation of this with Puppet Server, I did the following:

1) Broke out the contents of the puppet-server-release.jar that we normally ship in Puppet Server packages into three different jar files:

  • puppet-server-release.jar (just Puppet Server-specific code and its immediate non-JRuby related dependencies)
  • jruby-1_7.jar (just JRuby 1.7.26 dependencies)
  • jruby-9k.jar (just JRuby-9k dependencies)

2) Added some code to the start bash script (/opt/puppetlabs/server/apps/puppetserver/cli/apps/start) to compute the argument that it passes for the -cp to the Java service. The basic uberjar is always included in the -cp argument. When an environment variable named EXTRA_CLASSPATH is set and non-empty, its contents are appended to the classpath. When the EXTRA_CLASSPATH environment variable is not defined, a 'default' value - for Puppet Server this would be the path to "jruby-1_7.jar" but this would be empty / not used for other packages not needing to add on extra jars by default - is added onto -cp:

if [ -z "$EXTRA_CLASSPATH" ]; then
${JAVA_BIN} ${JAVA_ARGS} -Djava.security.egd=/dev/urandom \
  -cp "${CLASSPATH}" \

3) To cause the "jruby-9k.jar" to be used instead of the default "jruby-1_7.jar", added the following line to the /etc/sysconfig/puppetserver file:


The advantage of having a generic EXTRA_CLASSPATH variable like this is that it's potentially reusable for adding more than just the custom jruby jar to the classpath. The possibility for adding other jars to the classpath by default has been raised in SERVER-249 and SERVER-969.

On reviewing this approach with a few others, including Nicholas Fagerlund, there were some concerns expressed about the possibility that the EXTRA_CLASSPATH variable could be configured in such a way that no JRuby jar were included in the classpath at all, causing the service to fail to startup.

A couple of possibilities to mitigate this were raised:

a) Reserve an environment variable, named something JRUBY_JAR, whose express purpose is to configure a custom JRuby jar to be appended to the classpath, leaving the EXTRA_CLASSPATH variable to be used only for custom, non-internal jars.

This may be workable but we'd have to work through how to add this capability to ezbake in a generic way - such that projects other than Puppet Server could opt out of having any of these extra classpath variables be sourced in their scripts or could specify their own custom environment variables which can translate into extra classpath entries.

b) Consider adding some logic to the start/foreground/... scripts which, on detecting a failure to start the service, validates that the full classpath contains at least a minimal set of entries expected for the service to start - e.g., for Puppet Server would contain at least jruby-17.jar or jruby-9k.jar. On failing the validation check, a more descriptive error message about the failure could be written into the system's daemon log.

Support for this would again need to be handled in a fairly generic way in ezbake configuration, such that packages other than Puppet Server which wouldn't be using this capability would be unaffected by it.

Comment by Jeremy Barlow [ 2017/04/19 ]

On the question about whether we could just have a Puppet Server-specific environment variable like JRUBY_JAR be used, I had another idea.

The basic idea is that we would provide an option for a package being ezbaked to provide its own "cli-defaults.sh.erb" file as a jar resource, like how app-specific cli apps are done by being embedded in ./resources/ext/cli. When the resource is present, it would be evaluated and dropped into /opt/puppetlabs/server/<package name>/cli at installation time. The start and foreground scripts in ezbake proper - and any other custom apps, as desired - would be updated to be able to source the defaults script. The default app scripts could then be modified to have something like:

if [ -e "${INSTALL_DIR}/cli/cli-defaults.sh" ]; then
    . ${INSTALL_DIR}/cli/cli-defaults.sh
    if [ $? -ne 0 ]; then
        echo "Unable to initialize cli defaults, failing start." 1>&2
        exit 1

For Puppet Server's implementation of this, then, we could have a cli-defaults.sh with code like this, allowing the CLASSPATH to be overridden from the custom JRUBY_JAR setting and some custom error handling for failing when the jar is not present:

if [ ! -e "$JRUBY_JAR" ]; then
  echo "Unable to find specified JRUBY_JAR: ${JRUBY_JAR}" 1>&2
  return 1

With this, then, one could add the following to the /etc/sysconfig/puppetserver (or /etc/default/puppetserver) file to enable the use of jruby 9k:


If the user were to configure a path for a non-existent jar in the environment variable, a more specific error message should be written to the journal / daemon log along with a startup failure. Example on EL7:

Apr 20 00:01:40 t4it385b617138x puppetserver[1672]: Unable to find specified JRUBY_JAR: /opt/puppetlabs/server/apps/puppetserver/jruby-1_8.jar
Apr 20 00:01:40 t4it385b617138x puppetserver[1672]: Unable to initialize cli defaults, failing start.
Apr 20 00:01:40 t4it385b617138x systemd[1]: puppetserver.service: control process exited, code=exited status=1

With this approach, if we decide to have Puppet Server stop using the JRUBY_JAR setting at some point - because we intend to fold JRuby back into the puppet-server-release uberjar - we could customize the cli-defaults.sh script to ignore / warn on the presence of a value for the JRUBY_JAR variable.

Past Haus, Nicholas Fagerlund - thoughts on this? Assuming this is somewhat sane, Past Haus, what do you think would make the most sense about how an app could configure the cli-defaults.sh script location. Would it be worth making this externally configurable in an app's ezbake config?

Comment by Nicholas Fagerlund [ 2017/04/24 ]

Oh, interesting. So if I'm understanding this right: instead of trying to come up with a one-size-fits-all ezbake setting for classpath shenanigans, you're suggesting we add a shim that lets ezbake apps designate their own environment variables (and default values) for doing any form of pre-JVM-startup shenanigans. Right?

I think I'm in favor of that. I expect anyone else who's doing something like this for their app is going to want the same kind of error message specificity that I was asking for in that meeting, and this sounds like a good way to provide that?

Comment by Jeremy Barlow [ 2017/04/24 ]

Nicholas Fagerlund - yeah, that's what I was thinking of for this case at least. This approach would also allow us to be able to do other things like app-specific error detection / compensation for the values for other environment variables that are used to build up the full command line argument, like JAVA_ARGS, if needed/desired.

I think there's still some value in having an additional but more generic EXTRA_CLASSPATH variable for users to add on other optional jars that aren't required for the base service to work, like the mysql case mentioned in SERVER-249. I think we can defer the more generic capability out to whenever we design a solution for that ticket. The more I think about it, the more I like the idea of having the JRUBY_JAR setting be separate from an EXTRA_CLASSPATH one. I'm thinking it would be nice for a user to be able to add on more optional jars without having to re-specify - and possibility unintentionally mess up - the jruby jar that they want to be used as well.

Comment by Nicholas Fagerlund [ 2017/04/24 ]

I'm thinking it would be nice for a user to be able to add on more optional jars without having to re-specify - and possibility unintentionally mess up - the jruby jar that they want to be used as well.

Yeah!! This is my thinking exactly.

Comment by Kenn Hussey [ 2017/05/15 ]

Joe Pinsonault please provide release notes for this issue, if applicable.

Generated at Tue Aug 11 00:23:49 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.