BENNY BOTTEMA

Automating your Github library releases to Maven Central

After a long time performing manual releases on my own laptop, I decided to jump on the CI/CD bandwagon and automate everything.

I’ve been working professionally with Jenkins and Bamboo for many years, but I never took the time to properly set it up for my open source project, Simple Java Mail. I finally decided I would combine learning a new CI/CD tool with setting up auto-releases to Maven Central.

Now Simple Java Mail is a multi-modular Maven project, so that makes things a little more complicated, so for this blog I’ve created a test project that you can fork and study:

• https://github.com/bbottema/auto-deploy-test (tag: “circleci-deploy-final-without-orb”)

Plan of Attack

Here’s what we want to achieve!

1. Checkout the source code from the Github repository
2. Compile, test the project
3. Manually choose if the release entails a patch, minor or major version
4. Automatically update the POM with the new release semver based on the previous choice
5. Build the deployable artifacts (jar, source jar, Javadoc jar)
6. Sign the artifacts with GPG so OSS Sonatype will accept them
7. Deploy the artifacts to staging area, automatically closing and releasing to Maven Central upon successful upload
8. Commit the updated POM and tag the commit with the new version
9. Push changes back to the Github repository

Introducing CircleCI

My weapon of choice is CircleCI because of its intuitive design to build scripts, standard integration with Github, its potential for speeding up complex builds and standard Docker integration.

Frankly speaking, I was so glad I finally got everything working perfectly that I first wanted to write everything down before attempting the same setup with Azure Devops and Gitlab.

CircleCI (2.1) works with something called “workflows“, which is basically a pipeline of several build jobs, which if defined smartly, can run parallel. Moreover, one job type is a “manual approval” job, which can be used to force a specific path in a workflow. I use this technique to manually select an automated patch, minor or major release.

Checkout the source code from Github.com

CircleCI seamlessly integrates with public Github repo’s, so it can import Simple Java Mail automatically. CircleCI manages its own SSH key registration with the repository (with your confirmation) for read access and can checkout the code during the build.

Compile, test the project

To compile and test we need a docker image with Maven and specifically for Simple Java Mail: JDK 8. circleci/openjdk:8u171-jdk will do the trick nicely (complete list here).

Let’s define our initial flow with our selected container, run tests and collect our artifacts:

Since we have a separate build job for producing the deployable artifacts (because we don’t know the release version yet), we can skip some things here to speed up this job, such as producing javadoc.

Manually select patch, minor or major version release

In Jenkins or Bamboo I would configure target environments to pick up the “shared artifacts” and trigger the right version bump manually, but CircleCI works a bit differently with its “workflow” approach.

Instead of deployment pipelines, CircleCI has a special type of build job that will pause for manual confirmation. The subsequent build jobs will wait until it is approved. This way you can implement multiple deployment pipelines within one workflow. The way I’m using it though, I haven’t seen that on the web yet.

Here’s what the update CircleCI config looks like:

The result looks like this in CircleCI:

Auto-update POM with semver based on manual selection

Ok, so now that we know based on the workflow execution path what version bump we want to perform, how can we actually do the version bump?

There is a little bit of an obscure Maven feature that was undocumented for a long time: versions:set combined with build-helper:parse-version. For example, to bump the minor version (ie. 2.3.4 becomes 2.4.4), you can do the following:

How it works

What happens is that versions:set performs the actual update to the POM and will look for a property newVersion. We use build-helper:parse-version to produce that variable using properties available only to the build-helper. We need to escape the $-signs, because otherwise Bash will try to resolve them before they reach Maven. Finally versions:commit just gets rid of the POM backups from before the version bump.

Build the deployable artifacts (jar, source jar, javadoc jar)

Build your artifacts as you normally would, but use a custom maven settings.xml for your build. We’ll need it to configure GPG and OSS Sonatype login credentials in the next step.

We’ll use it in our deploy in the next step like so:

Since we have a separate build job for compiling and testing the code, we can skip things like testing, instrumentation, spotbugs/pmd etc. by providing the options -DskipTests and -Dspotbugs.skip=true.

Sign the artifacts with GPG so OSS Sonatype will accept them

Now it gets interesting, because you’ll have to configure some keys and secrets as environmental string variables so you can refer to it from your build script.

Here’s our checklist:
1. produce a GPG key pair with passphrase
2. distribute the public key to one of the public servers OSS Sonatype validates signed artifacts with
3. make the private available in CircleCI as environment variable
4. Include the passphrase as environment variable so you can use the private key for signing the deployable artifacts

Introducing OSS Sonatype

Sonatype is an artifact server that synchronizes to Maven Central if you release a non-SNAPSHOT deploy. It has some rules for artifacts it can accept such as source, javadoc and binary jars should all be present and signed with GPG.

To continue, please first register your OSS project with OSS Sonatype if you haven’t yet and then complete the steps outlined in Sonatype’s guide to GPG keys, including uploading it to one of the public key servers.

From CircleCI to OSS Sonatype

Now that we have an OSS Sonatype project and distributed a public GPG key, we can start signing and releasing artifacts to Maven Central.

Adding the private GPG key to CircleCI

Take your private key in ASCII, which should be something like secring.gpg.asc. If you only have a .gpg file, you need to convert it to ASCII first. This is dangerous, so throw it away after you’re done adding it to CircleCI:

To get your ASCII key on a single line, you can use sed in linux with some black magic regex, or much simpler: paste it in an base64 converter and convert it to a base64 string. Import this string as environment variable and also add you GPG passphrase:

Configure Maven to connect to OSS Sonatype

We’ll define a Maven profile for GPG signing that is deactivated by default, so that we don’t have to deal with that when testing things locally on our own laptops. What’s more, OSS Sonatype requires you to define a couple of things before it accepts your artifacts, such as a developer tag:

For the maven-release-settings.xml to work you need to add your OSS Sonatype credentials to CircleCI as well:

Now that we configured our deployment plugins to sign artifacts and connect to OSS Sonatype, deploy the signed artifacts to staging area, automatically closing and releasing to Maven Central upon successful upload (or else you still need to manually login into OSS Sonatype to release it):

First define a command we can call from our deploy job that will configure GPG by importing our base64 ASCII key into the GPG tool already included in the Docker image:

Then implement the deploy jobs for the three semver deploy paths:

If everything was configured correctly, your script should now build, test, sign and deploy to Maven Central via OSS Sonatype.

Commit the updated POM and tag the commit with the new version

In order to provide a commit message with the new Maven version as well as tagging with that version, you need Maven to tell you that version first so you can store it in a variable. This is a little tricky, but can be done with a Command substitution.

Notice the text “[skip ci]”? That’s so CircleCI doesn’t trigger another build for this commit. It’s a convention which is also supported by other vendors (for example TravisCI).

Push changes back to repo

CircleCI setup a read-only SSH key for checking out the repo, but now you need to push something back. This means you need to provide your own SSH key pair that has write access. Moreover, you will need to explicitly acknowledge github.com as a trusted host by providing the server’s fingerprint.

Adding Github.com as a trusted host

Following this SO, here is how you can obtain github.com’s fingerprint as base64 (1st command):

Manually verify the fingerprint (2nd command) is the same as the fingerprint Github published, and then add the entire content of the file we just created to CircleCI:

Finally add this fingerprint to trusted hosted in your deploy script:

Configuring GIT to use our SSH key and user

Generate a new key pair (I did without password) and save it to .\github_rsa.key (the command will prompt you for it):

Now copy paste the content of the public key (github_rsa.key.pub) to Github in your repo and make sure to check “Allow write access”:

Take the private key and again convert it to base64 and add it to CircleCI environment variables for your project:

Now you can refer to it from your CircleCI deploy script. Let’s take the fingerprint script and club it together with the SSH key config in a new command to keep things tidy:

Finally, performing the push to repo

With the fingerprint and SSH key in place, we can finally perform the last step in our CI/CD script: push the change and tag back to the repo.

To perform GIT commands with an SSH key, you need to write the commands a little differently:

The final deploy scripts

• .circleci/maven-release-settings.xml
• .circleci/config.yml
• pom.xml

To make this work you need github.com’s fingerprint as environment variable as well ass OSS Sonatype login credentials, GPG signing key and passphrase, and GIT read/write SSH key.