boulder/docs/release.md

Boulder Release Process

A description and demonstration of the full process for tagging a normal weekly release, a "clean" hotfix release, and a "dirty" hotfix release.

Once a release is tagged, it will be generally deployed to staging and then to production over the next few days.

Goals

1. All development, including reverts and hotfixes needed to patch a broken release, happens on the main branch of this repository. Code is never deployed without being reviewed and merged here first, and code is never landed on a release branch that isn't landed on main first.

2. Doing a normal release requires approximately zero thought. It Just Works.

3. Doing a hotfix release differs as little as possible from the normal release process.

Release Schedule

Boulder developers make a new release at the beginning of each week, typically around 10am PST Monday. Operations deploys the new release to the staging environment on Tuesday, typically by 2pm PST. If there have been no issues discovered with the release from its time in staging, then on Thursday the operations team deploys the release to the production environment.

Holidays, unexpected bugs, and other resource constraints may affect the above schedule and result in staging or production updates being skipped. It should be considered a guideline for normal releases but not a strict contract.

Release Structure

All releases are tagged with a tag of the form release-YYYY-MM-DD[x], where the YYYY-MM-DD is the date that the initial release is cut (usually the Monday of the current week), and the [x] is an optional lowercase letter suffix indicating that the release is an incremental hotfix release. For example, the second hotfix release (i.e. third release overall) in the third week of January 2022 was release-2022-01-18b.

All release tags are signed with a key associated with a Boulder developer. Tag signatures are automatically verified by GitHub using the public keys that developer has uploaded, and are additionally checked before being built and deployed to our staging and production environments. Note that, due to how Git works, in order for a tag to be signed it must also have a message; we set the tag message to just be a slightly more readable version of the tag name.

Making a Release

Prerequisites

• You must have a GPG key with signing capability:

  • Checking for existing GPG keys

• If you don't have a GPG key with signing capability, create one:

  • Generating a new local GPG key
  • Generating a new Yubikey GPG key

• The signing GPG key must be added to your GitHub account:

  • Adding a new GPG key to your GitHub account

• git may need to be configured to call the correct GPG binary:

  • The default: git config --global gpg.program gpg is correct for most Linux platforms
  • On macOS and some Linux platforms: git config --global gpg.program gpg2 is correct

• git must be configured to use the correct GPG key:

  • Telling Git about your GPG key

• Understand the process for signing tags

Regular Releases

Simply create a signed tag whose name and message both include the date that the release is being tagged (not the date that the release is expected to be deployed):

go run github.com/letsencrypt/boulder/tools/release/tag@main

This will print the newly-created tag and instructions on how to push it after you are satisfied that it is correct. Alternately you can run the command with the -push flag to push the resulting tag automatically.

Hotfix Releases

Sometimes it is necessary to create a new release which looks like a prior release but with one or more additional commits added. This is usually the case when we discover a critical bug in the currently-deployed version that needs to be fixed, but we don't want to include other changes that have already been merged to main since the currently-deployed release was tagged.

In this situation, we create a new hotfix release branch starting at the point of the previous release tag. We then use the normal GitHub PR and code-review process to merge the necessary fix(es) to the branch. Finally we create a new release tag at the tip of the release branch instead of the tip of main.

To create the new release branch, substitute the name of the release tag which you want to use as the starting point into this command:

go run github.com/letsencrypt/boulder/tools/release/branch@main v0.YYYYMMDD.0

This will create a release branch named release-branch-v0.YYYYMMDD. When all necessary PRs have been merged into that branch, create the new tag by substituting the branch name into this command:

go run github.com/letsencrypt/boulder/tools/release/tag@main release-branch-v0.YYYYMMDD

Deploying Releases

When doing a release, SRE's tooling will check that:

1. GitHub shows that tests have passed for the commit at the planned release tag.

2. The planned release tag is an ancestor of the current main on GitHub, or the planned release tag is equal to the head of a branch named release-branch-XXX, and all commits between main and the head of that branch are cherry-picks of commits which landed on main following the normal review process.