Tools: Support multiple Go versions in CI/Dev. (#3622)

This PR updates the `test/boulder-tools/tag_and_upload.sh` script to template a `Dockerfile` for building multiple copies of `boulder-tools`: one per supported Go version. Unfortunately this is required because only Docker 17+ supports an env var in a Dockerfile `FROM`. It's best if we can stay on package manger installed versions of Docker which precludes 17+ 😞.

The `docker-compose.yml` is updated to version "3" to allow specifying a `GO_VERSION` env var in the respective Boulder `image` directives. This requires `docker-compose` version 1.10.0+ which in turn requires Docker engine version 1.13.0+. The README is updated to reflect these new requirements. This Docker engine version is commonly available in package managers (e.g. Ubuntu 16.04). A sufficient `docker-compose` version is not, but this is a simple one binary Go application that is easy to update outside of package managers.

The `.travis.yml` config file is updated to set the `GO_VERSION` in the build matrix, allowing build tasks for different Go versions. Since the `docker-compose.yml` now requires `docker-compose` 1.10.0+ the
`.travis.yml` also gains a new `before_install` for setting up a modern `docker-compose` version.

Lastly tools and images are updated to support both Go 1.10 (our current Go version) and Go 1.10.1 (the new point release). By default Go 1.10 is used, we can switch this once staging/prod are updated.

_*TODO*: One thing I haven't implemented yet is a `sed` expression in `tag_and_upload.sh` that updates both `image` lines in `docker-compose.yml` with an up-to-date tag. Putting this up for review while I work on that last creature comfort._

Resolves https://github.com/letsencrypt/boulder/issues/3551

Replaces https://github.com/letsencrypt/boulder/pull/3620 (GH got stuck from a yaml error)

.travis.yml

@@ -32,6 +32,9 @@ branches:
 
 matrix:
   include:
+    #
+    # Current Go version build tasks:
+    #
     - env: RUN="vet fmt migrations integration godep-restore errcheck generate dashlint rpm"
     # Config changes that have landed in master but not yet been applied to
     # production can be made in boulder-config-next.json.
@@ -39,13 +42,34 @@ matrix:
     - env: RUN="unit"
     - env: RUN="unit-next" BOULDER_CONFIG_DIR="test/config-next"
     - env: RUN="coverage"
+
+    #
+    # Next Go version build tasks:
+    #
+    - env: GO_VERSION="1.10.1" RUN="vet fmt migrations integration godep-restore errcheck generate dashlint rpm"
+    # Config changes that have landed in master but not yet been applied to
+    # production can be made in boulder-config-next.json.
+    - env: GO_VERSION="1.10.1" RUN="integration" BOULDER_CONFIG_DIR="test/config-next"
+    - env: GO_VERSION="1.10.1" RUN="unit"
+    - env: GO_VERSION="1.10.1" RUN="unit-next" BOULDER_CONFIG_DIR="test/config-next"
   fast_finish: true
   allow_failures:
     - env: RUN="coverage"
 
+# We require a newer version of docker-compose than is installed by way of the
+# "services: docker" directive. Per the travis docs[0] this is best remedied
+# with the following `before_install` commands.
+# [0]: https://docs.travis-ci.com/user/docker/#Using-Docker-Compose
+before_install:
+  # We install `docker-compose` to $HOME/bin to avoid needing root perms.
+  # Afterwards where required we refer to the `docker-compose` binary by
+  # absolute path.
+  - curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-$(uname -s)-$(uname -m) > $HOME/bin/docker-compose
+  - chmod +x $HOME/bin/docker-compose
+
 install:
   - ./test/travis-before-install.sh
-  - docker-compose pull
+  - $HOME/bin/docker-compose pull
 
 script:
-  - docker-compose run -e BOULDER_CONFIG_DIR="${BOULDER_CONFIG_DIR}" -e RUN="${RUN}" -e TRAVIS="${TRAVIS}" -e TRAVIS_COMMIT="${TRAVIS_COMMIT}" -e TRAVIS_PULL_REQUEST="${TRAVIS_PULL_REQUEST}" -e TRAVIS_PULL_REQUEST="${TRAVIS_PULL_REQUEST}" -e TRAVIS_JOB_ID="${TRAVIS_JOB_ID}" -e COVERALLS_TOKEN="${COVERALLS_TOKEN}" boulder  ./test.sh
+  - $HOME/bin/docker-compose run -e BOULDER_CONFIG_DIR="${BOULDER_CONFIG_DIR}" -e RUN="${RUN}" -e TRAVIS="${TRAVIS}" -e TRAVIS_COMMIT="${TRAVIS_COMMIT}" -e TRAVIS_PULL_REQUEST="${TRAVIS_PULL_REQUEST}" -e TRAVIS_PULL_REQUEST="${TRAVIS_PULL_REQUEST}" -e TRAVIS_JOB_ID="${TRAVIS_JOB_ID}" -e COVERALLS_TOKEN="${COVERALLS_TOKEN}" boulder ./test.sh

README.md

@@ -20,8 +20,8 @@ in that directory:
     git clone https://github.com/letsencrypt/boulder/ $GOPATH/src/github.com/letsencrypt/boulder
     cd $GOPATH/src/github.com/letsencrypt/boulder
 
-Additionally, make sure you have Docker Engine 1.10.0+ and Docker Compose
-1.6.0+ installed. If you do not, you can follow Docker's [installation
+Additionally, make sure you have Docker Engine 1.13.0+ and Docker Compose
+1.10.0+ installed. If you do not, you can follow Docker's [installation
 instructions](https://docs.docker.com/compose/install/).
 
 We recommend having **at least 2GB of RAM** available on your Docker host. In

docker-compose.yml

@@ -1,8 +1,8 @@
-version: '2'
+version: '3'
 services:
     boulder:
         # To minimize fetching this should be the same version used below
-        image: letsencrypt/boulder-tools:2018-03-07
+        image: letsencrypt/boulder-tools-go${GO_VERSION:-1.10.0}:2018-04-05
         environment:
             FAKE_DNS: 127.0.0.1
             PKCS11_PROXY_SOCKET: tcp://boulder-hsm:5657
@@ -43,7 +43,7 @@ services:
         working_dir: /go/src/github.com/letsencrypt/boulder
     bhsm:
         # To minimize fetching this should be the same version used above
-        image: letsencrypt/boulder-tools:2018-03-07
+        image: letsencrypt/boulder-tools-go${GO_VERSION:-1.10.0}:2018-04-05
         environment:
             PKCS11_DAEMON_SOCKET: tcp://0.0.0.0:5657
         command: /usr/local/bin/pkcs11-daemon /usr/lib/softhsm/libsofthsm2.so

test/boulder-tools/Dockerfile.tmpl

@@ -1,4 +1,4 @@
-FROM golang:1.10
+FROM golang:%%GO_VERSION%%
 
 ADD build.sh /tmp/build.sh
 RUN /tmp/build.sh

test/boulder-tools/README.md

@@ -0,0 +1,23 @@
+# Boulder-Tools Docker Image Utilities
+
+In CI and our development environment we do not rely on the Go environment of
+the host machine, and instead use Go installed in a container. To simplify
+things we separate all of Boulder's build dependencies into its own
+`boulder-tools` Docker image.
+
+## Go Versions
+
+Rather than install multiple versions of Go within the same `boulder-tools`
+container we maintain separate images for each Go version we support.
+
+When a new Go version is available we perform serveral steps to integrate it to our workflow:
+
+1. We add it to the `GO_VERSIONS` array in `tag_and_upload.sh`.
+2. We run the `tag_and_upload.sh` script to build, tag, and upload
+   a `boulder-tools` image for each of the `GO_VERSIONS`
+3. We update `.travis.yml`, duplicating the existing build tasks, adding new
+   `GO_VERSION=` `ENV` entries for the new Go version.
+
+After some time when we have spot checked the new Go release and coordinated
+a staging/prod environment upgrade with the operations team we can remove the
+old `GO_VERSIONS` entries and delete their respective build matrix items.

test/boulder-tools/tag_and_upload.sh

@@ -3,15 +3,48 @@
 cd $(dirname $0)
 
 DATESTAMP=$(date +%Y-%m-%d)
-TAG_NAME="letsencrypt/boulder-tools:$DATESTAMP"
+BASE_TAG_NAME="letsencrypt/boulder-tools"
+GO_VERSIONS=( "1.10.0" "1.10.1" )
 
-echo "Building boulder-tools image $TAG_NAME"
-docker build . -t $TAG_NAME --no-cache
+# Build a tagged image for each GO_VERSION
+for GO_VERSION in "${GO_VERSIONS[@]}"
+do
+  TAG_NAME="$BASE_TAG_NAME-go$GO_VERSION:$DATESTAMP"
+  echo "Building boulder-tools image $TAG_NAME"
 
-echo "Image ready."
+  # NOTE(@cpu): Generating DOCKERFILE's on disk with the %%GO_VERSION%%
+  # templated out by `sed` is required because only Docker v17+ supports env var
+  # interpolation in Dockerfile `FROM` directives. This version isn't commonly
+  # packaged so we rely on this technique for the time being. Similarly, it
+  # would be cleaner if we could just output the `sed` directly to the `docker
+  # build` stdin but that requires Docker 17+ too! :'(
+  DOCKERFILE="golang.$GO_VERSION.Dockerfile"
+  sed -r \
+    -e 's!%%GO_VERSION%%!'"$GO_VERSION"'!g' \
+    "Dockerfile.tmpl" > "$DOCKERFILE"
+
+  # Build the docker image using the templated Dockerfile, tagging it with
+  # TAG_NAME
+  docker build . \
+    -t $TAG_NAME \
+    --no-cache \
+    -f "$DOCKERFILE"
+
+  # Clean up the temp. Dockerfile
+  rm "$DOCKERFILE"
+done
+
+# Log in once now that images are ready to upload
+echo "Images ready, please login to allow Dockerhub push"
 docker login
 
-echo "Pushing $TAG_NAME to Dockerhub"
-docker push $TAG_NAME
+# Upload a tagged image for each GO_VERSION
+for GO_VERSION in "${GO_VERSIONS[@]}"
+do
+  TAG_NAME="$BASE_TAG_NAME-go$GO_VERSION:$DATESTAMP"
+  echo "Pushing $TAG_NAME to Dockerhub"
+  docker push $TAG_NAME
+done
 
-sed -i "s,image: letsencrypt/boulder-tools.*,image: $TAG_NAME," ../../docker-compose.yml
+# TODO(@cpu): Figure out a `sed` for updating the date in `docker-compose.yml`'s
+# `image` lines with $DATESTAMP