Commit Graph

213 Commits

Author SHA1 Message Date
Aaron Gable 18216a7ea8
Run CI tests on go1.20 (#6550)
Add go1.20 as a new version to run tests on, and to build release
artifacts from. Fix one test which was failing because it was
accidentally relying on consistent (i.e. unseeded) non-cryptographic
random number generation, which go1.20 now automatically seeds at import
time.

Update the version of golangci-lint used in our docker containers to the
new version that has go1.20 support. Remove a number of nolint comments
that were required due to an old version of the gosec linter.
2023-02-03 11:57:07 -08:00
Jacob Hoffman-Andrews 85e8f1f5cf
Change GHA release workflow to not use artifacts (#6590)
Fixes #6571
2023-01-19 14:30:26 -08:00
Aaron Gable 86622654fc
Run tests on go1.19.5 (#6576)
Run go1.19.5 alongside go1.19.2 for a while.

Fixes #6574
2023-01-11 11:37:02 -08:00
Samantha 6c6da76400
ROCSP: Replace Redis Cluster with a consistently sharded all-primary nodes (#6516) 2022-12-19 15:06:47 -05:00
Aaron Gable 89f7fb1636
Clean up go1.19 TODOs (#6464)
Clean up several spots where we were behaving differently on
go1.18 and go1.19, now that we're using go1.19 everywhere. Also
re-enable the lint and generate tests, and fix the various places where
the two versions disagreed on how comments should be formatted.

Also clean up the OldTLS codepaths, now that both go1.19 and our
own feature flags have forbidden TLS < 1.2 everywhere.

Fixes #6011
2022-10-21 15:54:18 -07:00
Aaron Gable 410732e8a7
Remove go1.18 from testing (#6459)
We are no longer running on go1.18 in production.
2022-10-21 14:55:37 -07:00
Samantha 90eb90bdbe
test: Replace sd-test-srv with consul (#6389)
- Add a dedicated Consul container
- Replace `sd-test-srv` with Consul
- Add documentation for configuring Consul
- Re-issue all gRPC credentials for `<service-name>.service.consul`

Part of #6111
2022-09-19 16:13:53 -07:00
Samantha bc1bf0fde4
test: Support multiple database schemas (#6344)
In dev docker we've always used a single schema (`boulder_sa`), with two
environments (`test` and `integration`) making for a combined total of two
databases sharing the same users and schema (e.g. `boulder_sa_test` and
`boulder_sa_integration`). There are also two versions of this schema. `db` and
`db-next`. The former is the schema as it should exist in production and the
latter is everything from `db` with some un-deployed schema changes. This change
adds support for additional schemas with the same aforementioned environments
and versions.

- Add support for additional schemas in `test/create_db.sh` and sa/migrations.sh
- Add new schema `incidents_sa` with its own users
- Replace `bitbucket.org/liamstask/goose/` with `github.com/rubenv/sql-migrate`

Part of #6328
2022-09-07 14:59:08 -07:00
Samantha badd64c66e
docker-compose: Fix indentation (#6346) 2022-09-02 14:02:49 -07:00
Aaron Gable 00734a6edf
Stop rsyslog from de-duplicating log lines (#6291)
When rsyslog receives multiple identical log lines in a row, it can
collapse those lines into a single instance of the log line and a
follow-up line saying "message repeated X times". However, that
rsyslog-generated line does not contain our log line checksum, so it
immediately causes log-validator to complain about the line. In
addition, the rsyslog docs themselves state that this feature is a
misfeature and should never be turned on. Despite this, Ubuntu turns the
feature on by default when the rsyslog package is installed from apt.

Add an additional command to our dockerfile which overwrites Ubuntu's
default setting to disable this misfeature, and update our test
environment to use the new docker image.

Fixes #6252
2022-08-11 12:37:16 -07:00
Aaron Gable d1b211ec5a
Start testing on go1.19 (#6227)
Run the Boulder unit and integration tests with go1.19.

In addition, make a few small changes to allow both sets of
tests to run side-by-side. Mark a few tests, including our lints
and generate checks, as go1.18-only. Reformat a few doc
comments, particularly lists, to abide by go1.19's stricter gofmt.

Causes #6275
2022-08-10 15:30:43 -07:00
Aaron Gable b3c42639c1
Update Boulder to go1.18.4 (#6240)
Version go1.18.4 contains a number of security fixes related
to stack exhaustion in a variety of standard library packages,
some of which we (directly or indirectly) rely on.

Full release notes are at:
https://groups.google.com/g/golang-announce/c/nqrv9fbR0zE/m/3SeTTJs9AwAJ
2022-07-22 11:14:47 -07:00
Aaron Gable 9b4ca235dd
Update boulder-tools dependencies (#6129)
Update:
- golangci-lint from v1.42.1 to v1.46.2
- protoc from v3.15.6 to v3.20.1
- protoc-gen-go from v1.26.0 to v1.28.0
- protoc-gen-go-grpc from v1.1.0 to v1.2.0
- fpm from v1.14.0 to v1.14.2

Also remove a reference to go1.17.9 from one last place.

This does result in updating all of our generated .pb.go files, but only
to update the version number embedded in each file's header.

Fixes #6123
2022-05-20 14:24:01 -07:00
Aaron Gable 7b6b914697
Use go1.18.1 by default (#6081)
This also updates the version built by the build and release action.
2022-05-03 13:19:02 -07:00
Aaron Gable 802acc510f
Use Redis 6.2.7 because Redis 7.0.0 breaks go-redis (#6073)
Redis recently released version 7.0.0, which has several breaking
changes. The go-redis library that we rely on does not yet support
communicating with a Redis 7.0.0 cluster.

Pin ourselves to the latest non-7.0.0 version, 6.2.7, until such time
as go-redis releases a version with support for 7.0.0.

Fixes #6071
2022-05-02 11:42:02 -07:00
Jacob Hoffman-Andrews cf9df961ba
Add feature flags for upcoming deprecations (#6043)
This adds three features flags: SHA1CSRs, OldTLSOutbound, and
OldTLSInbound. Each controls the behavior of an upcoming deprecation
(except OldTLSInbound, which isn't yet scheduled for a deprecation
but will be soon). Note that these feature flags take advantage of
`features`' default values, so they can default to "true" (that is, each
of these features is enabled by default), and we set them to "false"
in the config JSON to turn them off when the time comes.

The unittest for OldTLSOutbound requires that `example.com` resolves
to 127.0.0.1. This is because there's logic in the VA that checks
that redirected-to hosts end in an IANA TLD. The unittest relies on
redirecting, and we can't use e.g. `localhost` in it because of that
TLD check, so we use example.com.

Fixes #6036 and #6037
2022-04-15 12:14:00 -07:00
Samantha bafe45f301
Update Go to 1.17.9 and 1.18.1 (#6047)
go1.17.9 (released 2022-04-12) includes security fixes to the crypto/elliptic and encoding/pem packages, as well as bug fixes to the linker and runtime. See the [Go 1.17.9 milestone](https://github.com/golang/go/issues?q=milestone%3AGo1.17.9+label%3ACherryPickApproved) on our issue tracker for details.

go1.18.1 (released 2022-04-12) includes security fixes to the crypto/elliptic, crypto/x509, and encoding/pem packages, as well as bug fixes to the compiler, linker, runtime, the go command, vet, and the bytes, crypto/x509, and go/types packages. See the [Go 1.18.1 milestone](https://github.com/golang/go/issues?q=milestone%3AGo1.18.1+label%3ACherryPickApproved) on our issue tracker for details.
2022-04-12 19:32:42 -07:00
Jacob Hoffman-Andrews 7d00d9fbcf
Use go1.18 in CI, and fix up Docker image (#6002)
- Remove GOPATH-style path structure, which isn't needed with Go
modules.
- Remove check for existing of docker buildx builder instance, since it
was unreliable.
2022-03-21 12:24:13 -07:00
Aaron Gable 3f26ac0768
Remove false positive in go version comment (#5995) 2022-03-15 10:41:28 -07:00
alexzorin 73ed095f0d
docker: re-export OCSP ports (#5994) 2022-03-14 22:56:59 -07:00
Jacob Hoffman-Andrews edcf4ac18b
Unexport some ports in docker-compose.yml (#5990) 2022-03-14 11:54:02 -07:00
Samantha 21416b4d72
Post Debian package on release tag cut (#5977)
- Add a CI workflow which publishes a GitHub Release containing a Debian package
  when a release tag is pushed
- Add a script, called by the CI host, that installs all of the dependencies
  necessary to `make` a Debian package
- Remove the, now defunct, goreleaser config file

Fixes #5970
2022-03-10 16:00:39 -08:00
Jacob Hoffman-Andrews 6395701244
Update CI to test go1.18beta2 (#5982)
This requires using GODEBUG to enable a couple of thing turned off by go1.18 (TLS 1.0/1.1, SHA-1 CSRs).

Also add help for a failure mode of cross builds.
2022-03-09 13:42:15 -08:00
Andrew Gabbitas 3bb3421631
Remove go 1.17.5 from test matrix (#5940) 2022-02-12 12:03:23 -07:00
Aaron Gable ad0e56ec4a
Remove test coverage on go1.17 (#5882) 2022-01-14 16:22:24 -08:00
Samantha 7d4facc403
test: Install arm64 protobuf for arm64 docker images (#5880) 2022-01-13 13:45:38 -08:00
Samantha 8a1b51f81b
Use go install for dep binaries when building docker (#5858)
Update the way our docker build script installs the binaries we rely on
at runtime to avoid "go get" deprecation warnings.

Fixes #5744
2021-12-16 19:02:46 -08:00
Aaron Gable c88640c816
Run tests on Go 1.17.5 (#5845)
Build a new docker container for the new Go 1.17.5 security release,
which includes a fix for the `net/http` package. Update our CI to run
tests on both our current and the new go versions.
2021-12-10 14:44:23 -08:00
Aaron Gable 5c02deabfb
Remove wfe1 integration tests (#5840)
These tests are testing functionality that is no longer in use in
production deployments of Boulder. As we go about removing wfe1
functionality, these tests will break, so let's just remove them
wholesale right now. I have verified that all of the tests removed in
this PR are duplicated against wfe2.

One of the changes in this PR is to cease starting up the wfe1 process
in the integration tests at all. However, that component was serving
requests for the AIA Issuer URL, which gets queried by various OCSP and
revocation tests. In order to keep those tests working, this change also
adds an integration-test-only handler to wfe2, and updates the CA
configuration to point at the new handler.

Part of #5681
2021-12-10 12:40:22 -08:00
Samantha 65ed9c7974
test: Add support for multi-arch docker images (#5837) 2021-12-08 10:51:01 -08:00
Jacob Hoffman-Andrews 6b1ad1ce21
Enumerate bredis_N containers explicitly (#5774)
Previously we were using the `deploy:` config field, but that's not
supported in some cases. Splitting things out also allows us to
explicitly assign IP addresses rather than relying on their most-likely
assignment to containers.
2021-11-03 17:16:19 -07:00
Jacob Hoffman-Andrews 7fab32a000
Add rocsp-tool to manually store OCSP responses in Redis (#5758)
This is a sort of proof of concept of the Redis interaction, which will
evolve into a tool for inspection and manual repair of missing entries,
if we find ourselves needing to do that.

The important bits here are rocsp/rocsp.go and
cmd/rocsp-tool/main.go. Also, the newly-vendored Redis client.
2021-11-02 11:04:03 -07:00
Jacob Hoffman-Andrews c1d221abe6
Add Redis to Boulder's docker-compose (#5747)
This gets us ready to add writing to Redis from ocsp-updater. The Go
redis client requires different configuration for cluster operation
than non-cluster, so we need to simulate a cluster in our integration
environment. Cluster operation requires a manual initialization step,
which you can do like so:

```
docker-compose up -d bredis docker-compose exec bredis bash
/test/redis-create.sh
```

I still need to figure out how to make that happen automatically during
integration tests and when you run docker-compose up.

The hex values in redis.config are randomly generated passwords for the
different users.

Fixes #5723
2021-10-28 10:36:11 -07:00
Aaron Gable eb5d0e9ba9
Update golangci-lint from v1.29.0 to v1.42.1 (#5745)
Update the version of golangci-lint we use in our docker image,
and update the version of the docker image we use in our tests.
Fix a couple places where we were violating lints (ineffective assign
and calling `t.Fatal` from outside the main test goroutine), and add
one lint (using math/rand) to the ignore list.

Fixes #5710
2021-10-22 16:26:59 -07:00
Jacob Hoffman-Andrews ba0ea090b2
integration: save hierarchy across runs (#5729)
This allows repeated runs using the same hiearchy, and avoids spurious
errors from ocsp-updater saying "This CA doesn't have an issuer cert
with ID XXX"

Fixes #5721
2021-10-20 17:06:33 -07:00
Jacob Hoffman-Andrews 1309da6275
Consolidate name resolution in sd-test-srv. (#5709)
Previously we relied on aliases in Docker's DNS for some names, and
sd-test-srv for some other names. This moves them all into sd-test-srv.
2021-10-13 18:38:38 -07:00
Andrew Gabbitas 258e0cf7c1
Remove go 1.16.6 from testing (#5636)
Fixes: #5585
2021-09-07 15:34:54 -06:00
Aaron Gable 4c581436a3
Add go1.17beta1 to CI (#5483)
Add go1.17beta1 docker images to the set of things we build,
and integrate go1.17beta1 into the set of environments CI runs.
Fix one test which breaks due to an underlying refactoring in
the `crypto/x509` stdlib package. Fix one other test which breaks
due to new guarantees in the stdlib's TLS ALPN implementation.

Also removes go1.16.5 from CI so we're only running 2 versions.

Fixes #5480
2021-07-13 10:00:04 -07:00
Andrew Gabbitas fc9ac77a69
Add testing for go1.16.6 (#5524) 2021-07-12 23:41:06 -06:00
Aaron Gable 923aef5839
Update to go1.16.5 (#5482)
Includes a number of updates to packages we use: math/big,
net, net/http/httputil, and archive/zip. See release notes at
https://golang.org/doc/devel/release#go1.16.minor

Fixes #5464
2021-06-15 11:42:04 -07:00
Andrew Gabbitas 401df1c48a
Remove go1.16.2 from ci matrix (#5417)
Fixes #5416
2021-05-14 11:04:25 -07:00
Aaron Gable a19ebfa0e9
VA: Query SRV to preload/cache DNS resolver addrs (#5360)
Abstract out the way that the bdns library keeps track of the
resolvers it uses to do DNS lookups. Create one implementation,
the `StaticProvider`, which behaves exactly the same as the old
mechanism (providing whatever names or addresses were given
in the config). Create another implementation, `DynamicProvider`,
which re-resolves the provided name on a regular basis.

The dynamic provider consumes a single name, does a lookup
on that name for any SRV records suggesting that it is running a
DNS service, and then looks up A records to get the address of
all the names returned by the SRV query. It exports its successes
and failures as a prometheus metric.

Finally, update the tests and config-next configs to work with
this new mechanism. Give sd-test-srv the capability to respond
to SRV queries, and put the names it provides into docker's
default DNS resolver.

Fixes #5306
2021-04-20 10:11:53 -07:00
Jacob Hoffman-Andrews 795348483e
Fix boulder-tools tag in docker-compose.yml. (#5379) 2021-04-01 17:54:45 -07:00
Jacob Hoffman-Andrews 7194624191
Update grpc and protobuf to latest. (#5369)
protoc now generates grpc code in a separate file from protobuf code.
Also, grpc servers are now required to embed an "unimplemented"
interface from the generated .pb.go file, which provides forward
compatibility.

Update the generate.go files since the invocation for protoc has changed
with the split into .pb.org and _grpc.pb.go.

Fixes #5368
2021-04-01 17:18:15 -07:00
Andrew Gabbitas 3d9d5e2306
Cleanup go1.15.7 (#5374)
Remove code that is no longer needed after migrating to go1.16.x.
Remove testing with go1.15.7 in the test matrix.

Fixes #5321
2021-04-01 10:50:18 -07:00
Jacob Hoffman-Andrews b5471f36f0
Switch to OS-provided SoftHSM2. (#5365)
The OS-provided packages used to be too old for us, but now they are
not!
2021-03-30 17:37:58 -07:00
Samantha 1e5d89e6c8
Remove Coveralls (#5363)
- Remove `goveralls`, `gover`, and `cover` from `build.sh`.
- Remove `--coverage` option from `test.sh`.
- Update Docker image in `docker-compose.yml` and
  `.github/workflows/boulder-ci.yml`

Fixes #5357
2021-03-29 11:56:52 -07:00
Andrew Gabbitas 8315393eb1
Update tests to go1.16.2 (#5342)
* Update to go1.16.2

Fixes #5333
2021-03-16 14:56:20 -06:00
Samantha e5bff38c93
Cleanup docker-compose configuration (#5335)
- Remove environment variables no longer referenced in CI
- Standardize `environment` mapping to use key + value
- Standardize on two space indent
- Alias `image` to make version updates easier

Fixes #5334
2021-03-11 17:33:13 -08:00
Samantha fc53482cac
Run db-next migrations with config-next configuration (#5320)
Docker container should load the appropriate schema (`sa/_db` or
`sa/_db-next`) for the given configuration.

- Add `docker-compose.next.yml` docker-compose overrides
- Detect when to apply `sa/_db-next/migrations`
- Detect mismatch between `goose dbversion` and the latest migration
- Symlink `promoted` schema back to `sa/_db-next/migrations`
- Add tooling to consistently promote/demote schema migrations

Fixes #5300
2021-03-11 14:45:32 -08:00
Andrew Gabbitas ceffe18dfc
Add testing for golang 1.16 (#5313)
- Add 1.16.1 to the GitHub CI test matrix
- Fix tlsalpn tests for go 1.16.1 but maintain compatibility with 1.15.x
- Fix integration tests.

Fix: #5301
Fix: #5316
2021-03-11 11:47:41 -08:00
Andrew Gabbitas 066ed3427e
Add GitHub actions workflow for Boulder CI tests (#5308)
- Add GitHub actions workflow for Boulder CI tests in parity with Travis
  CI except the coverage test.
- Change boulder-tools docker image to push to a static docker repo
  instead of creating a new one each time. Use docker version tags and git
  hash to identify go versions in the repo.
- Change docker-compose to pull from the static boulder-tools repo. This
  breaks using the TRAVIS_GO_VERSION env variable to pull the docker image, but
  the default will still work with intent of decommissioning Travis-CI for
  GitHub CI.

Fix: #5289
2021-02-26 12:59:47 -08:00
Samantha 7cbfe94f30
Switch docker base image from buster-scm to focal-scm (#5287)
- Switch docker base image from`buster-scm` to `focal-scm`
- Add `dockerfile` lines to correct `/etc/rsyslog.conf` settings

Fixes #5295
2021-02-22 10:33:50 -08:00
Andrew Gabbitas e8a6eabbd5
Update to go1.15.7 (#5241)
Go version 1.15.7 is a security release.

Fixes: #5237
2021-01-20 11:38:38 -08:00
Samantha e8ca1987cd
Python upgrade os upgrades and travis config cleanup (#5186)
Modified the Dockerfile to build using Debian Buster, an upgrade from
Debian Stretch. The default Python 3 version for Stretch is 3.5.x which
is soon to de deprecated by Python-cryptography a dependency we rely on
for our integration test suite. The default Python 3 version for Debian
Buster is 3.7.x

In the .travis.yml file we are instructing travis to provision Xenial
instances and install two versions of Go. This change bumps Xenial
(16.04) -> Focal (20.04) and removes the installation of the two Go
versions; all of our testing happens inside of a docker container so
having Go installed on the Docker parent isn't necessary.

In the docker-compose.yml file we configure which docker image to pull
from Dockerhub, I've updated these to reflect the Debian Buster images
already built and pushed.

Modified build.sh to install mariadb-client-core 10.3, there is no 10.1
install candidate for Debian Buster and release notes for 10.2 and 10.3
indicate that these were both security releases.

Modified test.sh to use python3 instead of system python (usually 2.7)
for test/grafana/lints.py

Fixes #5180
2020-11-23 18:12:04 -08:00
Aaron Gable ad89e0b658
Update to go1.15.5 (#5179)
Go version 1.15.5 is a security release which introduces fixes
both to the big.Int package (which we use) and the go compiler
itself (which we use).

Release notes: https://golang.org/doc/go1.15

This change builds go1.15.5 versions of our docker containers,
adds tests on the new version to our travis config, and sets the
default to be the new version.

Fixes #5173
2020-11-12 13:31:23 -08:00
Phil Porada f0b54a8968
Upgrade mariadb from 10.3 to 10.5 (#5088) 2020-09-14 11:27:15 -07:00
Aaron Gable d1d9269530
Switch to testing only on go1.15 (#5022)
Now that our prod deployment has updated to using go1.15,
it is no longer be necessary to duplicate our tests across two
different go versions.
2020-08-24 13:47:50 -07:00
Aaron Gable 316ffd6241
Update to stable go1.15 (#5021)
This updates our test infrastructure from using go1.15rc2 to the
newly-released go1.15 stable version.
2020-08-17 12:32:20 -07:00
Aaron Gable d873e01fa8
Update to go1.15rc2 for tests (#5013)
Go 1.15rc2 was released today. The diff from rc1 only includes one
change to the crypto/ package, but worth upgrading just to be ready
for the official 1.15 stable release.
2020-08-10 11:24:32 -07:00
Roland Bracewell Shoemaker 75b034637b
Update travis go versions (remove 1.14.1, add 1.15rc1) (#5002)
Fixes #4919.
2020-08-04 12:13:09 -07:00
Aaron Gable 09917baf47
Add gosec linter to golangci-lint (#4990)
This enables the gosec linter. It also disables a number of
warnings which it emits on the current codebase. Some of these
(e.g. G104: Errors unhandled) we expect to leave disabled
permanently; others (e.g. G601: Implicit memory aliasing in for loop)
we expect to fix and then enable to prevent regressions.

Part of #4948
2020-07-28 16:42:17 -07:00
Aaron Gable 8923fa29a7
Remove infinite memlock rlimit from docker config (#4967)
This was necessary to work around a poor interaction between
Go 1.4.x and unpatched linux kernels. Although we are still using
the same version of Go, and the Linux project only released the
fix in kernel 5.4.2 and later, Ubuntu has backported the fix into
Focal Fossa 20.04's 5.4.0 kernel. Therefore this workaround is
no longer needed.
https://github.com/golang/go/issues/37436#issuecomment-657436406

This also removes one need for elevated permissions, making it
easier to use docker rootless for development.
2020-07-20 14:12:45 -07:00
Aaron Gable 696fc4e122
Enable testing on go1.14.5 (#4953)
New go version 1.14.5 was released today:
https://groups.google.com/g/golang-announce/c/XZNfaiwgt2w/m/E6gHDs32AQAJ
It includes a security fix for X.509 verification
(CVE-2020-14039, https://golang.org/issue/39360).
2020-07-14 11:28:39 -07:00
Jacob Hoffman-Andrews 18e7a57e24
Improve integration tests. (#4847)
Add passthrough for certain environment variables to
docker-compose.yml, making it easier to set them:

    RUN=unit docker-compose run --use-aliases boulder ./test.sh

Use 4001 instead of 4443 to monitor boulder-wfe2's health. This avoids
a spurious error log about a failed TLS handshake.

Remove unused code around running Certbot in integation tests.
2020-06-04 19:43:28 -07:00
Roland Bracewell Shoemaker 7673f02803
Use cmd/ceremony in integration tests (#4832)
This ended up taking a lot more work than I expected. In order to make the implementation more robust a bunch of stuff we previously relied on has been ripped out in order to reduce unnecessary complexity (I think I insisted on a bunch of this in the first place, so glad I can kill it now).

In particular this change:

* Removes bhsm and pkcs11-proxy: softhsm and pkcs11-proxy don't play well together, and any softhsm manipulation would need to happen on bhsm, then require a restart of pkcs11-proxy to pull in the on-disk changes. This makes manipulating softhsm from the boulder container extremely difficult, and because of the need to initialize new on each run (described below) we need direct access to the softhsm2 tools since pkcs11-tool cannot do slot initialization operations over the wire. I originally argued for bhsm as a way to mimic a network attached HSM, mainly so that we could do network level fault testing. In reality we've never actually done this, and the extra complexity is not really realistic for a handful of reasons. It seems better to just rip it out and operate directly on a local softhsm instance (the other option would be to use pkcs11-proxy locally, but this still would require manually restarting the proxy whenever softhsm2-util was used, and wouldn't really offer any realistic benefit).
* Initializes the softhsm slots on each integration test run, rather than when creating the docker image (this is necessary to prevent churn in test/cert-ceremonies/generate.go, which would need to be updated to reflect the new slot IDs each time a new boulder-tools image was created since slot IDs are randomly generated)
* Installs softhsm from source so that we can use a more up to date version (2.5.0 vs. 2.2.0 which is in the debian repo)
* Generates the root and intermediate private keys in softhsm and writes out the root and intermediate public keys to /tmp for use in integration tests (the existing test-{ca,root} certs are kept in test/ because they are used in a whole bunch of unit tests. At some point these should probably be renamed/moved to be more representative of what they are used for, but that is left for a follow-up in order to keep the churn in this PR as related to the ceremony work as possible)
Another follow-up item here is that we should really be zeroing out the database at the start of each integration test run, since certain things like certificates and ocsp responses will be signed by a key/issuer that is no longer is use/doesn't match the current key/issuer.

Fixes #4832.
2020-06-03 15:20:23 -07:00
Roland Bracewell Shoemaker 348b13f693
Set docker volumes to cache consistency mode (#4838)
(Only applies to OS X)

boulder is, typically, not a long lived docker container, and we don't
really care about synchronous consistency between the host fs and
container fs. cached provides the best performance for read-heavy
workloads, which is what is typically slowest on container startup
(at least from my experience).

This change provides a 30-40% speedup on OS X.
2020-06-02 15:00:57 -07:00
Jacob Hoffman-Andrews b1347fb3b3
Upgrade to latest protoc and protoc-gen-go (#4794)
There are some changes to the code generated in the latest version, so
this modifies every .pb.go file.

Also, the way protoc-gen-go decides where to put files has changed, so
each generate.go gets the --go_opt=paths=source_relative flag to
tell protoc to continue placing output next to the input.

Remove staticcheck from build.sh; we get it via golangci-lint now.

Pass --no-document to gem install fpm; this is recommended in the fpm docs.
2020-04-23 18:54:44 -07:00
Jacob Hoffman-Andrews 0b3223494c
Clean up tag_and_upload.sh. (#4791)
We used a template and sed in #3622 because common versions of Docker
didn't support build args. But now they do, so we can use the convenient
build args feature to parameterize which Go version to use.

Also, remove the --no-cache flag to docker build, which slows things
down unnecessarily.
2020-04-23 11:52:29 -07:00
Jacob Hoffman-Andrews 87fb6028c1
Add log validator to integration tests (#4782)
For now this mainly provides an example config and confirms that
log-validator can start up and shut down cleanly, as well as provide a
stat indicating how many log lines it has handled.

This introduces a syslog config to the boulder-tools image that will write
logs to /var/log/program.log. It also tweaks the various .json config
files so they have non-default syslogLevel, to ensure they actually
write something for log-validator to verify.
2020-04-20 13:33:42 -07:00
Jacob Hoffman-Andrews 908e8b5590
Test with Go 1.14.1 only. (#4785)
Now that we've deployed Go 1.14.1 in prod, we can remove 1.13.2 from our
test matrix.
2020-04-16 16:57:05 -07:00
Jacob Hoffman-Andrews db1a1a8595
Set memlock ulimit to unlimited. (#4770)
This works around a kernel bug that is tickled by Go 1.14:

https://github.com/golang/go/issues/37436
2020-04-13 15:49:38 -07:00
Jacob Hoffman-Andrews 9fbab1a1cd
Switch to golangci-lint for our lints. (#4766)
This makes it easier to configure additional linters, and provides us an
easy command to run locally.

The initial set of linters reflects those we are already running:

govet gofmt ineffassign errcheck misspell staticcheck

Note that misspell is in addition to the Python codespell package.

Since the invocation of these linters from golangci-lint is slightly
different from how we currently invoke them, there are some new
findings. This PR won't pass tests until #4763, #4764, and #4765 are
merged.

Incidentally, rename strat -> strategy to appeal misspell.
2020-04-09 09:42:24 -07:00
Jacob Hoffman-Andrews 28dbbce8da
Make boulder-tools image smaller. (#4732)
Instead of installing Certbot from the repo, install the python-acme
library (the only piece we need) from the apt repository. This also
allows us to skip installing build dependencies for Certbot.

Uninstall cmake after building.

Clean the various Go caches.

Move codespell and acme into requirements.txt. Don't use virtualenv anymore.

This reduces image size from 1.4 GB to 1.0 GB.

Incidentally, move the Go install to its own phase in the Dockerfile.
This will give it its own image layer, making rebuilds faster.
2020-04-06 16:53:42 -07:00
Jacob Hoffman-Andrews 995e25e8c3
Start testing Go 1.14.1 in Travis. (#4718)
As part of the process, pin specific versions of protoc-gen-go, mockgen,
and goveralls. Protoc-gen-go recently released a version that was incompatible
with our current version of gRPC. Mockgen has a version that was generating
spurious diffs in our generate test phase, and goveralls recently added
some code that calls git branch --format=..., which breaks on the version of
git in our Docker image.

Pinning versions required forcing go get into module-aware mode, since the
old-style go get doesn't understand versions.
2020-04-06 09:18:00 -07:00
Daniel McCarney 5573760723 CI: Add codespell to detect common typos. (#4637)
The `codespell` tool will be run during the "lints" phase of `test.sh`.
See `.codespell.ignore.txt for ignored words. Note that these ignored
words should be listed one per-line, in **lowercase** form.

The boulder-tools `build.sh` script is updated to include `codespell` in
the tools image. I built and pushed new images with this script that are
ref'd by `docker-compose.yml`.

Resolves #4635
2020-01-07 17:09:51 -08:00
Jacob Hoffman-Andrews 1146eecac3 integration: use python3 (#4582)
Python 2 is over in 1 month 4 days: https://pythonclock.org/

This rolls forward most of the changes in #4313.

The original change was rolled back in #4323 because it
broke `docker-compose up`. This change fixes those original issues by
(a) making sure `requests` is installed and (b) sourcing a virtualenv
containing the `requests` module before running start.py.

Other notable changes in this:
 - Certbot has changed the developer instructions to install specific packages
rather than rely on `letsencrypt-auto --os-packages-only`, so we follow suit.
 - Python3 now has a `bytes` type that is used in some places that used to
provide `str`, and all `str` are now Unicode. That means going from `bytes` to
`str` and back requires explicit `.decode()` and `.encode()`.
 - Moved from urllib2 to requests in many places.
2019-11-28 09:54:58 -05:00
Jacob Hoffman-Andrews 211985eae7 integration: Fail tests on unindexed queries. (#4581)
Also incidentally remove the database flag from the mysql invocation
since it was unnecessary.

Fixes #4406
2019-11-27 13:19:36 -05:00
Jacob Hoffman-Andrews 36e504f21f integration: log slow queries and check for them (#4554)
I couldn't get this to work cleanly with
`--log-queries-not-using-indexes` because a couple of queries show up
during integration test runs, seemingly because the tables involved are
small enough that the optimizer finds it faster to skip the index.

Some possible followups:
 - Allow list those queries, or
 - Preload the DB with a certain number of certificates before the start
   of testing.
2019-11-19 13:44:24 -05:00
Jacob Hoffman-Andrews 5ff750076c Move to Go 1.13.2. (#4490) 2019-10-17 15:39:04 -04:00
Daniel McCarney 2926074a29
CI/Dev: enable TLS 1.3 (#4489)
Also update the VA's TLS-ALPN-01 TLS 1.3 unit test to not expect
a failure.
2019-10-17 14:01:38 -04:00
Roland Bracewell Shoemaker 215818a13a Remove godeps cruft (#4467)
Boulder is fully transitioned to using 1st party Go modules/dependency management.
2019-10-07 14:55:14 -04:00
Daniel McCarney 93902965e5 Add Go 1.13 support, temporarily disable TLS 1.3 default. (#4435)
A unit test is included to verify that a TLS-ALPN-01 challenge to
a TLS 1.3 only server doesn't succeed when the `GODEBUG` value to
disable TLS 1.3 in `docker-compose.yml` is set. Without this env var
the test fails on the Go 1.13 build because of the new default:

```
=== RUN   TestTLSALPN01TLS13
--- FAIL: TestTLSALPN01TLS13 (0.04s)
    tlsalpn_test.go:531: expected problem validating TLS-ALPN-01 challenge against a TLS 1.3 only server, got nil
    FAIL
    FAIL        github.com/letsencrypt/boulder/va       0.065s
```

With the env var set the test passes, getting the expected connection
problem reporting a tls error:

```
=== RUN   TestTLSALPN01TLS13
2019/09/13 18:59:00 http: TLS handshake error from 127.0.0.1:51240: tls: client offered only unsupported versions: [303 302 301]
--- PASS: TestTLSALPN01TLS13 (0.03s)
PASS
ok      github.com/letsencrypt/boulder/va       1.054s
```

Since we plan to eventually enable TLS 1.3 support and the `GODEBUG`
mechanism tested in the above test is platform-wide vs package
specific I decided it wasn't worth the time investment to write a
similar HTTP-01 unit test that verifies the TLS 1.3 behaviour on a
HTTP-01 HTTP->HTTPS redirect.

Resolves https://github.com/letsencrypt/boulder/issues/4415
2019-09-17 11:00:58 -07:00
Daniel McCarney 4a6e34fc4e
va: clean up DNS error handling for HTTP-01 challenges. (#4409)
This PR changes the VA to return `dns` problem type for errors when performing
HTTP-01 challenges for domains that have no IP addresses, or errors looking up
the IP addresses.

The `va.getAddrs` function is internal to the VA and can return
`berrors.BoulderError`s with a DNS type when there is an error, allowing the
calling code to convert this to a problem when required
using an updated `detailedError` function. This avoids some clunky conversion
the HTTP-01 code was doing that misrepresented DNS level errors as connection
problems with a DNS detail message.

In order to add an integration test for challenge validation that results in
`getAddrs` DNS level errors the Boulder tools image had to be bumped to a tag
that includes the latest `pebble-challtestsrv` that
supports mocking SERVFAILs. It isn't possible to mock this case with internal IP
addresses because our VA test configuration does not filter internal addresses
to support the testing context.

Additionally this branch removes the `UnknownHostProblem` from the `probs`
package:

1. It isn't used anywhere after 532c210
2. It's not a real RFC 8555 problem type. We should/do use the
   DNS type for this.

Resolves https://github.com/letsencrypt/boulder/issues/4407
2019-08-28 15:47:35 -04:00
Roland Bracewell Shoemaker 04fb34c282 tools: add golang1.12.8 to travis (#4400) 2019-08-13 17:32:19 -04:00
Roland Bracewell Shoemaker 2e4531342d tests: add authorization deactivation integration tests (#4381)
Add pending and valid authorization deactivation integration tests
2019-07-31 17:47:52 -04:00
Jacob Hoffman-Andrews 3af49a16be
Revert "integration: move to Python3 (#4313)" (#4323)
This reverts commit 796a7aa2f4.

People's tests have been breaking on `docker-compose up` with the following output:

```
ImportError: No module named requests
```

Fixes #4322
2019-07-03 11:35:45 -07:00
Jacob Hoffman-Andrews 796a7aa2f4 integration: move to Python3 (#4313)
* integration: move to Python3

- Add parentheses to all print and raise calls.
- Python3 distinguishes bytes from strings. Add encode() and
  decode() calls as needed to provide the correct type.
- Use requests library consistently (urllib3 is not in Python3).
- Remove shebang from Python files without a main, and update
  shebang for integration-test.py.
2019-07-02 09:28:49 -04:00
Roland Bracewell Shoemaker af41bea99a Switch to more efficient multi nonce-service design (#4308)
Basically a complete re-write/re-design of the forwarding concept introduced in
#4297 (sorry for the rapid churn here). Instead of nonce-services blindly
forwarding nonces around to each other in an attempt to find out who issued the
nonce we add an identifying prefix to each nonce generated by a service. The
WFEs then use this prefix to decide which nonce-service to ask to validate the
nonce.

This requires a slightly more complicated configuration at the WFE/2 end, but
overall I think ends up being a way cleaner, more understandable, easy to
reason about implementation. When configuring the WFE you need to provide two
forms of gRPC config:

* one gRPC config for retrieving nonces, this should be a DNS name that
resolves to all available nonce-services (or at least the ones you want to
retrieve nonces from locally, in a two DC setup you might only configure the
nonce-services that are in the same DC as the WFE instance). This allows
getting a nonce from any of the configured services and is load-balanced
transparently at the gRPC layer. 
* a map of nonce prefixes to gRPC configs, this maps each individual
nonce-service to it's prefix and allows the WFE instances to figure out which
nonce-service to ask to validate a nonce it has received (in a two DC setup
you'd want to configure this with all the nonce-services across both DCs so
that you can validate a nonce that was generated by a nonce-service in another
DC).

This balancing is implemented in the integration tests.

Given the current remote nonce code hasn't been deployed anywhere yet this
makes a number of hard breaking changes to both the existing nonce-service
code, and the forwarding code.

Fixes #4303.
2019-06-28 12:58:46 -04:00
Jacob Hoffman-Andrews 1014a81606 Switch to modules (#4211)
Because the package versions in go.mod match what we use in Godeps.json,
there are no substantive code diffs. However, there are some tiny
differences resulting from how go mod vendors things differently than
godep:

go mod does not preserve executable permissions on shell scripts
Some packages have import lines like:
package ocsp // import "golang.org/x/crypto/ocsp"

godep used to remove the comment from these lines, but go mod vendor does not.

This introduces several indirect dependencies that we didn't have
before. This is because godep used to operate at a package level, but
go mod operates at a module (~= repository) level. So if we used a
given repository, but didn't use all of its packages, we wouldn't
previously care about the transitive dependencies of the packages we
weren't using. However, in the go mod world, once we care about the
repository, we care about all of that repository's transitive
dependencies. AFAICT this doesn't affect vendoring.

Fixes #4116
2019-05-23 14:14:00 -07:00
Jacob Hoffman-Andrews d2d5f0a328 Update miekg/dns and golang/protobuf. (#4150)
Precursor to #4116. Since some of our dependencies impose a minimum
version on these two packages higher than what we have in Godeps, we'll
have to bump them anyhow. Bumping them independently of the modules
update should keep things a little simpler.

In order to get protobuf tests to pass, I had to update protoc-gen-go in
boulder-tools. Now we download a prebuilt binary instead of using the
Ubuntu package, which is stuck on 3.0.0. This also meant I needed to
re-generate our pb.go files, since the new version generates somewhat
different output.

This happens to change the tag for pbutil, but it's not a substantive change - they just added a tagged version where there was none.

$ go test github.com/miekg/dns/...
ok      github.com/miekg/dns    4.675s
ok      github.com/miekg/dns/dnsutil    0.003s

ok      github.com/golang/protobuf/descriptor   (cached)
ok      github.com/golang/protobuf/jsonpb       (cached)
?       github.com/golang/protobuf/jsonpb/jsonpb_test_proto     [no test files]
ok      github.com/golang/protobuf/proto        (cached)
?       github.com/golang/protobuf/proto/proto3_proto   [no test files]
?       github.com/golang/protobuf/proto/test_proto     [no test files]
ok      github.com/golang/protobuf/protoc-gen-go        (cached)
?       github.com/golang/protobuf/protoc-gen-go/descriptor     [no test files]
ok      github.com/golang/protobuf/protoc-gen-go/generator      (cached)
ok      github.com/golang/protobuf/protoc-gen-go/generator/internal/remap       (cached)
?       github.com/golang/protobuf/protoc-gen-go/grpc   [no test files]
?       github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok      github.com/golang/protobuf/ptypes       (cached)
?       github.com/golang/protobuf/ptypes/any   [no test files]
?       github.com/golang/protobuf/ptypes/duration      [no test files]
?       github.com/golang/protobuf/ptypes/empty [no test files]
?       github.com/golang/protobuf/ptypes/struct        [no test files]
?       github.com/golang/protobuf/ptypes/timestamp     [no test files]
?       github.com/golang/protobuf/ptypes/wrappers      [no test files]
2019-04-09 09:27:28 -07:00
Jacob Hoffman-Andrews 2f6626afca Add Go 1.12 to Travis. (#4097)
* Add Go 1.12 to Travis.
* Update tag date.
* Fix go vet.
2019-03-06 16:45:11 -05:00
Roland Bracewell Shoemaker a6a00cb2ee Add ineffassign to CI tests (#4056) 2019-02-14 09:47:16 -05:00
Daniel McCarney 1c0be52e53 VA: Add integration test for HTTP timeouts. (#4050)
Also update `TestHTTPTimeout` to test with the `SimplifiedVAHTTP`
feature flag enabled.
2019-02-12 13:42:01 -08:00
Daniel McCarney 7369bf0354
CI/Dev: Remove boulder-tools Certbot version pin. (#4047)
The upstream Certbot project acme module supports initiating TLS-ALPN-01
challenges again and so we can remove the version pin we had in place.
This lets us keep the Certbot version we're testing with in-sync with
master at the time of building the tools image again.
2019-02-11 12:36:48 -05:00
Jacob Hoffman-Andrews 3cbc291af9
Update Go versions used in build. (#4029) 2019-01-23 16:23:27 -08:00
Daniel McCarney b29c762167 CI/Dev: Update Go 1.11.3 -> 1.11.4, update challtestsrv. (#3996)
1. Updates both boulder tools images to use an update `pebble-challtestsrv`
2. Updates the Go 1.11.3 boulder tools image to Go 1.11.4
3. Updates the vendored `challtestsrv` dep to 1.0.2

This fixes a panic in the `challtestsrv` library and prepares us to move directly 
to 1.11.4 after we've resolved the outstanding issues keeping us on the 1.10.x 
stream in prod/staging.

There are no unit tests to run for item 3.
2019-01-09 11:27:43 -08:00
Daniel McCarney 11433e1ea0
VA: Fix SimplifiedVAHTTP01 redirect query param handling. (#3988)
When the `SimplifiedVAHTTP01` feature flag is enabled we need to
preserve query parameters when reconstructing a redirect URL for the
resolved IP address.

To add integration testing for this condition the Boulder tools images
are updated to in turn pull in an updated `pebble-challtestsrv` command
that tracks request history.

A new Python wrapper for the `pebble-challtestsrv` HTTP API is added to
centralize interacting with the chall test srv to add mock data and to
get the history of HTTP requests that have been processed.
2019-01-04 14:20:44 -05:00
Jacob Hoffman-Andrews 340c1e4681
Bump Go versions in boulder-tools. (#3987) 2018-12-13 16:26:20 -08:00
Daniel McCarney f72c371bdc
Set pebble-challtestsrv IP from FAKE_DNS at startup. (#3984)
`pebble-challtestsrv` added a `-defaultIPv4` arg we can use to simplify
the integration tests and fix FAKE_DNS usage outside of integration
tests.

A new boulder-tools image with an updated `pebble-challtestsrv` is used
and `test/startservers.py` is changed to populate `-defaultIPv4` via the
`FAKE_DNS` env var.
2018-12-13 13:49:12 -05:00