Fork the pieces of the Go standard library's crypto/x509
package which are relevant to parsing, handling, and
signing CRLs.
In our fork, fix an upstream parsing bug, hoist the reasonCode
out of the crlEntryExtensions for easier usability, and enforce
that CRL Numbers are never longer than 20 octets.
Part of #6199
Add Honeycomb tracing to all Boulder components which act as
HTTP servers, gRPC servers, or gRPC clients. Add many values
which we currently emit to logs to the trace spans. Add a way to
configure the Honeycomb integration to our config files, and by
default configure all of our tests to "mute" (send nothing).
Followup changes will refine the configuration, attempt to reduce
the new dependency load, and introduce better sampling.
Part of https://github.com/letsencrypt/dev-misc-tickets/issues/218
Update the pinned version of zlint from v2.2.1 to v3.1.0.
Also update the relevant path from v2 to v3 in both go.mod
and in individual imports. Update the vendored files to match.
No changes from v2.2.1 to v3.1.0 appear to affect the lints
we directly care about (e.g. those that we explicitly ignore).
Fixes#5206
Unit tests are confirmed to pass:
```
~/go/src/golang.org/x/net$ git log --pretty=format:'%h' -n 1
2180aed
~/go/src/golang.org/x/net$ go test ./...
ok golang.org/x/net/bpf 0.494s
ok golang.org/x/net/context 0.058s
ok golang.org/x/net/context/ctxhttp 0.104s
? golang.org/x/net/dict [no test files]
ok golang.org/x/net/dns/dnsmessage 0.074s
ok golang.org/x/net/html 0.097s
ok golang.org/x/net/html/atom 0.002s
ok golang.org/x/net/html/charset 0.020s
ok golang.org/x/net/http/httpguts 0.028s
ok golang.org/x/net/http/httpproxy 0.003s
ok golang.org/x/net/http2 125.352s
ok golang.org/x/net/http2/h2c 0.015s
? golang.org/x/net/http2/h2i [no test files]
ok golang.org/x/net/http2/hpack 0.042s
ok golang.org/x/net/icmp 0.002s
ok golang.org/x/net/idna 0.012s
? golang.org/x/net/internal/iana [no test files]
ok golang.org/x/net/internal/socket 4.560s
ok golang.org/x/net/internal/socks 0.222s
ok golang.org/x/net/internal/sockstest 0.015s
ok golang.org/x/net/internal/timeseries 0.020s
ok golang.org/x/net/ipv4 0.053s
ok golang.org/x/net/ipv6 0.043s
ok golang.org/x/net/nettest 1.057s
ok golang.org/x/net/netutil 0.819s
ok golang.org/x/net/proxy 0.039s
ok golang.org/x/net/publicsuffix 0.146s
ok golang.org/x/net/trace 0.007s
ok golang.org/x/net/webdav 0.091s
ok golang.org/x/net/webdav/internal/xml 0.010s
ok golang.org/x/net/websocket 0.026s
ok golang.org/x/net/xsrftoken 0.019s
```
* cmd: update prometheus.NewProcessCollector args.
There's a new struct `prometheus.ProcessCollectorOpts` that is expected
to be used as the sole argument to `prometheus.NewProcessCollector`. We
don't need to specify `os.Getpid` as the `PidFn` of the struct because
the default is to assume `os.Getpid`. Similarly we don't need to set the
namespace to `""` explicitly, it is the default.
* SA: reimplement db metrics as custom collector.
The modern Prometheus golang API supports translating between legacy
metric sources on the fly with a custom collector. We can use this
approach to collect the metrics from `gorp.DbMap`'s via the `sql.DB`
type's `Stats` function and the returned `sql.DbStats` struct.
This is a cleaner solution overall (we can lose the DB metrics updating
go routine) and it avoids the need to use the now-removed `Set` method
of the `prometheus.Counter` type.
* test: Update CountHistogramSamples.
The `With` function of `prometheus.HistogramVec` types we tend to use as
the argument to `test.CountHistogramSamples` changed to return
a `prometheus.Observer`. Since we only use this function in test
contexts, and only with things that cast back to
a `prometheus.Histogram` we take that approach to fix the problem
without updating call-sites.
A unit test is included to verify that a TLS-ALPN-01 challenge to
a TLS 1.3 only server doesn't succeed when the `GODEBUG` value to
disable TLS 1.3 in `docker-compose.yml` is set. Without this env var
the test fails on the Go 1.13 build because of the new default:
```
=== RUN TestTLSALPN01TLS13
--- FAIL: TestTLSALPN01TLS13 (0.04s)
tlsalpn_test.go:531: expected problem validating TLS-ALPN-01 challenge against a TLS 1.3 only server, got nil
FAIL
FAIL github.com/letsencrypt/boulder/va 0.065s
```
With the env var set the test passes, getting the expected connection
problem reporting a tls error:
```
=== RUN TestTLSALPN01TLS13
2019/09/13 18:59:00 http: TLS handshake error from 127.0.0.1:51240: tls: client offered only unsupported versions: [303 302 301]
--- PASS: TestTLSALPN01TLS13 (0.03s)
PASS
ok github.com/letsencrypt/boulder/va 1.054s
```
Since we plan to eventually enable TLS 1.3 support and the `GODEBUG`
mechanism tested in the above test is platform-wide vs package
specific I decided it wasn't worth the time investment to write a
similar HTTP-01 unit test that verifies the TLS 1.3 behaviour on a
HTTP-01 HTTP->HTTPS redirect.
Resolves https://github.com/letsencrypt/boulder/issues/4415
Because the package versions in go.mod match what we use in Godeps.json,
there are no substantive code diffs. However, there are some tiny
differences resulting from how go mod vendors things differently than
godep:
go mod does not preserve executable permissions on shell scripts
Some packages have import lines like:
package ocsp // import "golang.org/x/crypto/ocsp"
godep used to remove the comment from these lines, but go mod vendor does not.
This introduces several indirect dependencies that we didn't have
before. This is because godep used to operate at a package level, but
go mod operates at a module (~= repository) level. So if we used a
given repository, but didn't use all of its packages, we wouldn't
previously care about the transitive dependencies of the packages we
weren't using. However, in the go mod world, once we care about the
repository, we care about all of that repository's transitive
dependencies. AFAICT this doesn't affect vendoring.
Fixes#4116
$ go test -count=1 golang.org/x/net/{bpf,context,context/ctxhttp,http/httpguts,http2,http2/hpack,idna,internal/iana,internal/socket,internal/timeseries,ipv4,ipv6,trace} golang.org/x/sys/unix golang.org/x/crypto/...
ok golang.org/x/net/bpf 0.464s
ok golang.org/x/net/context 0.064s
ok golang.org/x/net/context/ctxhttp 0.109s
ok golang.org/x/net/http/httpguts 0.008s
ok golang.org/x/net/http2 83.376s
ok golang.org/x/net/http2/hpack 0.049s
ok golang.org/x/net/idna 0.003s
? golang.org/x/net/internal/iana [no test files]
ok golang.org/x/net/internal/socket 0.003s
ok golang.org/x/net/internal/timeseries 0.017s
ok golang.org/x/net/ipv4 0.022s
ok golang.org/x/net/ipv6 0.015s
ok golang.org/x/net/trace 0.010s
ok golang.org/x/sys/unix 0.576s
ok golang.org/x/crypto/acme 4.417s
ok golang.org/x/crypto/acme/autocert 0.222s
? golang.org/x/crypto/acme/autocert/internal/acmetest [no test files]
ok golang.org/x/crypto/argon2 0.072s
ok golang.org/x/crypto/bcrypt 2.331s
ok golang.org/x/crypto/blake2b 0.041s
ok golang.org/x/crypto/blake2s 0.068s
ok golang.org/x/crypto/blowfish 0.007s
ok golang.org/x/crypto/bn256 0.355s
ok golang.org/x/crypto/cast5 3.829s
ok golang.org/x/crypto/chacha20poly1305 0.047s
ok golang.org/x/crypto/cryptobyte 0.003s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
ok golang.org/x/crypto/curve25519 0.026s
ok golang.org/x/crypto/ed25519 0.121s
? golang.org/x/crypto/ed25519/internal/edwards25519 [no test files]
ok golang.org/x/crypto/hkdf 0.030s
ok golang.org/x/crypto/internal/chacha20 0.091s
ok golang.org/x/crypto/internal/subtle 0.013s
ok golang.org/x/crypto/md4 0.001s
ok golang.org/x/crypto/nacl/auth 1.805s
ok golang.org/x/crypto/nacl/box 0.017s
ok golang.org/x/crypto/nacl/secretbox 0.016s
ok golang.org/x/crypto/nacl/sign 0.022s
ok golang.org/x/crypto/ocsp 0.029s
ok golang.org/x/crypto/openpgp 7.507s
ok golang.org/x/crypto/openpgp/armor 0.022s
ok golang.org/x/crypto/openpgp/clearsign 21.458s
ok golang.org/x/crypto/openpgp/elgamal 0.009s
? golang.org/x/crypto/openpgp/errors [no test files]
ok golang.org/x/crypto/openpgp/packet 0.227s
ok golang.org/x/crypto/openpgp/s2k 8.758s
ok golang.org/x/crypto/otr 0.396s
ok golang.org/x/crypto/pbkdf2 0.060s
ok golang.org/x/crypto/pkcs12 0.069s
ok golang.org/x/crypto/pkcs12/internal/rc2 0.003s
ok golang.org/x/crypto/poly1305 0.012s
ok golang.org/x/crypto/ripemd160 0.043s
ok golang.org/x/crypto/salsa20 0.006s
ok golang.org/x/crypto/salsa20/salsa 0.002s
ok golang.org/x/crypto/scrypt 0.626s
ok golang.org/x/crypto/sha3 0.168s
ok golang.org/x/crypto/ssh 1.290s
ok golang.org/x/crypto/ssh/agent 0.597s
ok golang.org/x/crypto/ssh/knownhosts 0.004s
ok golang.org/x/crypto/ssh/terminal 0.008s
ok golang.org/x/crypto/ssh/test 0.081s
ok golang.org/x/crypto/tea 0.002s
ok golang.org/x/crypto/twofish 0.023s
ok golang.org/x/crypto/xtea 0.009s
ok golang.org/x/crypto/xts 0.001s
Now that Pebble has a `pebble-challtestsrv` we can remove the `challtestrv`
package and associated command from Boulder. I switched CI to use
`pebble-challtestsrv`. Notably this means that we have to add our expected mock
data using the HTTP management interface. The Boulder-tools images are
regenerated to include the `pebble-challtestsrv` command.
Using this approach also allows separating the TLS-ALPN-01 and HTTPS HTTP-01
challenges by binding each challenge type in the `pebble-challtestsrv` to
different interfaces both using the same VA
HTTPS port. Mock DNS directs the VA to the correct interface.
The load-generator command that was previously using the `challtestsrv` package
from Boulder is updated to use a vendored copy of the new
`github.org/letsencrypt/challtestsrv` package.
Vendored dependencies change in two ways:
1) Gomock is updated to the latest release (matching what the Bouldertools image
provides)
2) A couple of new subpackages in `golang.org/x/net/` are added by way of
transitive dependency through the challtestsrv package.
Unit tests are confirmed to pass for `gomock`:
```
~/go/src/github.com/golang/mock/gomock$ git log --pretty=format:'%h' -n 1
51421b9
~/go/src/github.com/golang/mock/gomock$ go test ./...
ok github.com/golang/mock/gomock 0.002s
? github.com/golang/mock/gomock/internal/mock_matcher [no test files]
```
For `/x/net` all tests pass except two `/x/net/icmp` `TestDiag.go` test cases
that we have agreed are OK to ignore.
Resolves https://github.com/letsencrypt/boulder/issues/3962 and
https://github.com/letsencrypt/boulder/issues/3951
Aaron Gable
dependabot[bot]
Jacob Hoffman-Andrews
Daniel McCarney
Daniel McCarney