Add Honeycomb tracing to all Boulder components which act as
HTTP servers, gRPC servers, or gRPC clients. Add many values
which we currently emit to logs to the trace spans. Add a way to
configure the Honeycomb integration to our config files, and by
default configure all of our tests to "mute" (send nothing).
Followup changes will refine the configuration, attempt to reduce
the new dependency load, and introduce better sampling.
Part of https://github.com/letsencrypt/dev-misc-tickets/issues/218
Update the pinned version of zlint from v2.2.1 to v3.1.0.
Also update the relevant path from v2 to v3 in both go.mod
and in individual imports. Update the vendored files to match.
No changes from v2.2.1 to v3.1.0 appear to affect the lints
we directly care about (e.g. those that we explicitly ignore).
Fixes#5206
* cmd: update prometheus.NewProcessCollector args.
There's a new struct `prometheus.ProcessCollectorOpts` that is expected
to be used as the sole argument to `prometheus.NewProcessCollector`. We
don't need to specify `os.Getpid` as the `PidFn` of the struct because
the default is to assume `os.Getpid`. Similarly we don't need to set the
namespace to `""` explicitly, it is the default.
* SA: reimplement db metrics as custom collector.
The modern Prometheus golang API supports translating between legacy
metric sources on the fly with a custom collector. We can use this
approach to collect the metrics from `gorp.DbMap`'s via the `sql.DB`
type's `Stats` function and the returned `sql.DbStats` struct.
This is a cleaner solution overall (we can lose the DB metrics updating
go routine) and it avoids the need to use the now-removed `Set` method
of the `prometheus.Counter` type.
* test: Update CountHistogramSamples.
The `With` function of `prometheus.HistogramVec` types we tend to use as
the argument to `test.CountHistogramSamples` changed to return
a `prometheus.Observer`. Since we only use this function in test
contexts, and only with things that cast back to
a `prometheus.Histogram` we take that approach to fix the problem
without updating call-sites.
$ go test -count=1 golang.org/x/net/{bpf,context,context/ctxhttp,http/httpguts,http2,http2/hpack,idna,internal/iana,internal/socket,internal/timeseries,ipv4,ipv6,trace} golang.org/x/sys/unix golang.org/x/crypto/...
ok golang.org/x/net/bpf 0.464s
ok golang.org/x/net/context 0.064s
ok golang.org/x/net/context/ctxhttp 0.109s
ok golang.org/x/net/http/httpguts 0.008s
ok golang.org/x/net/http2 83.376s
ok golang.org/x/net/http2/hpack 0.049s
ok golang.org/x/net/idna 0.003s
? golang.org/x/net/internal/iana [no test files]
ok golang.org/x/net/internal/socket 0.003s
ok golang.org/x/net/internal/timeseries 0.017s
ok golang.org/x/net/ipv4 0.022s
ok golang.org/x/net/ipv6 0.015s
ok golang.org/x/net/trace 0.010s
ok golang.org/x/sys/unix 0.576s
ok golang.org/x/crypto/acme 4.417s
ok golang.org/x/crypto/acme/autocert 0.222s
? golang.org/x/crypto/acme/autocert/internal/acmetest [no test files]
ok golang.org/x/crypto/argon2 0.072s
ok golang.org/x/crypto/bcrypt 2.331s
ok golang.org/x/crypto/blake2b 0.041s
ok golang.org/x/crypto/blake2s 0.068s
ok golang.org/x/crypto/blowfish 0.007s
ok golang.org/x/crypto/bn256 0.355s
ok golang.org/x/crypto/cast5 3.829s
ok golang.org/x/crypto/chacha20poly1305 0.047s
ok golang.org/x/crypto/cryptobyte 0.003s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
ok golang.org/x/crypto/curve25519 0.026s
ok golang.org/x/crypto/ed25519 0.121s
? golang.org/x/crypto/ed25519/internal/edwards25519 [no test files]
ok golang.org/x/crypto/hkdf 0.030s
ok golang.org/x/crypto/internal/chacha20 0.091s
ok golang.org/x/crypto/internal/subtle 0.013s
ok golang.org/x/crypto/md4 0.001s
ok golang.org/x/crypto/nacl/auth 1.805s
ok golang.org/x/crypto/nacl/box 0.017s
ok golang.org/x/crypto/nacl/secretbox 0.016s
ok golang.org/x/crypto/nacl/sign 0.022s
ok golang.org/x/crypto/ocsp 0.029s
ok golang.org/x/crypto/openpgp 7.507s
ok golang.org/x/crypto/openpgp/armor 0.022s
ok golang.org/x/crypto/openpgp/clearsign 21.458s
ok golang.org/x/crypto/openpgp/elgamal 0.009s
? golang.org/x/crypto/openpgp/errors [no test files]
ok golang.org/x/crypto/openpgp/packet 0.227s
ok golang.org/x/crypto/openpgp/s2k 8.758s
ok golang.org/x/crypto/otr 0.396s
ok golang.org/x/crypto/pbkdf2 0.060s
ok golang.org/x/crypto/pkcs12 0.069s
ok golang.org/x/crypto/pkcs12/internal/rc2 0.003s
ok golang.org/x/crypto/poly1305 0.012s
ok golang.org/x/crypto/ripemd160 0.043s
ok golang.org/x/crypto/salsa20 0.006s
ok golang.org/x/crypto/salsa20/salsa 0.002s
ok golang.org/x/crypto/scrypt 0.626s
ok golang.org/x/crypto/sha3 0.168s
ok golang.org/x/crypto/ssh 1.290s
ok golang.org/x/crypto/ssh/agent 0.597s
ok golang.org/x/crypto/ssh/knownhosts 0.004s
ok golang.org/x/crypto/ssh/terminal 0.008s
ok golang.org/x/crypto/ssh/test 0.081s
ok golang.org/x/crypto/tea 0.002s
ok golang.org/x/crypto/twofish 0.023s
ok golang.org/x/crypto/xtea 0.009s
ok golang.org/x/crypto/xts 0.001s
Now that Pebble has a `pebble-challtestsrv` we can remove the `challtestrv`
package and associated command from Boulder. I switched CI to use
`pebble-challtestsrv`. Notably this means that we have to add our expected mock
data using the HTTP management interface. The Boulder-tools images are
regenerated to include the `pebble-challtestsrv` command.
Using this approach also allows separating the TLS-ALPN-01 and HTTPS HTTP-01
challenges by binding each challenge type in the `pebble-challtestsrv` to
different interfaces both using the same VA
HTTPS port. Mock DNS directs the VA to the correct interface.
The load-generator command that was previously using the `challtestsrv` package
from Boulder is updated to use a vendored copy of the new
`github.org/letsencrypt/challtestsrv` package.
Vendored dependencies change in two ways:
1) Gomock is updated to the latest release (matching what the Bouldertools image
provides)
2) A couple of new subpackages in `golang.org/x/net/` are added by way of
transitive dependency through the challtestsrv package.
Unit tests are confirmed to pass for `gomock`:
```
~/go/src/github.com/golang/mock/gomock$ git log --pretty=format:'%h' -n 1
51421b9
~/go/src/github.com/golang/mock/gomock$ go test ./...
ok github.com/golang/mock/gomock 0.002s
? github.com/golang/mock/gomock/internal/mock_matcher [no test files]
```
For `/x/net` all tests pass except two `/x/net/icmp` `TestDiag.go` test cases
that we have agreed are OK to ignore.
Resolves https://github.com/letsencrypt/boulder/issues/3962 and
https://github.com/letsencrypt/boulder/issues/3951
Aaron Gable
Daniel McCarney
Jacob Hoffman-Andrews
Daniel McCarney