This updates to current `master`,
bc7acd89f703743d050f5cd4a3b9746808e0fdae
Notably, it includes a bug-fix to error handling in the HTTP client,
which we found was hiding errors from CT logs, hindering our debugging.
That fix is
https://github.com/google/certificate-transparency-go/pull/1695
No release has been tagged since this PR merged, so using the `master`
commit.
A few mutual dependencies used by both Boulder and ct-go are updated,
including mysql, otel, and grpc.
Update github.com/google/certificate-transparency-go from v1.1.6 to
v1.3.1. This updates the loglist file schema to recognize logs which are
tagged as being tiled logs / implementing the static CT API.
Transitively update:
- github.com/go-sql-driver/mysql from v1.7.1 to v1.8.1
- github.com/prometheus/client_golang from v1.15.1 to v1.22.0
- github.com/prometheus/client_model from v0.4.0 to v0.6.1
- go.opentelemetry.io/otel from v1.30.0 to v1.31.0
- google.golang.org/grpc from v1.66.1 to v1.69.4
- google.golang.org/protobuf from v1.34.2 to v1.36.5
- and a variety of indirect dependencies
Remove one indirect dependency:
- github.com/matttproud/golang_protobuf_extensions
Add two new indirect dependencies:
- filippo.io/edwards25519@v1.1.0 (used by go-sql-driver to handle
mariadb's custom encryption implementation)
- github.com/munnerz/goautoneg@v0.0.0-20191010083416-a7dc8b61c822
(previously inlined into prometheus/common)
Also fix two unit tests which need minor modifications to work with
updated type signatures and behavior.
Part of https://github.com/letsencrypt/boulder/issues/7872
The schema tool used to parse log_list_schema.json doesn't work well
with the updated schema. This is going to be required to support
static-ct-api logs from current Chrome log lists.
Instead, use the loglist3 package inside the certificate-transparency-go
project, which Boulder already uses for CT submission otherwise.
As well, the Log IDs and keys returned from loglist3 have already been
base64 decoded, so this re-encodes them to minimize the impact on the
rest of the codebase and keep this change small.
The test log_list.json file needed to be made a bit more realistic for
loglist3 to parse without base64 or date parsing errors.
A unit test is included to verify that a TLS-ALPN-01 challenge to
a TLS 1.3 only server doesn't succeed when the `GODEBUG` value to
disable TLS 1.3 in `docker-compose.yml` is set. Without this env var
the test fails on the Go 1.13 build because of the new default:
```
=== RUN TestTLSALPN01TLS13
--- FAIL: TestTLSALPN01TLS13 (0.04s)
tlsalpn_test.go:531: expected problem validating TLS-ALPN-01 challenge against a TLS 1.3 only server, got nil
FAIL
FAIL github.com/letsencrypt/boulder/va 0.065s
```
With the env var set the test passes, getting the expected connection
problem reporting a tls error:
```
=== RUN TestTLSALPN01TLS13
2019/09/13 18:59:00 http: TLS handshake error from 127.0.0.1:51240: tls: client offered only unsupported versions: [303 302 301]
--- PASS: TestTLSALPN01TLS13 (0.03s)
PASS
ok github.com/letsencrypt/boulder/va 1.054s
```
Since we plan to eventually enable TLS 1.3 support and the `GODEBUG`
mechanism tested in the above test is platform-wide vs package
specific I decided it wasn't worth the time investment to write a
similar HTTP-01 unit test that verifies the TLS 1.3 behaviour on a
HTTP-01 HTTP->HTTPS redirect.
Resolves https://github.com/letsencrypt/boulder/issues/4415
This updates Boulder's vendored dependency for `github.com/google/certificate-transparency-go` to c25855a, the tip of master at the time of writing.
Unit tests are confirmed to pass:
```
$ git log --pretty=format:'%h' -n 1
c25855a
$ go test ./...
ok github.com/google/certificate-transparency-go (cached)
ok github.com/google/certificate-transparency-go/asn1 (cached)
ok github.com/google/certificate-transparency-go/client 22.985s
? github.com/google/certificate-transparency-go/client/configpb [no test files]
? github.com/google/certificate-transparency-go/client/ctclient [no test files]
ok github.com/google/certificate-transparency-go/ctpolicy (cached)
ok github.com/google/certificate-transparency-go/ctutil (cached)
? github.com/google/certificate-transparency-go/ctutil/sctcheck [no test files]
? github.com/google/certificate-transparency-go/ctutil/sctscan [no test files]
ok github.com/google/certificate-transparency-go/dnsclient (cached)
ok github.com/google/certificate-transparency-go/fixchain 0.091s
? github.com/google/certificate-transparency-go/fixchain/chainfix [no test files]
ok github.com/google/certificate-transparency-go/fixchain/ratelimiter (cached)
ok github.com/google/certificate-transparency-go/gossip (cached)
? github.com/google/certificate-transparency-go/gossip/gossip_server [no test files]
ok github.com/google/certificate-transparency-go/gossip/minimal 0.028s
? github.com/google/certificate-transparency-go/gossip/minimal/configpb [no test files]
? github.com/google/certificate-transparency-go/gossip/minimal/goshawk [no test files]
? github.com/google/certificate-transparency-go/gossip/minimal/gosmin [no test files]
ok github.com/google/certificate-transparency-go/gossip/minimal/x509ext (cached)
ok github.com/google/certificate-transparency-go/ingestor/ranges (cached)
ok github.com/google/certificate-transparency-go/jsonclient 0.007s
ok github.com/google/certificate-transparency-go/logid (cached)
ok github.com/google/certificate-transparency-go/loglist (cached)
? github.com/google/certificate-transparency-go/loglist/findlog [no test files]
ok github.com/google/certificate-transparency-go/loglist2 (cached)
? github.com/google/certificate-transparency-go/preload [no test files]
? github.com/google/certificate-transparency-go/preload/dumpscts [no test files]
? github.com/google/certificate-transparency-go/preload/preloader [no test files]
ok github.com/google/certificate-transparency-go/scanner 0.009s
? github.com/google/certificate-transparency-go/scanner/scanlog [no test files]
ok github.com/google/certificate-transparency-go/tls (cached)
ok github.com/google/certificate-transparency-go/trillian/ctfe (cached)
? github.com/google/certificate-transparency-go/trillian/ctfe/configpb [no test files]
? github.com/google/certificate-transparency-go/trillian/ctfe/ct_server [no test files]
? github.com/google/certificate-transparency-go/trillian/ctfe/testonly [no test files]
ok github.com/google/certificate-transparency-go/trillian/integration 0.023s
? github.com/google/certificate-transparency-go/trillian/integration/ct_hammer [no test files]
? github.com/google/certificate-transparency-go/trillian/migrillian [no test files]
? github.com/google/certificate-transparency-go/trillian/migrillian/configpb [no test files]
ok github.com/google/certificate-transparency-go/trillian/migrillian/core (cached)
? github.com/google/certificate-transparency-go/trillian/mockclient [no test files]
ok github.com/google/certificate-transparency-go/trillian/util (cached)
ok github.com/google/certificate-transparency-go/x509 (cached)
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
? github.com/google/certificate-transparency-go/x509util [no test files]
? github.com/google/certificate-transparency-go/x509util/certcheck [no test files]
? github.com/google/certificate-transparency-go/x509util/crlcheck [no test files]
```
* Re-vendor certificate-transparency-go to latest.
$ go test github.com/google/certificate-transparency-go{,/asn1,/client,/client/configpb,/jsonclient,/tls,/x509/pkix} golang.org/x/crypto/cryptobyte{,/asn1}
ok github.com/google/certificate-transparency-go 0.722s
ok github.com/google/certificate-transparency-go/asn1 0.011s
ok github.com/google/certificate-transparency-go/client 22.995s
? github.com/google/certificate-transparency-go/client/configpb [no test files]
ok github.com/google/certificate-transparency-go/jsonclient 0.020s
ok github.com/google/certificate-transparency-go/tls 0.096s
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
ok golang.org/x/crypto/cryptobyte 0.013s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
* Bring in latest ct-go master.
This pulls in multilog support (logs sharded by date). As a result,
it also pulls in new dependencies gogo/protobuf (for UnmarshalText) and
golang/protobuf/ptypes (for Timestamp).
Replaces #3202, adding a smaller set of dependencies. See also #3205.
Tests run:
```
$ go test github.com/gogo/protobuf/proto github.com/golang/protobuf/ptypes/... github.com/google/certificate-transparency-go/...
ok github.com/gogo/protobuf/proto 0.063s
ok github.com/golang/protobuf/ptypes 0.009s
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
ok github.com/google/certificate-transparency-go 1.005s
ok github.com/google/certificate-transparency-go/asn1 0.021s
ok github.com/google/certificate-transparency-go/client 22.034s
? github.com/google/certificate-transparency-go/client/ctclient [no test files]
ok github.com/google/certificate-transparency-go/fixchain 0.145s
? github.com/google/certificate-transparency-go/fixchain/main [no test files]
ok github.com/google/certificate-transparency-go/fixchain/ratelimiter 27.745s
ok github.com/google/certificate-transparency-go/gossip 0.772s
? github.com/google/certificate-transparency-go/gossip/main [no test files]
ok github.com/google/certificate-transparency-go/jsonclient 25.523s
ok github.com/google/certificate-transparency-go/merkletree 0.004s
? github.com/google/certificate-transparency-go/preload [no test files]
? github.com/google/certificate-transparency-go/preload/dumpscts/main [no test files]
? github.com/google/certificate-transparency-go/preload/main [no test files]
ok github.com/google/certificate-transparency-go/scanner 0.010s
? github.com/google/certificate-transparency-go/scanner/main [no test files]
ok github.com/google/certificate-transparency-go/tls 0.026s
ok github.com/google/certificate-transparency-go/x509 0.417s
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
? github.com/google/certificate-transparency-go/x509util [no test files]
```
Switches imports from `github.com/google/certificate-transparency` to `github.com/google/certificate-transparency-go` and vendors the new code. Also fixes a number of small breakages caused by API changes since the last time we vendored the code. Also updates `github.com/cloudflare/cfssl` since you can't vendor both `github.com/google/certificate-transparency` and `github.com/google/certificate-transparency-go`.
Side note: while doing this `godep` tried to pull in a number of imports under the `golang.org/x/text` repo that I couldn't find actually being used anywhere so I just dropped the changes to `Godeps/Godeps.json` and didn't add the vendored dir to the tree, let's see if this breaks any tests...
All tests pass
```
$ go test ./...
ok github.com/google/certificate-transparency-go 0.640s
ok github.com/google/certificate-transparency-go/asn1 0.005s
ok github.com/google/certificate-transparency-go/client 22.054s
? github.com/google/certificate-transparency-go/client/ctclient [no test files]
ok github.com/google/certificate-transparency-go/fixchain 0.133s
? github.com/google/certificate-transparency-go/fixchain/main [no test files]
ok github.com/google/certificate-transparency-go/fixchain/ratelimiter 27.752s
ok github.com/google/certificate-transparency-go/gossip 0.322s
? github.com/google/certificate-transparency-go/gossip/main [no test files]
ok github.com/google/certificate-transparency-go/jsonclient 25.701s
ok github.com/google/certificate-transparency-go/merkletree 0.006s
? github.com/google/certificate-transparency-go/preload [no test files]
? github.com/google/certificate-transparency-go/preload/dumpscts/main [no test files]
? github.com/google/certificate-transparency-go/preload/main [no test files]
ok github.com/google/certificate-transparency-go/scanner 0.013s
? github.com/google/certificate-transparency-go/scanner/main [no test files]
ok github.com/google/certificate-transparency-go/tls 0.033s
ok github.com/google/certificate-transparency-go/x509 1.071s
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
? github.com/google/certificate-transparency-go/x509util [no test files]
```
```
$ ./test.sh
...
ok github.com/cloudflare/cfssl/api 1.089s coverage: 81.1% of statements
ok github.com/cloudflare/cfssl/api/bundle 1.548s coverage: 87.2% of statements
ok github.com/cloudflare/cfssl/api/certadd 13.681s coverage: 86.8% of statements
ok github.com/cloudflare/cfssl/api/client 1.314s coverage: 55.2% of statements
ok github.com/cloudflare/cfssl/api/crl 1.124s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/gencrl 1.067s coverage: 72.5% of statements
ok github.com/cloudflare/cfssl/api/generator 2.809s coverage: 33.3% of statements
ok github.com/cloudflare/cfssl/api/info 1.112s coverage: 84.1% of statements
ok github.com/cloudflare/cfssl/api/initca 1.059s coverage: 90.5% of statements
ok github.com/cloudflare/cfssl/api/ocsp 1.178s coverage: 93.8% of statements
ok github.com/cloudflare/cfssl/api/revoke 2.282s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/scan 2.729s coverage: 62.1% of statements
ok github.com/cloudflare/cfssl/api/sign 2.483s coverage: 83.3% of statements
ok github.com/cloudflare/cfssl/api/signhandler 1.137s coverage: 26.3% of statements
ok github.com/cloudflare/cfssl/auth 1.030s coverage: 68.2% of statements
ok github.com/cloudflare/cfssl/bundler 15.014s coverage: 85.1% of statements
ok github.com/cloudflare/cfssl/certdb/dbconf 1.042s coverage: 78.9% of statements
ok github.com/cloudflare/cfssl/certdb/ocspstapling 1.919s coverage: 69.2% of statements
ok github.com/cloudflare/cfssl/certdb/sql 1.265s coverage: 65.7% of statements
ok github.com/cloudflare/cfssl/cli 1.050s coverage: 61.9% of statements
ok github.com/cloudflare/cfssl/cli/bundle 1.023s coverage: 0.0% of statements
ok github.com/cloudflare/cfssl/cli/crl 1.669s coverage: 57.8% of statements
ok github.com/cloudflare/cfssl/cli/gencert 9.278s coverage: 83.6% of statements
ok github.com/cloudflare/cfssl/cli/gencrl 1.310s coverage: 73.3% of statements
ok github.com/cloudflare/cfssl/cli/genkey 3.028s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/cli/ocsprefresh 1.106s coverage: 64.3% of statements
ok github.com/cloudflare/cfssl/cli/revoke 1.081s coverage: 88.2% of statements
ok github.com/cloudflare/cfssl/cli/scan 1.217s coverage: 36.0% of statements
ok github.com/cloudflare/cfssl/cli/selfsign 2.201s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/cli/serve 1.133s coverage: 39.0% of statements
ok github.com/cloudflare/cfssl/cli/sign 1.210s coverage: 54.8% of statements
ok github.com/cloudflare/cfssl/cli/version 2.475s coverage: 100.0% of statements
ok github.com/cloudflare/cfssl/cmd/cfssl 1.082s coverage: 0.0% of statements
ok github.com/cloudflare/cfssl/cmd/cfssljson 1.016s coverage: 4.0% of statements
ok github.com/cloudflare/cfssl/cmd/mkbundle 1.024s coverage: 0.0% of statements
ok github.com/cloudflare/cfssl/config 2.754s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/crl 1.063s coverage: 68.3% of statements
ok github.com/cloudflare/cfssl/csr 27.016s coverage: 89.6% of statements
ok github.com/cloudflare/cfssl/errors 1.081s coverage: 81.2% of statements
ok github.com/cloudflare/cfssl/helpers 1.217s coverage: 80.4% of statements
ok github.com/cloudflare/cfssl/helpers/testsuite 7.658s coverage: 65.8% of statements
ok github.com/cloudflare/cfssl/initca 205.809s coverage: 74.2% of statements
ok github.com/cloudflare/cfssl/log 1.016s coverage: 59.3% of statements
ok github.com/cloudflare/cfssl/multiroot/config 1.107s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/ocsp 1.524s coverage: 77.7% of statements
ok github.com/cloudflare/cfssl/revoke 1.775s coverage: 79.6% of statements
ok github.com/cloudflare/cfssl/scan 1.022s coverage: 1.1% of statements
ok github.com/cloudflare/cfssl/selfsign 1.119s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer 1.019s coverage: 20.0% of statements
ok github.com/cloudflare/cfssl/signer/local 3.146s coverage: 81.2% of statements
ok github.com/cloudflare/cfssl/signer/remote 2.328s coverage: 71.8% of statements
ok github.com/cloudflare/cfssl/signer/universal 2.280s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/transport 1.028s
ok github.com/cloudflare/cfssl/transport/ca/localca 1.056s coverage: 94.9% of statements
ok github.com/cloudflare/cfssl/transport/core 1.538s coverage: 90.9% of statements
ok github.com/cloudflare/cfssl/transport/kp 1.054s coverage: 37.1% of statements
ok github.com/cloudflare/cfssl/ubiquity 1.042s coverage: 88.3% of statements
ok github.com/cloudflare/cfssl/whitelist 2.304s coverage: 100.0% of statements
```
Fixes#2746.
Jacob Hoffman-Andrews
Matthew McPherrin
Aaron Gable
Matthew McPherrin
dependabot[bot]
Daniel McCarney
Daniel McCarney
Jacob Hoffman-Andrews
Roland Bracewell Shoemaker