Update github.com/google/certificate-transparency-go from v1.1.6 to
v1.3.1. This updates the loglist file schema to recognize logs which are
tagged as being tiled logs / implementing the static CT API.
Transitively update:
- github.com/go-sql-driver/mysql from v1.7.1 to v1.8.1
- github.com/prometheus/client_golang from v1.15.1 to v1.22.0
- github.com/prometheus/client_model from v0.4.0 to v0.6.1
- go.opentelemetry.io/otel from v1.30.0 to v1.31.0
- google.golang.org/grpc from v1.66.1 to v1.69.4
- google.golang.org/protobuf from v1.34.2 to v1.36.5
- and a variety of indirect dependencies
Remove one indirect dependency:
- github.com/matttproud/golang_protobuf_extensions
Add two new indirect dependencies:
- filippo.io/edwards25519@v1.1.0 (used by go-sql-driver to handle
mariadb's custom encryption implementation)
- github.com/munnerz/goautoneg@v0.0.0-20191010083416-a7dc8b61c822
(previously inlined into prometheus/common)
Also fix two unit tests which need minor modifications to work with
updated type signatures and behavior.
Part of https://github.com/letsencrypt/boulder/issues/7872
This upgrades otel to v1.15.0, and the /contrib/ packages to v0.41.0.
Several dependencies are upgraded as dependencies, notably grpc.
This contains a change to grpc, only mapping some grpc.Errors into span
errors if it's Unknown, DeadlineExceeded, Unimplemented, Internal,
Unavailable, or DataLoss, which should be helpful for us as we use grpc
errors semantically in Boulder, especially NotFound.
Followup from #5839.
I chose groupcache/lru as our LRU cache implementation because it's part
of the golang org, written by one of the Go authors, and very simple
and easy to read.
This adds an `AccountGetter` interface that is implemented by both the
AccountCache and the SA. If the WFE config includes an AccountCache field,
it will wrap the SA in an AccountCache with the configured max size and
expiration time.
We set an expiration time on account cache entries because we want a
bounded amount of time that they may be stale by. This will be used in
conjunction with a delay on account-updating pathways to ensure we don't
allow authentication with a deactivated account or changed key.
The account cache stores corepb.Registration objects because protobufs
have an established way to do a deep copy. Deep copies are important so
the cache can maintain its own internal state and ensure nothing external
is modifying it.
As part of this process I changed construction of the WFE. Previously,
"SA" and "RA" were public fields that were mutated after construction. Now
they are parameters to the constructor, along with the new "accountGetter"
parameter.
The cache includes stats for requests categorized by hits and misses.
Remove mock-vendor, which ensured that mockgen was
available, because we no longer use mockgen. As a result,
remove mockgen from our docker build script. Finally, make
the mock package an indirect dependency since we are no
longer using it directly.
Delete the PublisherClientWrapper and PublisherServerWrapper. Update
various structs and functions to expect a pubpb.PublisherClient instead
of a core.Publisher; these two interfaces differ only in that the
auto-generated PublisherClient takes a variadic CallOptions parameter.
Update all mock publishers in tests to match the new interface. Finally,
delete the now-unused core.Publisher interface and some already-unused
mock-generating code.
This deletes a single sanity check (for a nil SCT even when there is a
nil error), but that check was redundant with an identical check in the
only extant client code in ctpolicy.go.
Fixes#5323
There are some changes to the code generated in the latest version, so
this modifies every .pb.go file.
Also, the way protoc-gen-go decides where to put files has changed, so
each generate.go gets the --go_opt=paths=source_relative flag to
tell protoc to continue placing output next to the input.
Remove staticcheck from build.sh; we get it via golangci-lint now.
Pass --no-document to gem install fpm; this is recommended in the fpm docs.
Unit tests confirmed to pass:
```
~/go/src/github.com/golang/mock$ git log --pretty=format:'%h' -n 1
d74b935
~/go/src/github.com/golang/mock$ go test ./...
go: downloading golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: extracting golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: finding golang.org/x/tools v0.0.0-20190425150028-36563e24a262
ok github.com/golang/mock/gomock 0.003s
? github.com/golang/mock/gomock/internal/mock_gomock [no test files]
ok github.com/golang/mock/mockgen 0.008s
ok github.com/golang/mock/mockgen/internal/tests/aux_imports_embedded_interface 0.002s
? github.com/golang/mock/mockgen/internal/tests/aux_imports_embedded_interface/faux [no test files]
? github.com/golang/mock/mockgen/internal/tests/copyright_file [no test files]
? github.com/golang/mock/mockgen/internal/tests/custom_package_name/client/v1 [no test files]
ok github.com/golang/mock/mockgen/internal/tests/custom_package_name/greeter 0.003s
? github.com/golang/mock/mockgen/internal/tests/custom_package_name/validator [no test files]
? github.com/golang/mock/mockgen/internal/tests/dot_imports [no test files]
? github.com/golang/mock/mockgen/internal/tests/empty_interface [no test files]
ok github.com/golang/mock/mockgen/internal/tests/generated_identifier_conflict 0.006s
? github.com/golang/mock/mockgen/internal/tests/import_source [no test files]
? github.com/golang/mock/mockgen/internal/tests/import_source/definition [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg/reflect_output [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg/source_output [no test files]
ok github.com/golang/mock/mockgen/internal/tests/mock_in_test_package 0.045s [no tests to run]
ok github.com/golang/mock/mockgen/internal/tests/test_package 0.002s [no tests to run]
ok github.com/golang/mock/mockgen/internal/tests/unexported_method 0.002s
? github.com/golang/mock/mockgen/internal/tests/vendor_dep [no test files]
? github.com/golang/mock/mockgen/internal/tests/vendor_dep/source_mock_package [no test files]
? github.com/golang/mock/mockgen/internal/tests/vendor_pkg [no test files]
ok github.com/golang/mock/mockgen/model 0.007s
ok github.com/golang/mock/sample 0.003s
ok github.com/golang/mock/sample/concurrent 0.002s
? github.com/golang/mock/sample/concurrent/mock [no test files]
? github.com/golang/mock/sample/imp1 [no test files]
? github.com/golang/mock/sample/imp2 [no test files]
? github.com/golang/mock/sample/imp3 [no test files]
? github.com/golang/mock/sample/imp4 [no test files]
? github.com/golang/mock/sample/mock_user [no test files]
```
* cmd: update prometheus.NewProcessCollector args.
There's a new struct `prometheus.ProcessCollectorOpts` that is expected
to be used as the sole argument to `prometheus.NewProcessCollector`. We
don't need to specify `os.Getpid` as the `PidFn` of the struct because
the default is to assume `os.Getpid`. Similarly we don't need to set the
namespace to `""` explicitly, it is the default.
* SA: reimplement db metrics as custom collector.
The modern Prometheus golang API supports translating between legacy
metric sources on the fly with a custom collector. We can use this
approach to collect the metrics from `gorp.DbMap`'s via the `sql.DB`
type's `Stats` function and the returned `sql.DbStats` struct.
This is a cleaner solution overall (we can lose the DB metrics updating
go routine) and it avoids the need to use the now-removed `Set` method
of the `prometheus.Counter` type.
* test: Update CountHistogramSamples.
The `With` function of `prometheus.HistogramVec` types we tend to use as
the argument to `test.CountHistogramSamples` changed to return
a `prometheus.Observer`. Since we only use this function in test
contexts, and only with things that cast back to
a `prometheus.Histogram` we take that approach to fix the problem
without updating call-sites.
Because the package versions in go.mod match what we use in Godeps.json,
there are no substantive code diffs. However, there are some tiny
differences resulting from how go mod vendors things differently than
godep:
go mod does not preserve executable permissions on shell scripts
Some packages have import lines like:
package ocsp // import "golang.org/x/crypto/ocsp"
godep used to remove the comment from these lines, but go mod vendor does not.
This introduces several indirect dependencies that we didn't have
before. This is because godep used to operate at a package level, but
go mod operates at a module (~= repository) level. So if we used a
given repository, but didn't use all of its packages, we wouldn't
previously care about the transitive dependencies of the packages we
weren't using. However, in the go mod world, once we care about the
repository, we care about all of that repository's transitive
dependencies. AFAICT this doesn't affect vendoring.
Fixes#4116
Precursor to #4116. Since some of our dependencies impose a minimum
version on these two packages higher than what we have in Godeps, we'll
have to bump them anyhow. Bumping them independently of the modules
update should keep things a little simpler.
In order to get protobuf tests to pass, I had to update protoc-gen-go in
boulder-tools. Now we download a prebuilt binary instead of using the
Ubuntu package, which is stuck on 3.0.0. This also meant I needed to
re-generate our pb.go files, since the new version generates somewhat
different output.
This happens to change the tag for pbutil, but it's not a substantive change - they just added a tagged version where there was none.
$ go test github.com/miekg/dns/...
ok github.com/miekg/dns 4.675s
ok github.com/miekg/dns/dnsutil 0.003s
ok github.com/golang/protobuf/descriptor (cached)
ok github.com/golang/protobuf/jsonpb (cached)
? github.com/golang/protobuf/jsonpb/jsonpb_test_proto [no test files]
ok github.com/golang/protobuf/proto (cached)
? github.com/golang/protobuf/proto/proto3_proto [no test files]
? github.com/golang/protobuf/proto/test_proto [no test files]
ok github.com/golang/protobuf/protoc-gen-go (cached)
? github.com/golang/protobuf/protoc-gen-go/descriptor [no test files]
ok github.com/golang/protobuf/protoc-gen-go/generator (cached)
ok github.com/golang/protobuf/protoc-gen-go/generator/internal/remap (cached)
? github.com/golang/protobuf/protoc-gen-go/grpc [no test files]
? github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok github.com/golang/protobuf/ptypes (cached)
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
Now that Pebble has a `pebble-challtestsrv` we can remove the `challtestrv`
package and associated command from Boulder. I switched CI to use
`pebble-challtestsrv`. Notably this means that we have to add our expected mock
data using the HTTP management interface. The Boulder-tools images are
regenerated to include the `pebble-challtestsrv` command.
Using this approach also allows separating the TLS-ALPN-01 and HTTPS HTTP-01
challenges by binding each challenge type in the `pebble-challtestsrv` to
different interfaces both using the same VA
HTTPS port. Mock DNS directs the VA to the correct interface.
The load-generator command that was previously using the `challtestsrv` package
from Boulder is updated to use a vendored copy of the new
`github.org/letsencrypt/challtestsrv` package.
Vendored dependencies change in two ways:
1) Gomock is updated to the latest release (matching what the Bouldertools image
provides)
2) A couple of new subpackages in `golang.org/x/net/` are added by way of
transitive dependency through the challtestsrv package.
Unit tests are confirmed to pass for `gomock`:
```
~/go/src/github.com/golang/mock/gomock$ git log --pretty=format:'%h' -n 1
51421b9
~/go/src/github.com/golang/mock/gomock$ go test ./...
ok github.com/golang/mock/gomock 0.002s
? github.com/golang/mock/gomock/internal/mock_matcher [no test files]
```
For `/x/net` all tests pass except two `/x/net/icmp` `TestDiag.go` test cases
that we have agreed are OK to ignore.
Resolves https://github.com/letsencrypt/boulder/issues/3962 and
https://github.com/letsencrypt/boulder/issues/3951
Retains the existing logging of orphaned certs until we are confident that this
solution can fully replace it (even then we may want to keep it just for auditing etc).
Fixes#3636.
In #3454, I tried to update certificate-transparency-go, but that pulled in a bunch of extra package updates, making for a complicated PR. This PR breaks out one of the packages that needed update, to allow us to bring things up to date in a simpler, more piecemeal fashion.
$ go test github.com/golang/protobuf/...
ok github.com/golang/protobuf/descriptor 0.004s
ok github.com/golang/protobuf/jsonpb 0.012s
? github.com/golang/protobuf/jsonpb/jsonpb_test_proto [no test
files]
ok github.com/golang/protobuf/proto 0.062s
? github.com/golang/protobuf/proto/proto3_proto [no test files]
? github.com/golang/protobuf/protoc-gen-go [no test files]
? github.com/golang/protobuf/protoc-gen-go/descriptor [no test
files]
ok github.com/golang/protobuf/protoc-gen-go/generator 0.002s
? github.com/golang/protobuf/protoc-gen-go/grpc [no test files]
? github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok github.com/golang/protobuf/ptypes 0.014s
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
This change is pulled out of #3294 in hopes of simplifying that change.
Tests run:
```
$ go test github.com/golang/mock/gomock/...
ok github.com/golang/mock/gomock 0.002s
? github.com/golang/mock/gomock/mock_matcher [no test files]
```
This pulls in multilog support (logs sharded by date). As a result,
it also pulls in new dependencies gogo/protobuf (for UnmarshalText) and
golang/protobuf/ptypes (for Timestamp).
Replaces #3202, adding a smaller set of dependencies. See also #3205.
Tests run:
```
$ go test github.com/gogo/protobuf/proto github.com/golang/protobuf/ptypes/... github.com/google/certificate-transparency-go/...
ok github.com/gogo/protobuf/proto 0.063s
ok github.com/golang/protobuf/ptypes 0.009s
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
ok github.com/google/certificate-transparency-go 1.005s
ok github.com/google/certificate-transparency-go/asn1 0.021s
ok github.com/google/certificate-transparency-go/client 22.034s
? github.com/google/certificate-transparency-go/client/ctclient [no test files]
ok github.com/google/certificate-transparency-go/fixchain 0.145s
? github.com/google/certificate-transparency-go/fixchain/main [no test files]
ok github.com/google/certificate-transparency-go/fixchain/ratelimiter 27.745s
ok github.com/google/certificate-transparency-go/gossip 0.772s
? github.com/google/certificate-transparency-go/gossip/main [no test files]
ok github.com/google/certificate-transparency-go/jsonclient 25.523s
ok github.com/google/certificate-transparency-go/merkletree 0.004s
? github.com/google/certificate-transparency-go/preload [no test files]
? github.com/google/certificate-transparency-go/preload/dumpscts/main [no test files]
? github.com/google/certificate-transparency-go/preload/main [no test files]
ok github.com/google/certificate-transparency-go/scanner 0.010s
? github.com/google/certificate-transparency-go/scanner/main [no test files]
ok github.com/google/certificate-transparency-go/tls 0.026s
ok github.com/google/certificate-transparency-go/x509 0.417s
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
? github.com/google/certificate-transparency-go/x509util [no test files]
```
Fixes https://github.com/letsencrypt/boulder/issues/3205.
Previously, we would only move aside Godeps.json before running `godep save ./...`. However, in order to get a true picture of what is needed, we must also remove the existing `vendor/` directory.
This change also removes some unnecessary dependencies that have piled up over the years, generally test dependencies. Godep used to vendor such dependencies but no longer does.
Update github.com/google/safebrowsing and block on database health before starting VA
before starting `boulder-va`.
```
$ go test .
ok github.com/google/safebrowsing 4.510s
$ go test .
ok github.com/golang/protobuf/ptypes 0.002s
```
Fixes#2742.
When running `gomock` to generate mocks in the boulder-tools image there is a requirement on `github.com/golang/mock/mockgen/model` but only during runtime (it is not required to build `gomock`). So that we don't require users to `go get` this package so that it exists in their GOPATH we need to vendor it so that it is always in the GOPATH of the boulder-tools image. In order to vendor this package (since it isn't actually used anywhere) we need to add a special file that imports this package and uses it for a variable that isn't actually used anywhere so that we can satisfy `godep`, this is done in the `test` package.
Fixes#2751.
Updates the various gRPC/protobuf libs (google.golang.org/grpc/... and github.com/golang/protobuf/proto) and the boulder-tools image so that we can update to the newest github.com/grpc-ecosystem/go-grpc-prometheus. Also regenerates all of the protobuf definition files.
Tests run on updated packages all pass.
Unblocks #2633fixes#2636.
The `letsencrypt/boulder-tools` image was recently updated, pulling in version
0.8.0 of certbot. That version stores the output of `certonly` requests in a
different path. In test.sh, we check out a specific tagged release of certbot in
order to get its integration tests. Prior to this commit, we were using
certbot 0.8.0 with the integration tests from version 0.6.0 of certbot,
which looked for `certonly` output in the wrong place, and failed.
This commit changes test.sh to checkout the 0.8.0 branch, and also removes a
temporary shim we used to make the `certbot` command call out to the
`letsencrypt` command.
Also, since the latest version of `letsencrypt/boulder-tools` includes an updated
`protoc-gen-go`, this change also updates the support packages to match.
Aaron Gable
dependabot[bot]
Jacob Hoffman-Andrews
Matthew McPherrin
Jacob Hoffman-Andrews
milgradesec
Daniel McCarney
Daniel McCarney
Roland Bracewell Shoemaker