Right now, Boulder expects to be able to connect to syslog, and panics
if it's not available. We'd like to be able to log to stdout/stderr as a
replacement for syslog.
- Add a detailed timestamp (down to microseconds, same as we collect in
prod via syslog).
- Remove the escape codes for colorizing output.
- Report the severity level numerically rather than with a letter prefix.
Add locking for stdout/stderr and syslog logs. Neither the [syslog] package
nor the [os] package document concurrency-safety, and the Go rule is: if
it's not documented to be concurrent-safe, it's not. Notably the [log.Logger]
package is documented to be concurrent-safe, and a look at its implementation
shows it uses a Mutex internally.
Remove places that use the singleton `blog.Get()`, and instead pass through
a logger from main in all the places that need it.
[syslog]: https://pkg.go.dev/log/syslog
[os]: https://pkg.go.dev/os
[log.Logger]: https://pkg.go.dev/log#Logger
Create a new crl-storer service, which receives CRL shards via gRPC and
uploads them to an S3 bucket. It ignores AWS SDK configuration in the
usual places, in favor of configuration from our standard JSON service
config files. It ensures that the CRLs it receives parse and are signed
by the appropriate issuer before uploading them.
Integrate crl-updater with the new service. It streams bytes to the
crl-storer as it receives them from the CA, without performing any
checking at the same time. This new functionality is disabled if the
crl-updater does not have a config stanza instructing it how to connect
to the crl-storer.
Finally, add a new test component, the s3-test-srv. This acts similarly
to the existing mail-test-srv: it receives requests, stores information
about them, and exposes that information for later querying by the
integration test. The integration test uses this to ensure that a
newly-revoked certificate does show up in the next generation of CRLs
produced.
Fixes#6162
Fork the pieces of the Go standard library's crypto/x509
package which are relevant to parsing, handling, and
signing CRLs.
In our fork, fix an upstream parsing bug, hoist the reasonCode
out of the crlEntryExtensions for easier usability, and enforce
that CRL Numbers are never longer than 20 octets.
Part of #6199
The gopkg.in/yaml.v2 package has a potential crash when
parsing malicious input. Although we only use the yaml
package to parse trusted configuration, update to v3 anyway.
Update the PSL from 7594db4f858a (Oct 2021) to 9a40b608a236
(March 2022). This adds approximately 165 new entries and removes
approximately 28 old entries.
Fixes#6022
Diff: https://github.com/prometheus/client_golang/compare/v1.7.1...v1.12.1
Changes:
* 1.12.1 / 2022-01-29
- [BUGFIX] Make the Go 1.17 collector concurrency-safe
- Use simpler locking in the Go 1.17 collector
- [BUGFIX] Reduce granularity of histogram buckets for Go 1.17 collector
- [ENHANCEMENT] API client: make HTTP reads more efficient
* 1.12.0 / 2022-01-19
- [CHANGE] example/random: Move flags and metrics into main()
- [FEATURE] API client: Support wal replay status api
- [FEATURE] Use the runtime/metrics package for the Go collector for 1.17+
- [ENHANCEMENT] API client: Update /api/v1/status/tsdb to include headStats
- [ENHANCEMENT] promhttp: Check validity of method and code label values
* 1.11.0 / 2021-06-07
- [CHANGE] Add new collectors package.
- [CHANGE] prometheus.NewExpvarCollector is deprecated, use collectors.NewExpvarCollector instead.
- [CHANGE] prometheus.NewGoCollector is deprecated, use collectors.NewGoCollector instead.
- [CHANGE] prometheus.NewBuildInfoCollector is deprecated, use collectors.NewBuildInfoCollector instead.
- [FEATURE] Add new collector for database/sql#DBStats.
- [FEATURE] API client: Add exemplars API support.
- [ENHANCEMENT] API client: Add newer fields to Rules API.
- [ENHANCEMENT] API client: Add missing fields to Targets API.
* 1.10.0 / 2021-03-18
- [CHANGE] Minimum required Go version is now 1.13.
- [CHANGE] API client: Add matchers to LabelNames and LabesValues.
- [FEATURE] API client: Add buildinfo call.
- [BUGFIX] Fix build on riscv64.
* 1.9.0 / 2020-12-17
- [FEATURE] NewPidFileFn helper to create process collectors for processes whose PID is read from a file.
- [BUGFIX] promhttp: Prevent endless loop in InstrumentHandler... middlewares with invalid metric or label names.
* 1.8.0 / 2020-10-15
- [CHANGE] API client: Use time.Time rather than string for timestamps in RuntimeinfoResult.
- [FEATURE] Export MetricVec to facilitate implementation of vectors of custom Metric types.
- [FEATURE] API client: Support /status/tsdb endpoint.
- [ENHANCEMENT] API client: Enable GET fallback on status code 501.
- [ENHANCEMENT] Remove Metric references after reslicing to free up more memory.
Additional transitive dependency updates:
* https://github.com/prometheus/common/compare/v0.10.0...v0.32.1
* https://github.com/prometheus/procfs/compare/v0.1.3...v0.7.3
* https://github.com/golang/appengine/compare/v1.6.5...v1.6.6
* cb27e3aa20...8632dd7979
* 0f9fa26af8...5a964db013
Reverts letsencrypt/boulder#5963
Turns out the tests are still flaky -- using the `grpc.WaitForReady(true)`
connection option results in sometimes seeing 9 entries added to the
purger queue, and sometimes 10 entries. Reverting because flakiness
on main should not be tolerated.
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.36.1 to 1.44.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.36.1...v1.44.0)
Also update akamai-purger integration test to avoid experimental API.
The `conn.GetState()` API is marked experimental and may change behavior
at any time. It appears to have changed between v1.36.1 and v1.44.0,
and so the akamai-purger integration tests which rely on it break.
Rather than writing our own loop which polls `conn.GetState()`, just
use the stable `WaitForReady(true)` connection option, and apply it to
all connections by setting it as a default option in the dial options.
The `go` directive inside go.mod determines certain behaviors of
the go command. Since we're using go 1.17 everywhere, we should
update our module's go directive to reflect that, and update its contents
to match the new behavior.
Particularly, updating to 1.17 here means that all indirect dependencies
are listed directly inside go.mod (in a separate block, to keep things clean),
and the go.sum and go.mod files are deleted from vendored dependencies
so that the go tool can correctly find the root of the module even when run
from a vendored dependency's subdirectory.
Followup from #5839.
I chose groupcache/lru as our LRU cache implementation because it's part
of the golang org, written by one of the Go authors, and very simple
and easy to read.
This adds an `AccountGetter` interface that is implemented by both the
AccountCache and the SA. If the WFE config includes an AccountCache field,
it will wrap the SA in an AccountCache with the configured max size and
expiration time.
We set an expiration time on account cache entries because we want a
bounded amount of time that they may be stale by. This will be used in
conjunction with a delay on account-updating pathways to ensure we don't
allow authentication with a deactivated account or changed key.
The account cache stores corepb.Registration objects because protobufs
have an established way to do a deep copy. Deep copies are important so
the cache can maintain its own internal state and ensure nothing external
is modifying it.
As part of this process I changed construction of the WFE. Previously,
"SA" and "RA" were public fields that were mutated after construction. Now
they are parameters to the constructor, along with the new "accountGetter"
parameter.
The cache includes stats for requests categorized by hits and misses.
This is a sort of proof of concept of the Redis interaction, which will
evolve into a tool for inspection and manual repair of missing entries,
if we find ourselves needing to do that.
The important bits here are rocsp/rocsp.go and
cmd/rocsp-tool/main.go. Also, the newly-vendored Redis client.
Update zlint from v3.2.0 to just past v3.3.0, pulling in both an update
to the zlint interface and a number of new and improved checks. In
particular, pull in `lint_dnsname_contains_prohibited_reserved_label`,
which checks that DNSNames do not begin with any two characters followed
by two dashes, unless those two leading characters are "xn".
Also, update our few custom lints to match the new zlint v3.3.0
interface.
Fixes#5720
Use the built-in grpc-go client and server interceptor chaining
utilities, instead of the ones provided by go-grpc-middleware.
Simplify our interceptors to call their handlers/invokers directly,
instead of delegating to the metrics interceptor, and add the
metrics interceptor to the chains instead.
Add Honeycomb tracing to all Boulder components which act as
HTTP servers, gRPC servers, or gRPC clients. Add many values
which we currently emit to logs to the trace spans. Add a way to
configure the Honeycomb integration to our config files, and by
default configure all of our tests to "mute" (send nothing).
Followup changes will refine the configuration, attempt to reduce
the new dependency load, and introduce better sampling.
Part of https://github.com/letsencrypt/dev-misc-tickets/issues/218
protoc now generates grpc code in a separate file from protobuf code.
Also, grpc servers are now required to embed an "unimplemented"
interface from the generated .pb.go file, which provides forward
compatibility.
Update the generate.go files since the invocation for protoc has changed
with the split into .pb.org and _grpc.pb.go.
Fixes#5368
Update the pinned version of zlint from v2.2.1 to v3.1.0.
Also update the relevant path from v2 to v3 in both go.mod
and in individual imports. Update the vendored files to match.
No changes from v2.2.1 to v3.1.0 appear to affect the lints
we directly care about (e.g. those that we explicitly ignore).
Fixes#5206
This brings in the following changes to zlint:
https://github.com/zmap/zlint/compare/v2.1.0...9ab0643
Importantly, this prevents the cert lifetime lint from triggering on
CA certs, and removes the OCSP url requirement lint entirely.
dependabot[bot]