Remove the secondsSinceSuccess metric, which was both never
being set and rather useless given the bursty nature of CRL update
scheduling.
Add an "issuer" label to the crl_updater_generated metric, to match
the other metrics exported by the updater and the storer.
Change the S3 object paths to not include the CRL Number.
At one point, the plan was to upload all of the CRL shards to
S3 paths containing their CRL Number (which monotonically
increases every generation), and then later move or symlink
them into paths not containing that number. However, we saw
that S3 does not have any atomic move or rename semantics,
so we decided to instead enable object versioning and upload
the shards to the same path every time. Apparently I never fixed
the object key computation to match the updated design.
The CRL Number is still stored on the object as a metadata tag.
- Create new package `crl`
- Add a common unique CRL identifier `crl.id` with constructor `crl.Id()`
- Replace `shardIdx` with `crl.Id` in `storer` and `updater` errors
- Add a common type for the `CRLNumber` field `crl.number` with constructor
`crl.Number()`
- Replace `CRLNumber` construction in CA and CRL package with `crl.Number()`
Resolves#6261
The iotuil package has been deprecated since go1.16; the various
functions it provided now exist in the os and io packages. Replace all
instances of ioutil with either io or os, as appropriate.
Create a new crl-storer service, which receives CRL shards via gRPC and
uploads them to an S3 bucket. It ignores AWS SDK configuration in the
usual places, in favor of configuration from our standard JSON service
config files. It ensures that the CRLs it receives parse and are signed
by the appropriate issuer before uploading them.
Integrate crl-updater with the new service. It streams bytes to the
crl-storer as it receives them from the CA, without performing any
checking at the same time. This new functionality is disabled if the
crl-updater does not have a config stanza instructing it how to connect
to the crl-storer.
Finally, add a new test component, the s3-test-srv. This acts similarly
to the existing mail-test-srv: it receives requests, stores information
about them, and exposes that information for later querying by the
integration test. The integration test uses this to ensure that a
newly-revoked certificate does show up in the next generation of CRLs
produced.
Fixes#6162
Realized that "ShardID" is a bad name, because a real unique
identifier of a shard would include the issuer, crl number, and
shard number. Switching to "ShardIdx" makes it clearer that
shards within a full and complete CRL are identified by a
zero-indexed integer.
Add a new config key `UpdateOffset` to crl-updater, which causes it to
run on a regular schedule rather than running immediately upon startup
and then every `UpdatePeriod` after that. It is safe for this new config
key to be omitted and take the default zero value.
Also add a new command line flag `runOnce` to crl-updater which causes
it to immediately run a single time and then exit, rather than running
continuously as a daemon. This will be useful for integration tests and
emergency situations.
Part of #6163
Add a collection of lints (structured similarly, but not identically,
to zlint's certificate lints) which check a variety of requirements
based on RFC 5280, the Baseline Requirements, and the Mozilla
Root Store Policy.
Add a method to lint CRLs to the existing linter package which
uses its fake issuer to sign the CRL, calls all of the above lints,
and returns all of their findings. Call this new method from within
the CA's new GenerateCRL method immediately before signing
the real CRL using the real issuer.
Fixes#6188
Create a new service named crl-updater. It is responsible for
maintaining the full set of CRLs we issue: one "full and complete" CRL
for each currently-active Issuer, split into a number of "shards" which
are essentially CRLs with arbitrary scopes.
The crl-updater is modeled after the ocsp-updater: it is a long-running
standalone service that wakes up periodically, does a large amount of
work in parallel, and then sleeps. The period at which it wakes to do
work is configurable. Unlike the ocsp-responder, it does all of its work
every time it wakes, so we expect to set the update frequency at 6-24
hours.
Maintaining CRL scopes is done statelessly. Every certificate belongs to
a specific "bucket", given its notAfter date. This mapping is generally
unchanging over the life of the certificate, so revoked certificate
entries will not be moving between shards upon every update. The only
exception is if we change the number of shards, in which case all of the
bucket boundaries will be recomputed. For more details, see the comment
on `getShardBoundaries`.
It uses the new SA.GetRevokedCerts method to collect all of the revoked
certificates whose notAfter timestamps fall within the boundaries of
each shard's time-bucket. It uses the new CA.GenerateCRL method to sign
the CRLs. In the future, it will send signed CRLs to the crl-storer to
be persisted outside our infrastructure.
Fixes#6163
Use the new //crl/x509 library in the CA, to make handling the
ReasonCode of each CRL entry significantly easier. This also
allows us to log the reason code along with each serial in the
CRL.
Also, make a couple tiny tweaks to the //crl/x509 package that
were discovered to be useful while writing this change. These
include moving it to a //crl/crl_x509 directory so that it doesn't
have to be aliased at import time.
Fixes#6199
Fork the pieces of the Go standard library's crypto/x509
package which are relevant to parsing, handling, and
signing CRLs.
In our fork, fix an upstream parsing bug, hoist the reasonCode
out of the crlEntryExtensions for easier usability, and enforce
that CRL Numbers are never longer than 20 octets.
Part of #6199
This is currently unused, and was a copy of the Go upstream package
anyhow. We will either re-copy when needed or wait for Go 1.15, which
will include the updates we need.
Copies in the code + test from go-review.googlesource.com/c/go/+/217298. Also requires a bunch of private crypto/x509 stuff that the change relies on, which somewhat bloats the diff. The only divergent code in crl/crl.go is in CreateCRL.
Fixes#4668.
Aaron Gable