Add a new feature flag `GetAuthzUseIndex` which causes the SA
to add `USE INDEX (regID_identifer_status_expires_idx)` to its authz2
database queries. This should encourage the query planner to actually
use that index instead of falling back to large table-scans.
Fixes#5822
These hashes are useful for OCSP computations, as they are the two
values that are used to uniquely identify the issuer of the given cert in
an OCSP request. Here, they are restricted to SHA1 only, as Boulder
only supports SHA1 for OCSP, as per RFC 5019.
In addition, because the `ID`, `NameID`, `NameHash`, and `KeyHash`
are relatively expensive to compute, introduce a new constructor for
`issuance.Certificate` that computes all four values at startup time and
then simply returns the precomputed values when asked.
Today, the revocation codepaths involve parsing the to-be-revoked
certificate multiple times: inside `admin-revoker` itself, inside the
RA's `AdministrativelyRevokeCertificate` method, and again in its helper
`revokeCertificate`. In addition, we use the fact that we have the full
certificate multiple times: to log various attributes of it, to compute
its `IssuerNameID`, and more. All of this will fail if we ever issue a
cert that is malformed to the point that it cannot be parsed.
Add a new argument to the `AdministrativelyRevokeCertificateRequest`
that allows the certificate to be identified by serial only, instead of
by full certificate bytes. Add support for this in the gRPC handler by
using the serial to construct a dummy in-memory Certificate object.
Support this in the `revokeCertificate` codepath by checking to see if
the passed-in cert has any underlying raw DER bytes, and if not,
triggering the new codepath that does everything via the serial.
In order to support this, unfortunately we have to add a second
in-memory map to the RA, so that it can look up issuer certs by either
name ID or old-style ID, as the IDs gleaned from the database (instead
of from the cert itself) may still be old-style. This will be removed
when the old-style Issuer IDs have aged out.
Fixes#5759
Also go back to using Gorp's Insert method to insert CertificateStatus.
In b557d870c7, we switched from Insert to an Exec
with explicitly listed fields, as a temporary measure for a table migration
(related to StoreIssuerInfo). In 3d9c31580a,
with the migration done and the feature flag turned on, we cleaned up the
feature flag but did not revert back to an Insert. This finishes that cleanup.
Adding the ID field (and telling Gorp it's the primary key) to Certificate
and CertificateStatus objects is useful for writing tests that rely on the
ID field. It also removes a little hack where `CertStatusMetadata` had an
ID field because CertificateStatus didn't.
Add a feature flag which causes the SA to switch between using the
traditional read-write database connector (pointed at the primary db)
or the newer read-only database connector (usually pointed at a
replica) when executing the `GetAuthorizations2` query.
Re-parsing the certificate after we're sure we issued it accomplishes
nothing except wasting CPU cycles. This duplicate work was left over
after the removal of the old codepath which was incapable of revoking
precertificates.
Make a few changes to admin-revoker to help its various modes all fit
the same general interface:
- Create a `revoker` struct with the various revocation mechanisms as
methods so that the gRPC clients and other infra don't need to be
passed as arguments.
- Standardize all methods to take a `ctx` as their first argument.
- Remove transactions from the mechanism cases in `main`, since the
transactions don't gain us anything: the reads use the transaction
object but the writes don't (they use gRPC) so we don't gain any
consistency).
- Move non-cli-parsing logic from `main` into revoker methods so that
cases in `main` all look basically the same.
- Reorder helper methods and cases to match the documented order
of the various subcommands.
This will make it easier to add new revocation methods to admin-revoker
in the future, e.g. for #5759 and #5785.
This pulls most of ocsp_updater into its own package. This follows more
closely the pattern we use elsewhere, where we have a struct for the
service, and a main.go that sets it up.
Also splits out the JSON config into a third package,
`ocsp_updater/config`, to avoid circular dependency issues.
The only changes to the implementation of OCSPUpdater are renaming
`newUpdater` to `ocsp_updater.New`, and `tick` to `Tick`. Also moved
some testdata into `ocsp_updater/testdata/` and updated the tests
appropriately. And changed some `configureDb` calls in the unittests to
use sa.NewDbMap instead.
Remove goreleaser since we're not currently using it.
Remove the script that moves PRs from master to main because it hasn't
been needed in practice.
Set persist-credentials: true for Boulder CI.
This splits rocsp-tool/main.go into main.go, client.go, issuers.go,
and inflight.go.
Adds tests for issuers and inflight, plus storeResponse in
client.go. Doesn't yet have a test for loadFromDB in client.go.
Part of #5786
Previously loadFromDB was calling cl.storeResponse, which parses, stores, and
then fetches a response, and logs it to stderr. Since we'll be storing
responses at high volume, we don't want to log them all to stderr. And
we're willing to trust that the CA signed a valid response, so we don't
need to parse it again. And we certainly don't need to fetch it right
after storing it.
Fixes#5782
Most Boulder logging is supposed to go through our logging subsystem, where a
checksum is added. However, very occasionally Boulder emits output on stdout or
stderr. For instance this can happen during panics, or if we load a pkcs11
module that emits messages on stdout or stderr.
When that happens, the logs are collected by systemd and sent into rsyslog with
the same programname as the lines that went through our logging subsystem. This
causes spurious alerts from log-validator because it can't find the checksum in
those log lines.
This change reduces the risk of spurious alerting by providing a separate metric
for "malformed log line" vs "well-formed log line with a checksum mismatch."
We'll still want to alert on "malformed log line", in case a future change to
logging causes all log lines to be malformed. But we can set the threshold for
it much higher.
Fixes#5771
This scans the database for certificateStatus rows, gets them signed by the CA, and writes them to Redis.
Also, bump the default PoolSize for Redis to 100.
Previously we would emit them all together in one log event. The log
event had interior newlines, but our log system removes those newlines
as a matter of course, resulting in an error message that is much too
long.
This change replaces that with a single log line per error.
This reverts commit e3ce816425,
which was reviewed in https://github.com/letsencrypt/boulder/pull/5607.
This change caused database queries to exceed the maximum packet size
and fail. Because this was an opportunistic optimization, reverting it
is the safest course moving forward.
Implement our label validation logic directly as specified in the
relevant RFCs: P-Labels are a subset of XN-Labels which are a
subset of Reserved LDH Labels which are a subset of LDH Labels.
This approach allows us to much more clearly document what each
check is doing, to remove two regular expressions, and to simplify
one additional regex.
Fixes https://github.com/letsencrypt/dev-misc-tickets/issues/247
The draft requires that the renewalInfo endpoint have a
Retry-After header indicating how often clients should poll
for their renewal information. As per our previous thinking,
set this timer to 6 hours for now.
Fixes#5765
When wait-for-it is trying to connect and failing, bash emits errors on
stderr. This captures those errors and sends them to /dev/null.
This also replaces an internal wait_tcp_port function inside
entrypoint.sh with a call to wait-for-it.sh.
When a valid authorization is stored in the database the authorization
column attemptedAt is set based on the challenge `Validated` value. Use
this value in `checkAuthorizationsCAA` to determine if an authorization
is sufficiently stale to need a recheck of the CAA DNS record. Error if the
time is nil. Keeps old codepath for safety check and increments a metric
if the old codepath is used.
Previously we were using the `deploy:` config field, but that's not
supported in some cases. Splitting things out also allows us to
explicitly assign IP addresses rather than relying on their most-likely
assignment to containers.
The `Source` interface in ocsp-responder defines a `Response` function.
Add a context to the function signature so that ocsp lookups can be more
tracable and cancelable. This is also a precursor to having cancelable
parallel lookups to multiple sources.
This is a sort of proof of concept of the Redis interaction, which will
evolve into a tool for inspection and manual repair of missing entries,
if we find ourselves needing to do that.
The important bits here are rocsp/rocsp.go and
cmd/rocsp-tool/main.go. Also, the newly-vendored Redis client.
Update the SA's CertStatusMetadata methods to include the "id"
column in the resulting object; also create a new struct representing
this object and delete the old unused methods. Plumb this id through
all of ocsp-updater, and use it in the SQL queries which update row
with new Expired statuses or with newly-signed OCSP responses.
This should allow the updates to be ever-so-slightly more efficient.
Fixes#5655Fixes#5587
This gets us ready to add writing to Redis from ocsp-updater. The Go
redis client requires different configuration for cluster operation
than non-cluster, so we need to simulate a cluster in our integration
environment. Cluster operation requires a manual initialization step,
which you can do like so:
```
docker-compose up -d bredis docker-compose exec bredis bash
/test/redis-create.sh
```
I still need to figure out how to make that happen automatically during
integration tests and when you run docker-compose up.
The hex values in redis.config are randomly generated passwords for the
different users.
Fixes#5723
Use `sa.SelectCertificates` instead of `sa.SelectCertificate` to
fetch the entire batch of certificates all at once, instead of doing
up to 10k individual certificate selections in serial.
Update our contributing guidelines to state that we prefer to
do error checking in a separate stanza, rather than doing error
assignment and checking on the same line. At this time, there
are less than 400 instances of the latter (unpreferred) pattern
in the Boulder codebase, and 1300+ instances of the former
(preferred) pattern.
Add a unit test and an integration test that both exercise the new
experimental ACME Renewal Info endpoint. These tests do not
yet validate the contents of the response, just that the appropriate
HTTP response code is returned, but they will be developed as the
code under test evolves.
Fixes#5674
Finally, the second-to-last step of switching from IssuerIDs to
IssuerNameIDs: have the CA store the IssuerNameID in the
certStatus database when first issuing precertificates and final
certificates.
This is the change we can't come back from: once this is deployed,
we've effectively changed our database schema (by changing
the semantic meaning of the certStatus table's "IssuerID" column).
Although it can be rolled back and no harm should come to anything,
rolling back (e.g. because some component actually *doesn't* handle
this gracefully) will not remove the data that was written while it
was deployed.
Part of #5152
Add functionality to purge by cache tags in our Akamai CachePurgeClient.
Use that functionality in a new manual mode of akamai-purger, which takes
a single tag with the `--tag` flag, or a file containing multiple tags
with `--tag-file`.
A tag file containing a random set of cache tags can be generated with:
printf "%x\n" $(seq 0 255) | shuf -n 5
Add a new feature flag to control whether or not the experimental ARI
information is exposed. Add a new entry to the Directory object which
provides the base URL for ARI requests. Add a new handler to the WFE
which parses incoming requests and returns reasonable renewalInfo.
Part of #5674
Introduce one cache tag: the last byte (hex-encoded) of the serial
number. This allows us to purge groups of responses, in chunks of
1/256 of our whole cache. We assume this is more or less evenly
distributed because serial numbers are random.
Fixes#5736.
- Add new function `SelectPrecertificates` to `SA` which returns `[]CertWithID`
- Replace `admin-revoker` calls to `sa.SelectCertificate(s)` with sa.SelectPrecertificate(s)
- Add SQL permissions for the `revoker` user to the `precertificates` table
Fixes#5708
Update the version of golangci-lint we use in our docker image,
and update the version of the docker image we use in our tests.
Fix a couple places where we were violating lints (ineffective assign
and calling `t.Fatal` from outside the main test goroutine), and add
one lint (using math/rand) to the ignore list.
Fixes#5710
Update zlint from v3.2.0 to just past v3.3.0, pulling in both an update
to the zlint interface and a number of new and improved checks. In
particular, pull in `lint_dnsname_contains_prohibited_reserved_label`,
which checks that DNSNames do not begin with any two characters followed
by two dashes, unless those two leading characters are "xn".
Also, update our few custom lints to match the new zlint v3.3.0
interface.
Fixes#5720
Jacob Hoffman-Andrews