This avoids serialization errors passing through gRPC.
Also, add a pass-through path in replaceInvalidUTF8 that saves an
allocation in the trivial case.
Fixes#6490
The things we need in crl/checker really only need x509.Certificate.
This allows us to remove a dependency on pkcs11key from the crl checker,
and transitively on CGO.
I've confirmed that `CGO_ENABLED=0 go build ./crl/checker` succeeds on
this branch, while it fails on main.
Make minor changes to our implementation of CAA Account and Method
Binding, as a result of reviewing the code in preparation for enabling
it in production. Specifically:
- Ensure that the validation method and account ID are included at the
request level, rather than waiting until we perform the checks which use
those parameters;
- Clean up code which assumed the validation method and account ID might
not be populated;
- Use the core.AcmeChallenge type (rather than plain string) for the
validation method everywhere;
- Update comments to reference the latest version and correct sections
of the CAA RFCs; and
- Remove the CAA feature flags from the config integration tests to
reflect that they are not yet enabled in prod.
I have reviewed this code side-by-side with RFC 8659 (CAA) and RFC 8657
(ACME CAA Account and Method Binding) and believe it to be compliant
with both.
We have a new method, omitZero, which is to allow `max_statement_time`
and `long_query_time` to be zeroed out, but copy/paste error left
`long_query_time` out. Fix it.
The SA's NewOrder and NewOrderAndAuthzs methods both write new rows (the
order itself, new authorizations, and new OrderToAuthz relations) to the
database, and then quickly turn around and query the database for a
bunch of authz rows to compute the status of the new order. This is
necessary because many orders are created already referencing existing
authorizations, and the state of those authorizations is not known at
order creation time, but does affect the order's status.
Due to replication lag, it can cause issues if the database writes go to
the primary database, but the follow-up read goes to a read-only
database replica which may be lagging slightly. To prevent this issue,
conduct the reads on the same transaction as the writes.
Fixes#6511
This requires setting --issuer-file and --url, but it allows (for
instance) collecting a big pile of serial numbers for a known issuer,
rather than having to keep whole certificates.
- Replace `-1` in return values with `0`. No callers were depending on
`-1`.
- Replace `count(` with `COUNT(` for the sake of readability.
- Replace `COUNT(1)` with `COUNT(*)` (https://mariadb.com/kb/en/count).
Both
versions provide identical outputs but let's standardize on the docs.
Fixes#6494
Right now the expiration mailer does one big SELECT on
`certificateStatus` to find certificates to work on, then several
thousand SELECTs of individual serial numbers in `certificates`.
Since it's more efficient to get that data as a stream from a single
query, rather than thousands of separate queries, turn that into a JOIN.
NOTE: We used to use a JOIN, and switched to the current approach in
#2440 for performance reasons. I _believe_ part of the issue was that at
the time we were not using READ UNCOMMITTED, so we may have been slowing
down the database by requiring it to keep copies of a lot of rows during
the query. Still, it's possible that I've misunderstood the performance
characteristics here and it will still be a regression to use JOIN. So
I've gated the new behavior behind a feature flag.
The feature flag required extracting a new function, `getCerts`. That in
turn required changing some return types so we are not as closely tied
to `core.Certificate`. Instead we use a new local type named
`certDERWithRegId`, which can be provided either by the new code path or
the old code path.
Deprecate these feature flags, which are consistently set in both prod
and staging and which we do not expect to change the value of ever
again:
- AllowReRevocation
- AllowV1Registration
- CheckFailedAuthorizationsFirst
- FasterNewOrdersRateLimit
- GetAuthzReadOnly
- GetAuthzUseIndex
- MozRevocationReasons
- RejectDuplicateCSRExtensions
- RestrictRSAKeySizes
- SHA1CSRs
Move each feature flag to the "deprecated" section of features.go.
Remove all references to these feature flags from Boulder application
code, and make the code they were guarding the only path. Deduplicate
tests which were testing both the feature-enabled and feature-disabled
code paths. Remove the flags from all config-next JSON configs (but
leave them in config ones until they're fully deleted, not just
deprecated). Finally, replace a few testdata CSRs used in CA tests,
because they had SHA1WithRSAEncryption signatures that are now rejected.
Fixes#5171Fixes#6476
Part of #5997
Move ocsp-responder's creation of its clock below its setup of metrics
and logging.
The call to `cmd.Clock()` gets the default logger and prints to the log
if the clock is being set to a fake time using an environment variable,
as we do in our integration tests. The call to `cmd.StatsAndLogging()`
unconditionally creates a logger and sets it as the default logger.
However, because `cmd.Clock()` has already used a (default) logger to
print its warning, the latter call was running into an error.
This actually means that, in our integration tests, ocsp-responder has
not been using the correctly-configured logger; it has been using the
default logger.
Fixes#6491
Create a new gRPC service named StorageAuthorityReadOnly which only
exposes a read-only subset of the existing StorageAuthority service's
methods.
Implement this by splitting the existing SA in half, and having the
read-write half embed and wrap an instance of the read-only half.
Unfortunately, many of our tests use exported read-write methods as part
of their test setup, so the tests are all being performed against the
read-write struct, but they are exercising the same code as the
read-only implementation exposes.
Expose this new service at the SA on the same port as the existing
service, but with (in config-next) different sets of allowed clients. In
the future, read-only clients will be removed from the read-write
service's set of allowed clients.
Part of #6454
Now that we have the ability to easily add multiple gRPC services to the
same server, and control access to each service individually, use that
capability to expose the CA's CertificateAuthority, OCSPGenerator, and
CRLGenerator services all on the same address/port. This will make
establishing connections to the CA easier, but no less secure.
Part of #6448
Add a new gRPC server interceptor (both unary and streaming) which
verifies that the mTLS info set on the persistent connection has a
client cert which contains a name which is allowlisted for the
particular service being called, not just for the overall server.
This will allow us to make more services -- particularly the CA and the
SA -- more similar to the VA. We will be able to run multiple services
on the same port, while still being able to control access to those
services on a per-client basis. It will also let us split those services
(e.g. into read-only and read-write subsets) much more easily, because a
client will be able to switch which service it is calling without also
having to be reconfigured to call a different address. And finally, it
will allow us to simplify configuration for clients (such as the RA)
which maintain connections to multiple different services on the same
server, as they'll be able to re-use the same address configuration.
Turn bgrpc.NewServer into a builder-pattern, with a config-based
initialization, multiple calls to Add to add new gRPC services, and a
final call to Build to produce the start() and stop() functions which
control server behavior. All calls are chainable to produce compact code
in each component's main() function.
This improves the process of creating a new gRPC server in three ways:
1) It avoids the need for generics/templating, which was slightly
verbose.
2) It allows the set of services to be registered on this server to be
known ahead of time.
3) It greatly streamlines adding multiple services to the same server,
which we use today in the VA and will be using soon in the SA and CA.
While we're here, add a new per-service config stanza to the
GRPCServerConfig, so that individual services on the same server can
have their own configuration. For now, only provide a "ClientNames" key,
which will be used in a follow-up PR.
Part of #6454
Have the database query return timestamps for when certificates
were issued for certain names. Use that information to compute
when the next time that name will be eligible for issuance again.
Include that timestamp in the error message and a Retry-After
HTTP header.
Fixes#6465
Add three new boolean config keys to the CA's config json:
- DisableCertService
- DisableOCSPService
- DisableCRLService
Setting any one of these flags to True and restarting the CA causes the
corresponding gRPC service (CertificateAuthority, OCSPGenerator, or
CRLGenerator, respectively) to not be started. The implementation is not
instantiated, the port is not opened, and no requests will be listened
for. All clients will receive failures to connect.
Also, create two temporary "dummy" implementations of the CRL and OCSP
services (and an interface for the fake OCSP service to meet), These are
necessary because the CertificateAuthority gRPC service currently
contains wrapper methods which replicate the methods of the OCSP and CRL
services, and which pass through calls to underlying implementation
objects. These objects must be replaced with fake objects which always
return an error when the service is disabled.
Fixes#6452
Remove the need for clients to explicitly call bgrpc.NewClientMetrics,
by moving that call inside bgrpc.ClientSetup. In case ClientSetup is
called multiple times, use the recommended method to gracefully recover
from registering duplicate metrics. This makes gRPC client setup much
more similar to gRPC server setup after the previous server refactoring
change landed.
When using `ct-test-srv` with Boulder infrastructure, it's important now
that the logIDs are correctly configured. This is a nice-to-have that
prints the logID for the provided EC privkey on startup, the same way
that `ct-test-srv` prints the EC pubkey.
Previously, the live-signing routine was lookking for
`rocsp.ErrRedisNotFound` errors in order to increment the
`certificate_not_found` metrics. But this was a bug, copy-pasted from
code higher in the file that does a similar check. The live-signing code
actually returns `responder.ErrNotFound`. Check for that error instead,
to properly increment our metrics.
Bumps
[github.com/prometheus/client_model](https://github.com/prometheus/client_model)
from 0.2.0 to 0.3.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="63fb9822ca"><code>63fb982</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/prometheus/client_model/issues/63">#63</a>
from prometheus/sparsehistogram</li>
<li><a
href="fdb567dcc1"><code>fdb567d</code></a>
Add note about native histograms to README</li>
<li><a
href="7f720d2282"><code>7f720d2</code></a>
Add note about experimental state of native histograms</li>
<li><a
href="1f8dcad122"><code>1f8dcad</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/prometheus/client_model/issues/59">#59</a>
from prometheus/beorn7/histogram</li>
<li><a
href="a7ff7138f2"><code>a7ff713</code></a>
Flatten the buckets of native histograms</li>
<li><a
href="421ad2b045"><code>421ad2b</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/prometheus/client_model/issues/58">#58</a>
from prometheus/beorn7/histogram</li>
<li><a
href="0da3265134"><code>0da3265</code></a>
Explain Span layout better</li>
<li><a
href="8171e83b1d"><code>8171e83</code></a>
Add float histograms and gauge histograms to proto spec</li>
<li><a
href="408689db4e"><code>408689d</code></a>
Merge branch 'master' into sparsehistogram</li>
<li><a
href="5c16fa2528"><code>5c16fa2</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/prometheus/client_model/issues/57">#57</a>
from prometheus/repo_sync</li>
<li>Additional commits viewable in <a
href="https://github.com/prometheus/client_model/compare/v0.2.0...v0.3.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Add two new config keys to the crl-updater:
* shardWidth, which controls the width of the chunks that we divide all
of time into, with a default value of "16h" (approximately the same as
today's shard width derived from 128 shards covering 90 days); and
* lookbackPeriod, which controls the amount of already-expired
certificates that should be included in our CRLs to ensure that even
certificates which are revoked immediately before they expire still show
up in aborts least one CRL, with a default value of "24h" (approximately
the same as today's lookback period derived from our run frequency of
6h).
Use these two new values to change the way CRL shards are computed.
Previously, we would compute the total time we care about based on the
configured certificate lifetime (to determine how far forward to look)
and the configured update period (to determine how far back to look),
and then divide that time evenly by the number of shards. However, this
method had two fatal flaws. First, if the certificate lifetime is
configured incorrectly, then the CRL updater will fail to query the
database for some certs that should be included in the CRLs. Second, if
the update period is changed, this would change the lookback period,
which in turn would change the shard width, causing all CRL entries to
suddenly change which shard they're in.
Instead, first compute all chunk locations based only on the shard width
and number of shards. Then determine which chunks we need to care about
based on the configured lookback period and by querying the database for
the farthest-future expiration, to ensure we cover all extant
certificates. This may mean that more than one chunk of time will get
mapped to a single shard, but that's okay -- each chunk will remain
mapped to the same shard for the whole time we care about it.
Fixes#6438Fixes#6440
Collapse most of our boilerplate gRPC creation steps (in particular,
creating default metrics, making the server and listener, registering
the server, creating and registering the health service, filtering
shutdown errors from the output, and gracefully stopping) into a single
function in the existing bgrpc package. This allows all but one of our
server main functions to drop their calls to NewServer and
NewServerMetrics.
To enable this, create a new helper type and method in the bgrpc
package. Conceptually, this could be just a new function, but it must be
attached to a new type so that it can be generic over the type of gRPC
server being created. (Unfortunately, the grpc.RegisterFooServer methods
do not accept an interface type for their second argument).
The only main function which is not updated is the boulder-va, which is
a special case because it creates multiple gRPC servers but (unlike the
CA) serves them all on the same port with the same server and listener.
Part of #6452
Add the Issuing Distribution Point extension to all of our end-entity
CRLs. The extension contains the Distribution Point, the URL from
which this CRL is meant to be downloaded. Because our CRLs are
sharded, this URL prevents an on-path attacker from substituting a
different shard than the client expected in order to hide a revocation.
The extension also contains the OnlyContainsUserCerts boolean,
because our CRLs only contain end-entity certificates.
The Distribution Point url is constructed from a configurable base URI,
the issuer's NameID, the shard index, and the suffix ".crl". The base
URI must use the "http://" scheme and must not end with a slash.
openssl displays the IDP extension as:
```
X509v3 Issuing Distribution Point: critical
Full Name:
URI:http://c.boulder.test/66283756913588288/0.crl Only User Certificates
```
Fixes#6410
Rather than simply refusing to write OCSP Response bytes to the
database (which is what ROCSP Stage 6 did), Stage 7 refuses to
even generate those bytes in the first place. We obviously can't
disable OCSP Response generation in the CA, since it still needs to
be usable by the ocsp-responder's live-signing path, so instead we
disable it in all of the non-live-signing codepaths (orphan finder,
issue precertificate, revoke certificate, and re-revoke certificate)
which have previously called GenerateOCSP.
Part of #6285
Clean up several spots where we were behaving differently on
go1.18 and go1.19, now that we're using go1.19 everywhere. Also
re-enable the lint and generate tests, and fix the various places where
the two versions disagreed on how comments should be formatted.
Also clean up the OldTLS codepaths, now that both go1.19 and our
own feature flags have forbidden TLS < 1.2 everywhere.
Fixes#6011
When the RA is generating OCSP (as part of new issuance, revocation,
or when its own GenerateOCSP method is called by the ocsp-responder)
have it use the CA's dedicated OCSPGenerator service, rather than
calling the method exposed by the CA's catch-all CertificateAuthority
service. To facilitate this, add a new GRPCClientConfig stanza to the
RA.
This change will allow us to remove the GenerateOCSP and GenerateCRL
methods from the catch-all CertificateAuthority service, allowing us to
independently control which kinds of objects the CA is willing to sign
by turning off individual service interfaces. The RA's new config stanza
will need to be populated in prod before further changes are possible.
Fixes#6451
These flags are set in both staging and prod. Deprecate them, make
all code gated behind them the only path, and delete code (multi_source)
which was only accessible when these flags were not set.
Part of #6285
Thing brings in a number of new lints, including those which check for
correct encoding of the KeyUsage bitstring, which has led to incidents
for a number of CAs recently.
This method simply returns the greatest notAfter timestamp in
the certificateStatus table. This will be used by the crl-updater
to ensure that it includes all unexpired certificates in its CRLs,
rather than only those which happen to fall within its configured
bounds.
Part of #6438
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.3.7...v0.3.8)
This also transitively updates x/tools and x/sync, which is good because those
are unversioned packages which are otherwise ignored by dependabot.
Note that we are not affected by the vulnerability which prompted the release
of version 0.3.8; the affected files are in the language subpackage which we
do not use or vendor.
Add a new configuration key to the CA which allows us to
specify the "base URL" for our CRLs. This will be necessary
before including an Issuing Distribution Point extension in our
CRLs, or a CRL Distribution Point in our certificates.
Part of #6410
Boulder builds a single binary which is symlinked to the different binary names, which are included in its releases.
However, requiring symlinks isn't always convenient.
This change makes the base `boulder` command usable as any of the other binary names. If the binary is invoked as boulder, runs the second argument as the command name. It shifts off the `boulder` from os.Args so that all the existing argument parsing can remain unchanged.
This uses the subcommand versions in integration tests, which I think is important to verify this change works, however we can debate whether or not that should be merged, since we're using the symlink method in production, that's what we want to test.
Issue #6362 suggests we want to move to a more fully-featured command-line parsing library that has proper subcommand support. This fixes one fragment of that, by providing subcommands, but is definitely nowhere near as nice as it could be with a more fully fleshed out library. Thus this change takes a minimal-touch approach to this change, since we know a larger refactoring is coming.
- Add a new field, `RetryAfter` to `BoulderError`s
- Add logic to wrap/unwrap the value of the `RetryAfter` field to our gRPC error
interceptor
- Plumb `RetryAfter` for `DuplicateCertificateError` emitted by RA to the WFE
client response header
Part of #6256
Add output listing the number of CRLs inspected, their total size in
bytes, and the total number of revocation entries seen. Also print the
oldest thisUpdate timestamp observed.
Allow the -issuer flag to take the special value "-", which causes no
issuer file to be loaded and no CRL signatures to be validated. Also
improve the error message if no -issuer flag is provided at all.
Fixes#6409
- Add a new gRPC client config field which overrides the dNSName checked in the
certificate presented by the gRPC server.
- Revert all test gRPC credentials to `<service>.boulder`
- Revert all ClientNames in gRPC server configs to `<service>.boulder`
- Set all gRPC clients in `test/config` to use `serverAddress` + `hostOverride`
- Set all gRPC clients in `test/config-next` to use `srvLookup` + `hostOverride`
- Rename incorrect SRV record for `ca` with port `9096` to `ca-ocsp`
- Rename incorrect SRV record for `ca` with port `9106` to `ca-crl`
Resolves#6424
Jacob Hoffman-Andrews
dependabot[bot]
Aaron Gable
J.C. Jones
Samantha
David
Matthew McPherrin