Simplify the way we load and handle CT logs: rather than keeping them
grouped by operator, simply keep a flat list and annotate each log with
its operator's name. At submission time, instead of shuffling operator
groups and submitting to one log from each group, shuffle the whole set
of individual logs.
Support tiled logs by similarly annotating each log with whether it is
tiled or not.
Also make the way we know when to stop getting SCTs more robust.
Previously we would stop as soon as we had two, since we knew that they
would be from different operator groups and didn't care about tiled
logs. Instead, introduce an explicit CT policy compliance evaluation
function which tells us if the set of SCTs we have so far forms a
compliant set.
This is not our desired end-state for CT log submission. Ideally we'd
like to: simplify things even further (don't race all the logs, simply
try to submit to two at a time), improve selection (intelligently pick
the next log to submit to, rather than just a random shuffle), and
fine-tune latency (tiled logs should have longer timeouts than classic
ones). Those improvements will come in future PRs.
Part of https://github.com/letsencrypt/boulder/issues/7872
Add a new config field for profiles which causes the profile to omit the
AIA OCSP URI. It can only be omitted if the CRLDP extension is
configured to be included instead. Enable this flag in config-next.
When a certificate is revoked, if it does not have an AIA OCSP URI,
don't bother with an Akamai OCSP purge.
Builds on #8089
Most of the changes in this PR relate to tests. Different from #8089, I
chose to keep testing of OCSP in the config-next world. This is because
we intend to keep operating OCSP even after we have stopped including it
in new certificates. So we should test it in as many environments as
possible.
Adds a WithURLFallback option to ocsp_helper. When
`ocsp_helper.ReqDer()` is called for a certificate with no OCSP URI, it
will query the fallback URL instead. As before, if the certificate has
an OCSP URI ocsp_helper will use that. Use that for all places in the
integration tests that call ocsp_helper.
Add a new custom lint which sends CRLs to PKIMetal, and configure it to
run in our integration test environment. Factor out most of the code
used to talk to the PKIMetal API so that it can be shared by the two
custom lints which do so. Add the ability to configure lints to the
CRLProfileConfig, so that zlint knows where to load the necessary custom
config from.
Replace the bpkilint container with a new bpkimetal container. Update
our custom lint which calls out to that API to speak PKIMetal's (very
similar) protocol instead. Update our zlint custom configuration to
configure this updated lint.
Fixes https://github.com/letsencrypt/boulder/issues/8009
We have upgraded to go1.24.1 in production, and no longer need to test
go1.23.x. Updating the version in our go.mod also allows us to begin
using x509.Certificate.Policies instead of .PolicyIdentifiers.
Populate the new x509.Certificate.Policies field everywhere we currently populate the x509.Certificate.PolicyIdentifiers field. This allows Go to use whichever field it prefers (go1.23 prefers PolicyIdentifiers, go1.24 prefers Policies) as the source of truth when serializing a certificate.
Part of https://github.com/letsencrypt/boulder/issues/7148
Increase pkilint timeout from 200ms to 2s. In #8006 I found that errors
were stemming from timeouts talking to the bpkilint container. These
probably showed up in TestRevocation particularly because that
integration test now issues for many certificates in parallel. Pkilint's
slowness, combined with the relatively small number of cores in CI,
probably resulted in some requests taking too long.
In https://github.com/letsencrypt/boulder/pull/7005 several safety
checks were added to the `ceremony` tool:
This change extracts the `RawSubject` to `RawIssuer` DER byte comparison
into the `//linter` package proper so that it can serve both `//ca` and
`//cmd/ceremony`.
Adds a helper function `verifyTBSCertificateDeterminism` to `//ca`
similar to an existing check in `//cmd/ceremony`. This code is not
shared because we want `//cmd/ceremony` to largely stand alone from
boulder proper. The helper performs a byte comparison on the
`RawTBSCertificate` DER bytes for a given linting certificate and leaf
certificate. The goal is to verify that `x509.CreateCertificate` was
deterministic and produced identical DER bytes after each signing
operation.
Fixes https://github.com/letsencrypt/boulder/issues/6965
Remove the CA's global "crldpBase" config item, and the code which used
it to produce a IDP URI in our CRLs if it was configured.
This config item has been replaced by per-issuer crlURLBase configs
instead, because we have switched our CRL URL format from
"commonURL/issuerID/shard.crl" to "issuerURL/shard.crl" in anticipation
of including these URLs directly in our end-entity certs.
IN-10046 tracked the corresponding change in prod
Add a new "LintConfig" item to the CA's config, which can point to a
zlint configuration toml file. This allows lints to be configured, e.g.
to control the number of rounds of factorization performed by the Fermat
factorization lint.
Leverage this new config to create a new custom zlint which calls out to
a configured pkilint API endpoint. In config-next integration tests,
configure the lint to point at a new pkilint docker container.
This approach has three nice forward-looking features: we now have the
ability to configure any of our lints; it's easy to expand this
mechanism to lint CRLs when the pkilint API has support for that; and
it's easy to enable this new lint if we decide to stand up a pkilint
container in our production environment.
No production configuration changes are necessary at this time.
Fixes https://github.com/letsencrypt/boulder/issues/7430
Improve our custom CRL IssuingDistributionPoint lint to more accurately
reflect the fact that an IssuingDistributionPoint's DistributionPoint
field's FullName field can contain multiple GeneralNames.
Part of https://github.com/letsencrypt/boulder/issues/7296
Remove the Profile field from issuance.Issuer, to reflect the fact that
profiles are in fact independent pieces of configuration which can be
shared across (and are configured independently of) multiple issuers.
Move the IssuerURL, OCSPUrl, and CRLURL fields from issuance.Profile to
issuance.Issuer, since they reflect fundamental attributes of the
issuer, rather than attributes of a particular profile. This also
reflects the location at which those values are configured, in
issuance.IssuerConfig.
All other changes are fallout from the above: adding a Profile argument
to various methods in the issuance and linting packages, adding a
profile field to the caImpl struct, etc. This change paves the way for
two future changes: moving OCSP and CRL creation into the issuance
package, and supporting multiple simultaneous profiles that the CA can
select between.
Part of https://github.com/letsencrypt/boulder/issues/7159
Part of https://github.com/letsencrypt/boulder/issues/6316
Part of https://github.com/letsencrypt/boulder/issues/6966
Zlint is deprecating lint.Lint in favour of lint.CertificateLint.
The main difference is that metadata is now its own struct, shared with
lint.RevocationListLint and presumably future lint types.
* Allows the ceremony tool to add the `onlyContainsCACerts` flag to the
`IssuingDistributionPoint` extension[1] for CRLs.
* Add a lint to detect basic usage of this new flag.
* Add a helper function which doesn't (yet) exist in golang
x/crypto/cryptobyte named `ReadOptionalASN1BooleanWithTag` which
searches for an optional DER-encoded ASN.1 element tagged with a given
tag e.g. onlyContainsUserCerts and reports values back to the caller.
* Each revoked certificate in the CRL config is checked for is `IsCA` to
maintain conformance with RFC 5280 Section 6.3.3 b.2.iii [2].
> (iii) If the onlyContainsCACerts boolean is asserted in the
> IDP CRL extension, verify that the certificate
> includes the basic constraints extension with the cA
> boolean asserted.
Fixes https://github.com/letsencrypt/boulder/issues/7047
1. https://datatracker.ietf.org/doc/html/rfc5280#section-5.2.5
2. https://datatracker.ietf.org/doc/html/rfc5280#section-6.3.3
Delete our forked version of the x509 library, and update all call-sites
to use the version that we upstreamed and got released in go1.21. This
requires making a few changes to calling code:
- replace crl_x509.RevokedCertificate with x509.RevocationListEntry
- replace RevocationList.RevokedCertificates with
RevocationList.RevokedCertificateEntries
- make RevocationListEntry.ReasonCode a non-pointer integer
Our lints cannot yet be updated to use the new types and fields, because
those improvements have not yet been adopted by the zcrypto/x509 package
used by the linting framework.
Fixes https://github.com/letsencrypt/boulder/issues/6741
Removes the SignatureAlgorithm field while constructing the lint issuer.
Depending on the realIssuer, which could be either an intermediate or
cross-signed intermediate, the SignatureAlgorithm of that certificate
may differ from the root certificate that had signed it causing the
following error during CA startup.
```
Starting service boulder-ca-a
16:49:29.437776 3 boulder-ca qM-tDQA [AUDIT] Couldn't load issuers: failed to create lint issuer: x509: requested SignatureAlgorithm does not match private key type
```
Related to work done in
https://github.com/letsencrypt/boulder/pull/7005.
---------
Co-authored-by: Aaron Gable <aaron@letsencrypt.org>
In `//cmd/ceremony`:
* Added `CertificateToCrossSignPath` to the `cross-certificate` ceremony
type. This new input field takes an existing certificate that will be
cross-signed and performs checks against the manually configured data in
each ceremony file.
* Added byte-for-byte subject/issuer comparison checks to root,
intermediate, and cross-certificate ceremonies to detect that signing is
happening as expected.
* Added Fermat factorization check from the `//goodkey` package to all
functions that generate new key material.
In `//linter`:
* The Check function now exports linting certificate bytes. The idea is
that a linting certificate's `tbsCertificate` bytes can be compared
against the final certificate's `tbsCertificate` bytes as a verification
that `x509.CreateCertificate` was deterministic and produced identical
DER bytes after each signing operation.
Other notable changes:
* Re-orders the issuers list in each CA config to match staging and
production. There is an ordering issue mentioned by @aarongable two
years ago on IN-5913 that didn't make it's way back to this repository.
> Order here matters – the default chain we serve for each intermediate
should be the first listed chain containing that intermediate.
* Enables `ECDSAForAll` in `config-next` CA configs to match Staging.
* Generates 2x new ECDSA subordinate CAs cross-signed by an RSA root and
adds these chains to the WFE for clients to download.
* Increased the test.sh startup timeout to account for the extra
ceremony run time.
Fixes https://github.com/letsencrypt/boulder/issues/7003
---------
Co-authored-by: Aaron Gable <aaron@letsencrypt.org>
Our linting system uses a throwaway key to sign an untrusted version of
the to-be-signed cert, then runs the lints over that. But this means
that, when linting a self-signed cert, the signature no longer matches
the embedded public key. This in turn causes a bunch of zlint's checks
to think they're linting a Subordinate CA cert, rather than a Root CA
cert.
Change our linting system to make the lint cert appear self-signed when
the input cert is intended to be self-signed.
Update zlint to v3.5.0, which introduces scaffolding for running lints
over CRLs.
Convert all of our existing CRL checks to structs which match the zlint
interface, and add them to the registry. Then change our linter's
CheckCRL function, and crl-checker's Validate function, to run all lints
in the zlint registry.
Finally, update the ceremony tool to run these lints as well.
This change touches a lot of files, but involves almost no logic
changes. It's all just infrastructure, changing the way our lints and
their tests are shaped, and moving test files into new homes.
Fixes https://github.com/letsencrypt/boulder/issues/6934
Fixes https://github.com/letsencrypt/boulder/issues/6979
The upstream zlint lints are organized not by what kind of certificate
they apply to, but what source they are from. This change rearranges
(and slightly renames) our custom lints to match the same structure.
This will make it easier for us to temporarily add lints (e.g. for our
CRLs) which we intend to upstream to zlint later.
Part of https://github.com/letsencrypt/boulder/issues/6934
Return the sentinel error indicative of lint violation from
`linter.ProcessResultSet()` instead of `issuance`. This removes a
potential source of false-positives.
It's useful to be able to get a copy of the linting certificate out of
the process, so we can store it in the database for use by processes
that want a certificate-shaped object (for instance, scanning
possibly-issued public keys for a newly discovered weakness in key
generation).
To do this, while still ensuring that it's impossible to issue a
certificate without linting it, return an IssuanceToken from the linting
process. The IssuanceToken has a private field containing the template
that was used for linting. To issue a final certificate, the
IssuanceToken has to be redeemed, and only the template stored in it can
be used.
Add the Issuing Distribution Point extension to all of our end-entity
CRLs. The extension contains the Distribution Point, the URL from
which this CRL is meant to be downloaded. Because our CRLs are
sharded, this URL prevents an on-path attacker from substituting a
different shard than the client expected in order to hide a revocation.
The extension also contains the OnlyContainsUserCerts boolean,
because our CRLs only contain end-entity certificates.
The Distribution Point url is constructed from a configurable base URI,
the issuer's NameID, the shard index, and the suffix ".crl". The base
URI must use the "http://" scheme and must not end with a slash.
openssl displays the IDP extension as:
```
X509v3 Issuing Distribution Point: critical
Full Name:
URI:http://c.boulder.test/66283756913588288/0.crl Only User Certificates
```
Fixes#6410
Add crl-checker, a simple tool which downloads, parses, lints,
and validates signatures on a list of CRLs. It takes its input in
the form of a JSON Array of Sharded CRL URLs, the exact
same format as we will be disclosing in CCADB.
We can add additional checks -- such as ensuring that a set of
known-revoked serials are present, or checking that all of the
downloaded CRLs are "recent enough" -- over time.
Add a collection of lints (structured similarly, but not identically,
to zlint's certificate lints) which check a variety of requirements
based on RFC 5280, the Baseline Requirements, and the Mozilla
Root Store Policy.
Add a method to lint CRLs to the existing linter package which
uses its fake issuer to sign the CRL, calls all of the above lints,
and returns all of their findings. Call this new method from within
the CA's new GenerateCRL method immediately before signing
the real CRL using the real issuer.
Fixes#6188
Add a new code path to the ctpolicy package which enforces Chrome's new
CT Policy, which requires that SCTs come from logs run by two different
operators, rather than one Google and one non-Google log. To achieve
this, invert the "race" logic: rather than assuming we always have two
groups, and racing the logs within each group against each other, we now
race the various groups against each other, and pick just one arbitrary
log from each group to attempt submission to.
Ensure that the new code path does the right thing by adding a new zlint
which checks that the two SCTs embedded in a certificate come from logs
run by different operators. To support this lint, which needs to have a
canonical mapping from logs to their operators, import the Chrome CT Log
List JSON Schema and autogenerate Go structs from it so that we can
parse a real CT Log List. Also add flags to all services which run these
lints (the CA and cert-checker) to let them load a CT Log List from disk
and provide it to the lint.
Finally, since we now have the ability to load a CT Log List file
anyway, use this capability to simplify configuration of the RA. Rather
than listing all of the details for each log we're willing to submit to,
simply list the names (technically, Descriptions) of each log, and look
up the rest of the details from the log list file.
To support this change, SRE will need to deploy log list files (the real
Chrome log list for prod, and a custom log list for staging) and then
update the configuration of the RA, CA, and cert-checker. Once that
transition is complete, the deletion TODOs left behind by this change
will be able to be completed, removing the old RA configuration and old
ctpolicy race logic.
Part of #5938
Add `stylecheck` to our list of lints, since it got separated out from
`staticcheck`. Fix the way we configure both to be clearer and not
rely on regexes.
Additionally fix a number of easy-to-change `staticcheck` and
`stylecheck` violations, allowing us to reduce our number of ignored
checks.
Part of #5681
Update zlint from v3.2.0 to just past v3.3.0, pulling in both an update
to the zlint interface and a number of new and improved checks. In
particular, pull in `lint_dnsname_contains_prohibited_reserved_label`,
which checks that DNSNames do not begin with any two characters followed
by two dashes, unless those two leading characters are "xn".
Also, update our few custom lints to match the new zlint v3.3.0
interface.
Fixes#5720
Add a new lint, which applies to all certs, not just one kind of
root/intermediate/subscriber cert, which checks that the validity
period is a round number of minutes; i.e. that it doesn't have an
extra second of validity, like 90d+1s.
In general, if all certs are issued with validities much less than
the limits imposed by root program requirements, being off by one
second one way or another shouldn't matter much. But since it is
easy to mistakenly configure a cert to have notBefore and notAfter
timestamps with the exact same second value, this lint will force
people configuring profiles to think critically about their cert
lifetimes.
Fixes#5669
Add a collection of custom lints to enforce that our issuance of
Subscriber (via normal Boulder operation) and Root and Intermediate CA
Certificates (via the Ceremony tool) abides by the requirements we
place on ourselves via our CPS. Provide a small collection of useful
constants for these lints to share. Import all of these lints from our
lint package, so that they are automatically registered with zlint's
`GlobalRegistry` and are automatically included in all of our lint
checks.
At this time, only three lints are included, checking that the validity
periods of our various certificate types do not exceed their CPS-set
maximums. Additional lints for key sizes, distinguished names, key
usages, policy OIDs, AIA URLs, and more will be added in the future.
Part of #5492
In go1.17, the `x509.CreateCertificate()` method fails if the provided
Signer (private key) and Parent (cert with public key) do not match.
This change both updates the lint library to create and use an issuer
cert whose public key matches the throwaway private key used for lint
signatures, and overhauls its public interface for readability and
simplicity.
Rename the `lint` library to `linter`, to allow other methods to be
renamed to reduce word repetition. Reduce the linter library interface
to three functions: `Check()`, `New()`, and `Linter.Check()` by making
all helper functions private. Refactor the top-level `Check()` method to
rely on `New()` and `Linter.Check()` behind the scenes. Finally, create
a new helper method for creating a lint issuer certificate, call this
new method from `New()`, and store the result in the `Linter` struct.
Part of #5480
Aaron Gable
Jacob Hoffman-Andrews
Samantha Frank
Eng Zer Jun
Phil Porada
Matthew McPherrin
cui fliter
Matthew McPherrin