Deprecate the IgnoreAccountContacts feature flag. This causes the SA to
never query the contact column when reading registrations from the
database, and to never write a value for the contact column when
creating a new registration.
This requires updating or disabling several tests. These tests could be
deleted now, but I felt it was more appropriate for them to be fully
deleted when their corresponding services (e.g. expiration-mailer) are
also deleted.
Fixes https://github.com/letsencrypt/boulder/issues/8176
Remove `ToDNSSlice`, `FromProtoWithDefault`, and
`FromProtoSliceWithDefault` now that all their callers are gone. All
protobufs but one have migrated from DnsNames to Identifiers.
Remove TODOs for the exception, `ValidationRecord`, where an identifier
type isn't appropriate and it really only needs a string.
Rename `corepb.ValidationRecord.DnsName` to `Hostname` for clarity, to
match the corresponding PB's field name.
Improve various comments and docs re: IP address identifiers.
Depends on #8221 (which removes the last callers)
Fixes#8023
Change most functions in `ratelimits` to use full ACMEIdentifier(s) as
arguments, instead of using their values as strings. This makes the
plumbing from other packages more consistent, and allows us to:
Rename `FQDNsToETLDsPlusOne` to `coveringIdentifiers` and handle IP
identifiers, parsing IPv6 addresses into their covering /64 prefixes for
CertificatesPerDomain[PerAccount] bucket keys.
Port improved IP/CIDR validation logic to NewRegistrationsPerIPAddress &
PerIPv6Range.
Rename `domain` parts of bucket keys to either `identValue` or
`domainOrCIDR`.
Rename other internal functions to clarify that they now handle
identifier values, not just domains.
Add the new reserved IPv6 address range from RFC 9780.
For deployability, don't (yet) rename rate limits themselves; and
because it remains the name of the database table, preserve the term
`fqdnSets`.
Fixes#8223
Part of #7311
All of the identifiers being passed into the bucket construction helpers
have already passed through policy.WellFormedIdentifiers in the WFE. We
can trust that function, and our own ability to construct bucket keys,
to reduce the amount of revalidation we do before sending bucket keys to
redis.
The validateIdForName function is still used to validate override bucket
keys loaded from yaml.
Add `IdentifierTypes` to validation profiles' config, defaulting to DNS
if not set.
In `NewOrder`, check that the order's profile permits each identifier's
type.
Fixes#8137
Depends on #8173
Add `pa.validIP` to test IP address validity & absence from IANA
reservations.
Modify `pa.WillingToIssue` and `pa.WellFormedIdentifiers` to support IP
address identifiers.
Add a map of allowed identifier types to the `pa` config.
Part of #8137
In `vapb.IsCAAValidRequest`, even though CAA is only for DNS names,
deprecate `Domain` in favour of `Identifier` for consistency.
In `va.DoCAA`, reject attempts to validate CAA for non-DNS identifiers.
Rename `identifier` to `ident` inside some VA functions, also for
consistency.
In `ra.checkDCVAndCAA` & `ra.checkAuthorizationsCAA`, bypass CAA checks
for IP address identifiers.
Part of #7995
In `ra.NewOrder`, improve safety of authz reuse logic by making it
explicit that only DNS identifiers might be wildcards. Also, now that
the conditional statements need to be more complicated, collapse them
for brevity.
In `vapb.PerformValidationRequest`, remove `DnsName`.
In `ra.PerformValidation`, pass an `Identifier` instead of a `DnsName`.
In `ra.RevokeCertByApplicant`, check that the requester controls
identifiers of all types (not just DNS).
Fixes#7995 (the RA now fully supports IP address identifiers, except
for rate limits)
Fixes#7647
Part of #8023
Simplify the way we load and handle CT logs: rather than keeping them
grouped by operator, simply keep a flat list and annotate each log with
its operator's name. At submission time, instead of shuffling operator
groups and submitting to one log from each group, shuffle the whole set
of individual logs.
Support tiled logs by similarly annotating each log with whether it is
tiled or not.
Also make the way we know when to stop getting SCTs more robust.
Previously we would stop as soon as we had two, since we knew that they
would be from different operator groups and didn't care about tiled
logs. Instead, introduce an explicit CT policy compliance evaluation
function which tells us if the set of SCTs we have so far forms a
compliant set.
This is not our desired end-state for CT log submission. Ideally we'd
like to: simplify things even further (don't race all the logs, simply
try to submit to two at a time), improve selection (intelligently pick
the next log to submit to, rather than just a random shuffle), and
fine-tune latency (tiled logs should have longer timeouts than classic
ones). Those improvements will come in future PRs.
Part of https://github.com/letsencrypt/boulder/issues/7872
Instead, simply return DER bytes from `issuePrecertificate`, and accept
regular parameters to `issueCertificateForPrecertificate` (instead of a
proto message).
Also, move the lookup of the certificate profile up to
`IssueCertificate`, and pass the selected `*certProfileWithId` to both
`issuePrecertificate` and `issueCertificateForPrecertificate`.
Also, change `issueCertificateForPrecertificate` to just return DER, not
a `*corepb.Certificate` (of which most fields were already being
ignored).
Remove the deprecated `DnsNames` field from the `corepb.Order` proto
message. All users of this struct use `Identifiers` instead.
This unblocks future changes that will require `Order` users to handle
different identifier types.
Part of #7311
Remove the deprecated `DnsName` field from the core `Authorization`
struct. All users of this struct use `Identifier` instead.
This unblocks future changes that will require `Authorization` users to
handle different identifier types.
Part of #7311
For explicitly sharded certificates, CRL status is read from the
`revokedCertificates` table. This table gets written at revocation time.
At re-revocation time (for key compromise), it only gets written by the
SA if the caller passes a nonzero ShardIdx to UpdateRevokedCertificate.
The RA was never passing a nonzero ShardIdx to UpdateRevokedCertificate.
Remove the .Error() method from probs.ProblemDetails, so that it can no
longer be returned from functions which return an error. Update various
call sites to use the .String() method to get a textual representation
of the problem instead. Simplify ProblemDetailsForError to not
special-case and pass-through ProblemDetails, since they are no longer a
valid input to that function.
This reduces instances of "boxed nil" bugs, and paves the way for all of
the WFE methods to be refactored to simply return errors instead of
writing them directly into the response object.
Part of https://github.com/letsencrypt/boulder/issues/4980
Add `identifier` fields, which will soon replace the `dnsName` fields,
to:
- `corepb.Authorization`
- `corepb.Order`
- `rapb.NewOrderRequest`
- `sapb.CountFQDNSetsRequest`
- `sapb.CountInvalidAuthorizationsRequest`
- `sapb.FQDNSetExistsRequest`
- `sapb.GetAuthorizationsRequest`
- `sapb.GetOrderForNamesRequest`
- `sapb.GetValidAuthorizationsRequest`
- `sapb.NewOrderRequest`
Populate these `identifier` fields in every function that creates
instances of these structs.
Use these `identifier` fields instead of `dnsName` fields (at least
preferentially) in every function that uses these structs. When crossing
component boundaries, don't assume they'll be present, for
deployability's sake.
Deployability note: Mismatched `cert-checker` and `sa` versions will be
incompatible because of a type change in the arguments to
`sa.SelectAuthzsMatchingIssuance`.
Part of #7311
Update the SA to re-query the database for the updated account after
deactivating it, and return this to the RA. Update the RA to pass this
value through to the WFE. Update the WFE to return this value, rather
than locally modifying the pre-deactivation account object, if it gets
one (for deployability).
Also remove the RA's requirement that the request object specify its
current status so that the request can be trimmed down to just an ID.
This proto change is backwards-compatible because the new
DeactivateRegistrationRequest's registrationID field has the same type
(int64) and field number (1) as corepb.Registration's id field.
Part of https://github.com/letsencrypt/boulder/issues/5554
Remove some RA tests that were checking for errors specific to the split
issuance flow. Make one of the tests test GetSCTs directly, which makes
for a much nicer test!
Add MaxNames to the set of things that can be configured on a
per-profile basis. Remove all references to the RA's global maxNames,
replacing them with reference's to the current profile's maxNames. Add
code to the RA's main() to copy a globally-configured MaxNames into each
profile, for deployability.
Also remove any understanding of MaxNames from the WFE, as it is
redundant with the RA and is not configured in staging or prod. Instead,
hardcode the upper limit of 100 into the ratelimit package itself.
Fixes https://github.com/letsencrypt/boulder/issues/7993
Add a new RPC to the CA: `IssueCertificate` covers issuance of both the
precertificate and the final certificate. In between, it calls out to
the RA's new method `GetSCTs`.
The RA calls the new `CA.IssueCertificate` if the `UnsplitIssuance`
feature flag is true.
The RA had a metric that counted certificates by profile name and hash.
Since the RA doesn't receive a profile hash in the new flow, simply
record the total number of issuances.
Fixes https://github.com/letsencrypt/boulder/issues/7983
Remove the RA's deprecated top-level config keys which used to control
order and authz lifetimes. Make the new profile-based config keys which
replaced them required.
Since configuring a profile and default profile name is now mandatory,
always supply a profile name to the CA when requesting issuance.
Fixes https://github.com/letsencrypt/boulder/issues/7986
If validation profiles haven't been explicitly configured, use the
default profile for all incoming requests regardless of which profile
they specify.
Fixes https://github.com/letsencrypt/boulder/issues/7605
Add three new fields to the ra.ValidationProfile structure, representing
the profile's pending authorization lifetime (used to assign an
expiration when a new authz is created), valid authorization lifetime
(used to assign an expiration when an authz is successfully validated),
and order lifetime (used to assign an expiration when a new order is
created). Remove the prior top-level fields which controlled these
values across all orders.
Add a "defaultProfileName" field to the RA as well, to facilitate
looking up a default set of lifetimes when the order doesn't specify a
profile. If this default name is explicitly configured, always provide
it to the CA when requesting issuance, so we don't have to duplicate the
default between the two services.
Modify the RA's config struct in a corresponding way: add three new
fields to the ValidationProfiles structure, and deprecate the three old
top-level fields. Also upgrade the ra.NewValidationProfile constructor
to handle these new fields, including doing validation on their values.
Fixes https://github.com/letsencrypt/boulder/issues/7605
In the RA, inspect the profile of all authorizations returned when
looking for authz reuse, and refuse to reuse any whose profile doesn't
match the requested profile of the current NewOrder request.
Fixes https://github.com/letsencrypt/boulder/issues/7949
Improve the test of the `ra.validationProfiles` field by providing a
constructed `map[string]*ValidationProfile` instead of constructing one
inside the test. Much like how this data is provided, or `nil`, in calls
to `ra.NewRegistrationAuthorityImpl()`.
Add support in the RA for an allow list of accounts permitted to request
certificates containing the OCSP Must-Staple extension. If no allow list
is configured, all accounts are permitted. When a list is provided,
Finalize requests with Must-Staple are rejected unless the account is on
the list, and metrics are updated to track allowed and denied requests.
Fixes#7914
In RA.RevokedCertificate, if the certificate being revoked has a
crlDistributionPoints extension, parse the URL and pass the appropriate
shard to the SA.
This required some changes to the `admin` tool. When a malformed
certificate is revoked, we don't have a parsed copy of the certificate
to extract a CRL URL from. So, specifically when a malformed certificate
is being revoked, allow specifying a CRL shard. Because different
certificates will have different shards, require one-at-a-time
revocation for malformed certificates.
To support that refactoring, move the serial-cleaning functionality
earlier in the `admin` tool's flow.
Also, split out one of the cases handled by the `revokeCertificate`
helper in the RA. For admin malformed revocations, we need to accept a
human-specified ShardIdx, so call the SA directly in that case (and skip
stat increment since admin revocations aren't useful for metrics). This
allows `revokeCertificate` to be a more helpful helper, by extracting
serial, issuer ID, and CRL shard automatically from an
`*x509.Certificate`.
Note: we don't yet issue certificates with the crlDistributionPoints
extension, so this code will not be active until we start doing so.
Part of #7094.
Part of #7920
There will be a followup removing the remaining places that set
`contactsPresent`.
---------
Co-authored-by: Jacob Hoffman-Andrews <jsha+github@letsencrypt.org>
This is the final stage of #5554: removing the old, combined
`UpdateRegistration` flow, which has been replaced by
`UpdateRegistrationContact` and `UpdateRegistrationKey`. Those new
functions have their own tests.
The RA's `UpdateRegistration` function no longer has any callers (as of
#7827's deployment), so it is safely deployable to remove it from the SA
too, and its request from gRPC.
Fixes#5554
---------
Co-authored-by: Jacob Hoffman-Andrews <jsha+github@letsencrypt.org>
Co-authored-by: Aaron Gable <aaron@letsencrypt.org>
Remove code using `certificatesPerName` & `newOrdersRL` tables.
Deprecate `DisableLegacyLimitWrites` & `UseKvLimitsForNewOrder` flags.
Remove legacy `ratelimit` package.
Delete these RA test cases:
- `TestAuthzFailedRateLimitingNewOrder` (rl:
`FailedAuthorizationsPerDomainPerAccount`)
- `TestCheckCertificatesPerNameLimit` (rl: `CertificatesPerDomain`)
- `TestCheckExactCertificateLimit` (rl: `CertificatesPerFQDNSet`)
- `TestExactPublicSuffixCertLimit` (rl: `CertificatesPerDomain`)
Rate limits in NewOrder are now enforced by the WFE, starting here:
5a9b4c4b18/wfe2/wfe.go (L781)
We collect a batch of transactions to check limits, check them all at
once, go through and find which one(s) failed, and serve the failure
with the Retry-After that's furthest in the future. All this code
doesn't really need to be tested again; what needs to be tested is that
we're returning the correct failure. That code is
`NewOrderLimitTransactions`, and the `ratelimits` package's tests cover
this.
The public suffix handling behavior is tested by
`TestFQDNsToETLDsPlusOne`:
5a9b4c4b18/ratelimits/utilities_test.go (L9)
Some other RA rate limit tests were deleted earlier, in #7869.
Part of #7671.
And in the RA, log the notBefore of the previous issuance.
To make this happen, I had to hoist the "check for previous certificate"
up a level into `issueCertificateOuter`. That meant I also had to hoist
the "split off a WithoutCancel context" logic all the way up to
`FinalizeOrder`.
Add a new `ratelimits.NewTransactionBuilderWithLimits` constructor which
takes pre-populated rate limit data, instead of filenames for reading it
off disk.
Use this new constructor to change rate limits during RA tests, instead
of using extra `testdata` files.
Fix ARI renewals' exception from rate limits: consider `isARIRenewal` as
part of the `isRenewal` arg to `checkNewOrderLimits`.
Remove obsolete RA tests for rate limits that are now only checked in
the WFE.
Update remaining new order rate limit tests from deprecated `ratelimit`s
to new Redis `ratelimits`.
Today, we have VA.PerformValidation, a method called by the RA at
challenge time to perform DCV and check CAA. We also have VA.IsCAAValid,
a method invoked by the RA at finalize time when a CAA re-check is
necessary. Both of these methods can be executed on remote VA
perspectives by calling the generic VA.performRemoteValidation.
This change splits VA.PerformValidation into VA.DoDCV and VA.DoCAA,
which are both called on remote VA perspectives by calling the generic
VA.doRemoteOperation. VA.DoDCV, VA.DoCAA, and VA.doRemoteOperation
fulfill the requirements of SC-067 V3: Require Multi-Perspective
Issuance Corroboration by:
- Requiring at least three distinct perspectives, as outlined in the
"Phased Implementation Timeline" in BRs section 3.2.2.9 ("Effective
March 15, 2025").
- Ensuring that the number of non-corroborating (failing) perspectives
remains below the threshold defined by the "Table: Quorum Requirements"
in BRs section 3.2.2.9.
- Ensuring that corroborating (passing) perspectives reside in at least
2 distinct Regional Internet Registries (RIRs) per the "Phased
Implementation Timeline" in BRs section 3.2.2.9 ("Effective March 15,
2026").
- Including an MPIC summary consisting of: passing perspectives, failing
perspectives, passing RIRs, and a quorum met for issuance (e.g., 2/3 or
3/3) in each validation audit log event, per BRs Section 5.4.1,
Requirement 2.8.
When the new SeparateDCVAndCAAChecks feature flag is enabled on the RA,
calls to VA.IsCAAValid (during finalization) and VA.PerformValidation
(during challenge) are replaced with calls to VA.DoCAA and a sequence of
VA.DoDCV followed by VA.DoCAA, respectively.
Fixes#7612Fixes#7614Fixes#7615Fixes#7616
Pending authz reuse is a nice-to-have feature because it allows us to
create fewer rows in the authz database table when creating new orders.
However, stats show that less than 2% of authorizations that we attach
to new orders are reused pending authzs. And as we move towards using a
more streamlined database schema to store our orders, authorizations,
and validation attempts, disabling pending authz reuse will greatly
simplify our database schema and code.
CPS Compliance Review: our CPS does not speak to whether or not we reuse
pending authorizations for new orders.
IN-10859 tracks enabling this flag in prod
Part of https://github.com/letsencrypt/boulder/issues/7715
Greatly simplify the two RA unit tests covering failed validations and
account+identifier pausing. Most importantly, directly manipulate the
ratelimit backing store during test setup, to avoid having to "perform"
extra validations.
Fixes https://github.com/letsencrypt/boulder/issues/7812
The purpose of these RA and WFE unit tests is to test how they deal with
certain rate limit conditions, not to test talking to an actual redis
instance. Streamline the tests by having them talk to an in-memory rate
limits store, rather than a redis-backed one.
- Make performRemoteValidation a more generic function that returns a
new remoteResult interface
- Modify the return value of IsCAAValid and PerformValidation to satisfy
the remoteResult interface
- Include compile time checks and tests that pass an arbitrary operation
When a client deactivates a pending authorization, count that towards
their FailedAuthorizationsPerDomainPerAccount and
FailedAuthorizationsForPausingPerDomainPerAccount rate limits. This
should help curb the few clients which constantly create new orders and
authzs, deactivate those pending authzs preventing reuse of them or
their orders, and then rinse and repeat.
Fixes https://github.com/letsencrypt/boulder/issues/7834
TestPerformValidation_FailedThenSuccessfulValidationResetsPauseIdentifiersRatelimit
checks for a bucket being empty after a reset. However, that bucket is
based on an account ID that is shared across multiple test cases.
Instead, use a unique account and domain for this test.
Fixes#7812
- Added a new key-value ratelimit
`FailedAuthorizationsForPausingPerDomainPerAccount` which is incremented
each time a client fails a validation.
- As long as capacity exists in the bucket, a successful validation
attempt will reset the bucket back to full capacity.
- Upon exhausting bucket capacity, the RA will send a gRPC to the SA to
pause the `account:identifier`. Further validation attempts will be
rejected by the [WFE](https://github.com/letsencrypt/boulder/pull/7599).
- Added a new feature flag, `AutomaticallyPauseZombieClients`, which
enables automatic pausing of zombie clients in the RA.
- Added a new RA metric `paused_pairs{"paused":[bool],
"repaused":[bool], "grace":[bool]}` to monitor use of this new
functionality.
- Updated `ra_test.go` `initAuthorities` to allow accessing the
`*ratelimits.RedisSource` for checking that the new ratelimit functions
as intended.
Co-authored-by: @pgporada
Fixes https://github.com/letsencrypt/boulder/issues/7738
---------
Co-authored-by: Phil Porada <pporada@letsencrypt.org>
Co-authored-by: Phil Porada <philporada@gmail.com>
Aaron Gable
James Renken
Samantha Frank
Jacob Hoffman-Andrews
Kruti Sutaria