Named field `DB`, in a each component configuration struct, acts as the
receiver for the value of `db` when component JSON files are
unmarshalled.
When `cmd.DBConfig` fields are received at the root of component
configuration struct instead of `DB` copy them to the `DB` field of the
component configuration struct.
Move existing `cmd.DBConfig` values from the root of each component's
JSON configuration in `test/config-next` to `db`
Part of #5275
`cai.Stop()` called from boulder-ca could potentially exit before errors
emitted by `caSrv` and `ocspSrv` are logged. This could lead to
boulder-ca erroneously exiting `0` instead of `1`.
Add a `sync.WaitGroup`. Increment the waitgroup before `caSrv.Serve()`
and `ocspSrv.Serv()` are spun off. Await the waitgroup before
`cai.Stop()` is called.
Fixes#5246
Publisher currently loads a PEM formatted certificate bundle from file
using LoadCertBundle a utility function in the core package.
LoadCertBundle parses the PEM file to a slice of x509.Certificates and
returns them to boulder-publisher (without checking validity). Using
these x509 Certificates, boulder-publisher to construct an ASN1Cert
bundle. This bundle is passed to each new publisher instance. When
publisher receives a request it unconditionally appends this bundle to
each end-entity precertificate for submission to CT logs.
This change augments this process to add support for multiple issuers
using the IssuerNameID concept in the Issuance package. Config field
Common.CT.CertificateBundleFilename has been replaced with the Chains
field. LoadChain, a utility function added in PR #5271, loads and
validates the chain (which nets us some added deploy-time safety) before
returning it to boulder-publisher. Using these x509 Certificates,
boulder-publisher constructs a mapping of IssuerNameID to ASN1Cert
bundle and passes this to each new publisher instance. When publisher
receives a request it determines the IssuerNameID of the precertificate
to select and append the correct ASN1Cert bundle for a given Issuer.
A followup issue #5269 has been created to address removal of the Common
field from the publisher configuration and code has been commented with
TODOs where code will need to be removed or refactored.
Fixes#1669
In #5235 we replaced MaxDBConns in favor of MaxOpenConns.
One week ago MaxDBConns was removed from all dev, staging, and
production configurations. This change completes the removal of
MaxDBConns from all components and test/config.
Fixes#5249
loadChain is an unexported utility function recently added to
boulder-wfe to support the loading and validating of PEM files that
represent a certificate chain
This change moves the core loadChain functionality out of boulder-wfe to
a new exported LoadChain function in the Issuance package. All
boulder-wfe unit tests have been preserved and most of them have been
pared down and added to the Issuance package as well.
Blocks #1669Fixes#5270
Currently, the CA is configured with a set of `internalIssuer`s,
and a mapping of public key algorithms (e.g. `x509.RSA`) to which
internalIssuer to use. In operation today, we use the same issuer
for all kinds of public key algorithms. In the future, we will use
different issuers for different algorithms (in particular, we will
use R3 to issue for RSA keys, and E1 to issue for ECDSA keys). But
we want to roll that out slowly, continuing to use our RSA issuer
to issue for all types of public keys, except for ECDSA keys which
are presented by a specific set of allowed accounts.
This change adds a new config field to the CA, which lets us specify
a small list of registration IDs which are allowed to have issuance
from our ECDSA issuer. If the config list is empty, then all accounts
are allowed. The CA checks to see if the key being issued for is
ECDSA: if it is, it then checks to make sure that the associated
registration ID is in the allowlist. If the account is not allowed,
it then overrides the issuance algorithm to use RSA instead,
mimicking our old behavior. It also adds a new feature flag, which
can be enabled to skip the allowlist entirely (effectively allowing
all registered accounts). This feature flag will be enabled when
we're done with our testing and confident in our ECDSA issuance.
Fixes#5259
[Go style says](https://blog.golang.org/package-names):
> Avoid stutter. Since client code uses the package name as a prefix
> when referring to the package contents, the names for those contents
> need not repeat the package name. The HTTP server provided by the
> http package is called Server, not HTTPServer. Client code refers to
> this type as http.Server, so there is no ambiguity.
Rename DNSClient, DNSClientImpl, NewDNSClientImpl,
NewTestDNSClientImpl, DNSError, and MockDNSClient to follow those
guidelines.
Unexport DNSClientImpl and MockTimeoutError (was only used internally).
Make New and NewTest return the Client interface rather than a concrete
`impl` type.
This change simplifies and hardens the wfe2's support for having
multiple issuers, and multiple chains for each issuer, configured
and loaded in memory.
The only config-visible change is replacing the old two separate config
values (`certificateChains` and `alternateCertificateChains`) with a
single value (`chains`). This new value does not require the user to
know and hand-code the AIA URLs at which the certificates are available;
instead the chains are simply presented as lists of files. If this new
config value is present, the old config values will be ignored; if it
is not, the old config values will be respected.
Behind the scenes, the chain loading code has been completely changed.
Instead of loading PEM bytes directly from the file, and then asserting
various things (line endings, no trailing bits, etc) about those bytes,
we now parse a certificate from the file, and in-memory recreate the
PEM from that certificate. This approach allows the file loading to be
much more forgiving, while also being stricter: we now check that each
certificate in the chain is correctly signed by the next cert, and that
the last cert in the chain is a self-signed root.
Within the WFE itself, most of the internal structure has been retained.
However, both the internal `issuerCertificates` (used for checking
that certs we are asked to revoke were in fact issued by us) and the
`certificateChains` (used to append chains to end-entity certs when
served to clients) have been updated to be maps keyed by IssuerNameID.
This allows revocation checking to not have to iterate through the
whole list of issuers, and also makes it easy to double-check that
the signatures on end-entity certs are valid before serving them. Actual
checking of the validity will come in a follow-up change, due to the
invasive nature of the necessary test changes.
Fixes#5164
Historically the only database/sql driver setting exposed via JSON
config was maxDBConns. This change adds support for maxIdleConns,
connMaxLifetime, connMaxIdleTime, and renames maxDBConns to
maxOpenConns. The addition of these settings will give our SRE team a
convenient method for tuning the reuse/closure of database connections.
A new struct, DBSettings, has been added to SA. The struct, and each of
it's fields has been commented.
All new fields have been plumbed through to the relevant Boulder
components and exported as Prometheus metrics. Tests have been
added/modified to ensure that the fields are being set. There should be
no loss in coverage
Deployability concerns for the migration from maxDBConns to maxOpenConns
have been addressed with the temporary addition of the helper method
cmd.DBConfig.GetMaxOpenConns(). This method can be removed once
test/config is defaulted to using maxOpenConns. Relevant sections of the
code have TODOs added that link back to an newly opened issue.
Fixes#5199
This adds a new component to the CA, ocspLogQueue, which batches up
OCSP generation events for audit logging. It will log accumulated
events when it reaches a certain line length, or when a maximum amount
of times has passed.
The /issuer-cert endpoint was a holdover from the v1 API, where
it is a critical part of the issuance flow. In the v2 issuance
flow, the issuer certificate is provided directly in the response
for the certificate itself. Thus, this endpoint is redundant.
Stats show that it receives approximately zero traffic (less than
one request per week, all of which are now coming from wget or
browser useragents). It also complicates the refactoring necessary
for the v2 API to support multiple issuers.
As such, it is a safe and easy decision to remove it.
Fixes#5196
Previously, configuration of the boulder-janitor was split into
two places: the actual json config file (which controlled which
jobs would be enabled, and what their rate limits should be), and
the janitor code itself (which controlled which tables and columns
those jobs should query). This resulted in significant duplicated
code, as most of the jobs were identical except for their table
and column names.
This change abstracts away the query which jobs use to find work.
Instead of having each job type parse its own config and produce
its own work query (in Go code), now each job supplies just a few
key values (the table name and two column names) in its JSON config,
and the Go code assembles the appropriate query from there. We are
able to delete all of the files defining individual job types, and
replace them with a single slightly smarter job constructor.
This enables further refactorings, namely:
* Moving all of the logic code into its own module;
* Ensuring that the exported interface of that module is safe (i.e.
that a client cannot create and run jobs without them being valid,
because the only exposed methods ensure validity);
* Collapsing validity checks into a single location;
* Various renamings.
The vast majority of Boulder components no longer care about
anything in the common config block. As such, we hope to
remove it entirely in the near future. So let's put the (not-yet-used)
IssuerCerts config item in the main OCSPResponder block,
rather than in the common block.
Part of #5204
It is possible for a CertificateStatus row to have a nil IssuerID
(there was a period of time in which we didn't write IssuerIDs into
CertificateStatus rows at all) but all such rows should be old and
therefore expired.
Unfortunately, this code was checking the IssuerID before it was
checking the Expiry, and therefore was generating panics when trying
to dereference a nil pointer.
This change simply moves the IssuerID check to be after the Expires
check, so that we'll only try to dereference the IssuerID on recent
CertificateStatus rows, where it is guaranteed to be non-nil.
Fixes#5200
This adds a new job type (and corresponding config) to the janitor
to clean up old rows from the `keyHashToSerial` table. Rows in this
table are no longer relevant after their corresponding certificate
has expired.
Fixes#4792
When making an OCSP request, the client provides three pieces of
information: the URL which it is querying to get OCSP info, the
hash of the issuer public which issued the cert in question, and
the serial number of the cert in question. In Boulder, the first
of these is only provided implicitly, based on which instance of
ocsp-responder is handling the request: we ensure (via configs)
that the ocsp-responder handling a given OCSP AIA URL has the
corresponding issuer cert loaded in memory.
When handling a request, the ocsp-responder checks three things:
that the request is using SHA1 to hash the issuer public key, that
the requested issuer public key matches one of the loaded issuer
certs, and that the requested serial number is one we could have
issued (i.e. has the correct prefix). It relies on the database
query to filter out requests for non-existent serials.
However, this means that a request to an ocsp-responder instance
with issuer cert A loaded could receive and handle a request which
specifies cert A as the issuer, but names a serial which was actually
issued by issuer cert B. The checks all pass and the database lookup
succeeds. But the returned OCSP response is for a certificate that
was issued by a different issuer, and the response itself was
signed by that other issuer.
In order to resolve this potentially confusing situation, this change
adds one additional check to the ocsp-responder: after it has
retrieved the ocsp response, it looks up which issuer produced that
ocsp response, confirms that it has that issuer cert loaded in
memory, and confirms that its issuer key hash matches that in the
original request.
There is still one wrinkle if issuer certs A and B are both loaded
in one ocsp-responder, and that one ocsp-responder is handling OCSP
requests to both of their AIA OCSP URLs. In this case, it is possible
that a request to a.ocsp.com, but requesting OCSP for a cert issued
by B, could still have its request answered. This is because the
ocsp-responder itself does not know which URL was requested. But
regardless, this change does guarantee that the response will match
the contents of the request (or no response will be given), no matter
what URL that request was sent to.
Fixes#5182
In all boulder services, we construct a single tls.Config object
and then pass it into multiple gRPC setup methods. In all boulder
services but one, we pass the object into multiple clients, and
just one server. In general, this is safe, because all of the client
setup happens on the main thread, and the server setup similarly
happens on the main thread before spinning off the gRPC server
goroutine.
In the CA, we do the above and pass the tlsConfig object into two
gRPC server setup functions. Thus the first server goroutine races
with the setup of the second server.
This change removes the post-hoc assignment of MinVersion,
MaxVersion, and CipherSuites of the tls.Config object passed
to grpc.ClientSetup and grpc.NewServer. And adds those same
values to the cmd.TLSConfig.Load, the method responsible for
constructing the tls.Config object before it's passed to
grpc.ClientSetup and grpc.NewServer.
Part of #5159
Having both of these very similar methods sitting around
only serves to increase confusion. This removes the last
few places which use `cmd.LoadCert` and replaces them
with `core.LoadCert`, and deletes the method itself.
Fixes#5163
The ocsp-responder takes a path to a certificate file as one of
its config values. It uses this path as one of the inputs when
constructing its DBSource, the object responsible for querying
the database for pregenerated OCSP responses to fulfill requests.
However, this certificate file is not necessary to query the
database; rather, it only acts as a filter: OCSP requests whose
IssuerKeyHash do not match the hash of the loaded certificate are
rejected outright, without querying the DB. In addition, there is
currently only support for a single certificate file in the config.
This change adds support for multiple issuer certificate files in
the config, and refactors the pre-database filtering of bad OCSP
requests into a helper object dedicated solely to that purpose.
Fixes#5119
The RA is responsible for contacting Akamai to purge cached OCSP
responses when a certificate is revoked and fresh OCSP responses need to
be served ASAP. In order to do so, it needs to construct the same OCSP
URLs that clients would construct, and that Akamai would cache. In order
to do that, it needs access to the issuing certificate to compute a hash
across its Subject Info and Public Key.
Currently, the RA holds a single issuer certificate in memory, and uses
that cert to compute all OCSP URLs, on the assumption that all certs
we're being asked to revoke were issued by the same issuer.
In order to support issuance from multiple intermediates at the same
time (e.g. RSA and ECDSA), and to support rollover between different
issuers of the same type (we may need to revoke certs issued by two
different issuers for the 90 days in which their end-entity certs
overlap), this commit changes the configuration to provide a list of
issuer certificates instead.
In order to support efficient lookup of issuer certs, this change also
introduces a new concept, the Chain ID. The Chain ID is a truncated hash
across the raw bytes of either the Issuer Info or the Subject Info of a
given cert. As such, it can be used to confirm issuer/subject
relationships between certificates. In the future, this may be a
replacement for our current IssuerID (a truncated hash over the whole
issuer certificate), but for now it is used to map revoked certs to
their issuers inside the RA.
Part of #5120
This change adds two new test assertion helpers, `AssertErrorIs`
and `AssertErrorWraps`. The former is a wrapper around `errors.Is`,
and asserts that the error's wrapping chain contains a specific (i.e.
singleton) error. The latter is a wrapper around `errors.As`, and
asserts that the error's wrapping chain contains any error which is
of the given type; it also has the same unwrapping side effect as
`errors.As`, which can be useful for further assertions about the
contents of the error.
It also makes two small changes to our `berrors` package, namely
making `berrors.ErrorType` itself an error rather than just an int,
and giving `berrors.BoulderError` an `Unwrap()` method which
exposes that inner `ErrorType`. This allows us to use the two new
helpers above to make assertions about berrors, rather than
having to hand-roll equality assertions about their types.
Finally, it takes advantage of the two changes above to greatly
simplify many of the assertions in our tests, removing conditional
checks and replacing them with simple assertions.
The akamai purger is now a standalone service, and the ocsp-updater does
not talk to it directly. All of the code in the ocsp-updater related to
the akami purger is gated behind tests that the corresponding config
values are not nil, and the resulting objects are never used by the
service's business logic.
Now that all of the corresponding config stanzas have been removed
from our production configs, we can remove them from the config
struct definitions as well.
errors.As checks for a specific error in a wrapped error chain
(see https://golang.org/pkg/errors/#As) as opposed to asserting
that an error is of a specific type.
Part of #5010
errors.As checks for a specific error in a wrapped error chain
(see https://golang.org/pkg/errors/#As) as opposed to asserting
that an error is of a specific type.
Part of #5010
This was already part done: There is an ID() method in issuance. This
change extends that by:
- Defining a type alias indicating something is an IssuerID.
- Defining issuance.Certificate, which also has an ID() method,
so that components that aren't the CA can load certificates and
use the type system to mark them as issuers (and get their IDs).
- Converting akamai-purger and ca to use the new types.
- Removing idForIssuer from ca.go.
This health service implements the gRPC Health Checking
Protocol, as defined in
https://github.com/grpc/grpc/blob/master/doc/health-checking.md
and as implemented by the gRPC authors in
https://pkg.go.dev/google.golang.org/grpc/health@v1.29.0
It simply instantiates a health service, and attaches it to the same
gRPC server that is handling requests to the primary (e.g. CA) service.
When the main service would be shut down (e.g. because it caught a
signal), it also sets the status of the service to NOT_SERVING.
This change also imports the health client into our grpc client,
ensuring that all of our grpc clients use the health service to inform
their load-balancing behavior.
This will be used to replace our current usage of polling the debug
port to determine whether a given service is up and running. It may
also be useful for more comprehensive checks and blackbox probing
in the future.
Part of #5074
The IssuerID shouldn't ever be 0 (it should always be NULL/nil or
an actual value), but we recently had an incident in which it was
being set to 0 instead of NULL. This ensures that functionality
will continue as intended even in the face of that circumstance.
Fixes#5098
This change adds `req.IssuerID` to the set of fields that the SA's
`AddPrecertificate` method requires be non-zero.
As a result, this also updates many tests, both unit and integration,
to ensure that they supply a value (usually just 1) for that field. The
most complex part of the test changes is a slight refactoring to the
orphan-finder code, which makes it easier to reason about the
separation between log line parsing and building and sending the
request.
Based on #5096Fixes#5097
One of the log lines describes the most frequent address corresponding
to a number of accounts, but it actually corresponds to a number of
lines in the input CSV.
Also, now that we escape newlines in log output, the dryRunMailer's
output looks messed up. Split the message body into lines and emit one
log message per line.
Generated keys have the same label on both the private and public key
objects. When looking up keys for signing, the label is used to find the
public key.
The CA is the only service which still defines its json config format
in the package itself, rather than in its corresponding boulder-ca cmd
package. This has allowed the CA's constructor interface to hide
arbitrary complexity inside its first argument, the whole config blob.
This change moves the CA's config to boulder-ca/main.go, to match
the other Boulder components. In the process, it makes a host of
other improvements:
It refactors the issuance package to have a cleaner configuration
interface. It also separates the config into a high-level profile (which
applies equally to all issuers), and issuer-level profiles (which apply
only to a single issuer). This does involve some code duplication,
but that will be removed when CFSSL goes away.
It adds helper functions to the issuance package to make it easier
to construct a new issuer, and takes advantage of these in the
boulder-ca package. As a result, the CA now receives fully-formed
Issuers at construction time, rather than constructing them from
nearly-complete configs during its own initialization.
It adds a Linter struct to the lint package, so that an issuer can
simply carry around a Linter, rather than a separate lint signing
key and registry of lints to run.
It makes CFSSL-specific code more clearly marked as such,
making future removal easier and cleaner.
Fixes#5070Fixes#5076
We define a "signer" to be a private key, or something that satisfies the
crypto.Signer interface. We define an "issuer" to be an object which has
both a signer (so it can sign things) and a certificate (so that the things
it signs can have appropriate issuer fields set).
As a result, this change:
- moves the new "signer" library to be called "issuance" instead
- renames several "signers" to instead be "issuers", as defined above
- renames several "issuers" to instead be "certs", to reduce confusion more
There are some further cleanups which could be made, but most of them
will be made irrelevant by the removal of the CFSSL code, so I'm leaving
them be for now.
This just changes the `loadCFSSLIssuers` signature to more closely match
the `loadBoulderIssuers` signature (it didn't need access to the whole config
object), and standardize our json on lowercase string keys.
These functions sign a random nonce with a newly-created issuance key,
in order to verify that the key was correctly generated and its public
component was correctly extracted. In general we can trust that keys are
correctly created by the HSM, and unit and integration tests can check
that we are correctly extracting public keys.
Removing these avoids the possibility of signing something that could be
construed as a "certificate, but malformed."
The ca's configuration already has support for containing multiple
issuers. However, when it comes time to actually sign a (pre)cert,
it always uses the defaultIssuer.
This change has the ca instead choose which issuer to use based
on the PublicKeyAlgorithm requested in the CSR (or, for final cert
issuances, based on the PublicKeyAlgorithm in the precert).
This will allow us to use our RSA issuers to sign certificates for
users who aren't ready to switch to ECDSA, while immediately switching
to our new ECDSA chain for subscribers who want to use it.
Fixed#5027
When issuing a new root or intermediate cert, we should take every
precaution possible to ensure that these certs are well-formed.
This change introduces a new step prior to issuing and writing a new CA
cert. We generate a new disposable private key based on the type of the
key being used in the real ceremony, then use this key to sign a fake
certificate for the sole purpose of linting. We then pass this through
the full suite of zlint's checks before proceeding with the actual
issuance.
Since this code path is largely similar to the pre-issuance linting done
by the new boulder signer tool, this change also factors it out into a
small, single-purpose `lint` package.
Fixes#5051
One slightly surprising / interesting thing: Since core types like
Order and Registration are still proto2 and have pointer fields,
there are actually some places in this PR where I had to add
a `*` rather than delete an `&`, because I was taking a pointer
field from one of those core types and passing it as a field in
an SA RPC request.
Fixes#5037.
Since we now sync caaChecks logs daily instead of continuously,
caa-log-checker can no longer assume that the validation logs it is
checking cover the exact same span of time as the issuance logs. This
commit adds -earliest and -latest parameters so that the script
that drives this tool can restrict verification to a timespan where we
know the data is valid.
Also adds a -debug flag to caa-log-checker to enable debug logs. At the
moment this makes the tool write to stderr how many issuance messages
were evaluated and how many were skipped due to -earliest and
-latest parameters.
Updates the Registration Authority to use proto3 for its
RPC methods. This turns out to be a fairly minimal change,
as many of the RA's request and response messages are
defined in core.proto, and are therefore still proto2.
Fixes#4955
Samantha