Remove the redis-tls, wfe-tls, and mail-test-srv keys which were
generated by minica and then checked in to the repo. All three are
replaced by the dynamically-generated ipki directory.
Part of https://github.com/letsencrypt/boulder/issues/7476
Rename "IssuerNameID" to just "NameID". Similarly rename the standalone
functions which compute it to better describe their function. Add a
.NameID() directly to issuance.Issuer, so that callers in other packages
don't have to directly access the .Cert member of an Issuer. Finally,
rearrange the code in issuance.go to be sensibly grouped as concerning
NameIDs, Certificates, or Issuers, rather than all mixed up between the
three.
Fixes https://github.com/letsencrypt/boulder/issues/5152
The last rows using the old-style IssuerID were written to the database
in late 2021. Those rows have long since aged out -- we no longer serve
certificates or revocation information for them -- so we can remove the
code which handles those old-style IDs. This allows for some nice
simplifications in the CA's ocspImpl and in the Issuance package, which
will be useful for further reorganization of the CA and issuance
packages.
Fixes https://github.com/letsencrypt/boulder/issues/5152
Integrate the key-value rate limits from #6947 into the WFE. Rate limits
are backed by the Redis source added in #7016, and use the SRV record
shard discovery added in #7042.
Part of #5545
Export new prometheus metrics for the `notBefore` and `notAfter` fields
to track internal certificate validity periods when calling the `Load()`
method for a `*tls.Config`. Each metric is labeled with the `serial`
field.
```
tlsconfig_notafter_seconds{serial="2152072875247971686"} 1.664821961e+09
tlsconfig_notbefore_seconds{serial="2152072875247971686"} 1.664821960e+09
```
Fixes https://github.com/letsencrypt/boulder/issues/6829
- Require `letsencrypt/validator` package.
- Add a framework for registering configuration structs and any custom
validators for each Boulder component at `init()` time.
- Add a `validate` subcommand which allows you to pass a `-component`
name and `-config` file path.
- Expose validation via exported utility functions
`cmd.LookupConfigValidator()`, `cmd.ValidateJSONConfig()` and
`cmd.ValidateYAMLConfig()`.
- Add unit test which validates all registered component configuration
structs against test configuration files.
Part of #6052
We rely on the ratelimit/ package in CI to validate our ratelimit
configurations. However, because that package relies on cmd/ just for
cmd.ConfigDuration, many additional dependencies get pulled in.
This refactors just that struct to a separate config package. This was
done using Goland's automatic refactoring tooling, which also organized
a few imports while it was touching them, keeping standard library,
internal and external dependencies grouped.
The iotuil package has been deprecated since go1.16; the various
functions it provided now exist in the os and io packages. Replace all
instances of ioutil with either io or os, as appropriate.
In practice Redis queries often complete in under 1ms, but we were using
the default Prometheus histogram buckets, the lowest of which is 5ms.
That means we don't get very detailed latency information from this
histogram. Use a smaller value (0.5ms) for the lowest bucket.
This enables ocsp-responder to talk to the RA and request freshly signed
OCSP responses.
ocsp/responder/redis_source is moved to ocsp/responder/redis/redis_source.go
and significantly modified. Instead of assuming a response is always available
in Redis, it wraps a live-signing source. When a response is not available,
it attempts a live signing.
If live signing succeeds, the Redis responder returns the result right away
and attempts to write a copy to Redis on a goroutine using a background
context.
To make things more efficient, I eliminate an unneeded ocsp.ParseResponse
from the storage path. And I factored out a FakeResponse helper to make
the unittests more manageable.
Commits should be reviewable one-by-one.
Fixes#6191
This should allow us to spot any configuration problems right away.
Otherwise, such problems may be swallowed in background tasks in the
go-redis library.
The metadata values were planned to be used for scanning Redis in
ocsp-updater. Since we won't do that, remove it. Happily, this also
allows us to get rid of shortIssuerId.
Removing the issuer check in rocsp_sa.go uncovered a "boxed nil" problem:
SA was doing a nil check against an interface field that in practice was
never nil (because it was promoted from a concrete type at construction
time). So we would always hit the ROCSP path. But one of the first steps
in that path was looking up an issuer ID. Since `test/config` never
had the issuers set, we would look up the issuer ID, not find it, and
return an error before we attempted to call storeResponse. To fix this,
I made `NewSQLStorageAuthority` take a concrete `*rocsp.WritingClient`
instead of an interface, and check for nil before assigning it to an
internal interface field.
Built on top of #6201.
The go-redis docs say default is 10 * NumCPU, but the actual code says 5.
Extra context:
2465baaab5/options.go (L143-L145)2465baaab5/cluster.go (L96-L98)
For Options, the default (documented) is 10 * NumCPUs. For ClusterOptions, the
default (undocumented) is 5 * NumCPUs. We use ClusterOptions. Also worth noting:
for ClusterOptions, the limit is per node.
Most of our existing Redis metrics are related to the connection pool,
but some sound like they could be request-oriented (like redis_hits and
redis_misses). Fix that with better naming. Also, join three related
metrics (hits, misses, and timeouts) under a single metric name with
labels so they more easily by grouped and counted.
Move the stat creation into rocsp/metrics.go so the names are in the
same file as the collector.
Add a unittest for rocsp metrics.
These new linters are almost all part of golangci-lint's collection
of default linters, that would all be running if we weren't setting
`disable-all: true`. By adding them, we now have parity with the
default configuration, as well as the additional linters we like.
Adds the following linters:
* unconvert
* deadcode
* structcheck
* typecheck
* varcheck
* wastedassign
If a Redis client is configured for the SA service, OCSP responses created
during a revocation event will be written to Redis on a best effort basis.
Use the OCSP response NextUpdate time as the expiration time for the
redis entry. Change the new issuance OCSP storage to do the same.
Fixes#5888
Adds a rocsp redis client to the sa if cluster information is provided in the
sa config. If a redis cluster is configured, all new certificate OCSP
responses added with sa.AddPrecertificate will attempt to be written to
the redis cluster, but will not block or fail on errors.
Fixes: #5871
When inside a closure, it is important to not accidentally assign
to variables declared outside the scope of the closure. Doing so
causes static analysis tools (such as `errcheck`) to be unable to
evaluate the lifetime of the variable, and unable to determine if
it is appropriately read from before being assigned to again.
Fix two instances where we assign to a variable declared in the
closure's enclosing scope, rather than declaring a new variable
with the same name.
Add `stylecheck` to our list of lints, since it got separated out from
`staticcheck`. Fix the way we configure both to be clearer and not
rely on regexes.
Additionally fix a number of easy-to-change `staticcheck` and
`stylecheck` violations, allowing us to reduce our number of ignored
checks.
Part of #5681
We can scan metadata and get the age of responses.
We can scan responses and print them in base64.
Note: this issues a GET for each key, and blocks on the result. For much
faster scanning we will want to introduce parallel GETs in a subsequent
PR.
Also, add a `get` operation to get a single entry.
Fixes#5830
Add a not found error type to rocsp.
Handle redis value not found lookup errors in the ocsp-responder different
than other redis lookup errors.
Add labels to the to ocspLookup metric and delete the source used
metric. This can now be determined based on which lookup metric
reports success.
Fixes#5833
If configured, ocsp-updater will write responses to Redis in parallel
with MariaDB, giving up if Redis is slower and incrementing a stat.
Factors out the ShortIDIssuer concept from rocsp-tool into
rocsp_config.
This scans the database for certificateStatus rows, gets them signed by the CA, and writes them to Redis.
Also, bump the default PoolSize for Redis to 100.
This is a sort of proof of concept of the Redis interaction, which will
evolve into a tool for inspection and manual repair of missing entries,
if we find ourselves needing to do that.
The important bits here are rocsp/rocsp.go and
cmd/rocsp-tool/main.go. Also, the newly-vendored Redis client.
Aaron Gable
Viktor Szépe
Samantha
dependabot[bot]
Phil Porada
Matthew McPherrin
Jacob Hoffman-Andrews
Andrew Gabbitas
Jacob Hoffman-Andrews