Commit Graph

5663 Commits

Author SHA1 Message Date
dependabot[bot] 7bb573ca94
Bump github.com/prometheus/client_golang from 1.7.1 to 1.12.1 (#5974)
Diff: https://github.com/prometheus/client_golang/compare/v1.7.1...v1.12.1

Changes:
* 1.12.1 / 2022-01-29
  - [BUGFIX] Make the Go 1.17 collector concurrency-safe
  - Use simpler locking in the Go 1.17 collector
  - [BUGFIX] Reduce granularity of histogram buckets for Go 1.17 collector
  - [ENHANCEMENT] API client: make HTTP reads more efficient
* 1.12.0 / 2022-01-19
  - [CHANGE] example/random: Move flags and metrics into main()
  - [FEATURE] API client: Support wal replay status api
  - [FEATURE] Use the runtime/metrics package for the Go collector for 1.17+
  - [ENHANCEMENT] API client: Update /api/v1/status/tsdb to include headStats
  - [ENHANCEMENT] promhttp: Check validity of method and code label values
* 1.11.0 / 2021-06-07
  - [CHANGE] Add new collectors package.
  - [CHANGE] prometheus.NewExpvarCollector is deprecated, use collectors.NewExpvarCollector instead.
  - [CHANGE] prometheus.NewGoCollector is deprecated, use collectors.NewGoCollector instead.
  - [CHANGE] prometheus.NewBuildInfoCollector is deprecated, use collectors.NewBuildInfoCollector instead.
  - [FEATURE] Add new collector for database/sql#DBStats.
  - [FEATURE] API client: Add exemplars API support.
  - [ENHANCEMENT] API client: Add newer fields to Rules API.
  - [ENHANCEMENT] API client: Add missing fields to Targets API.
* 1.10.0 / 2021-03-18
  - [CHANGE] Minimum required Go version is now 1.13.
  - [CHANGE] API client: Add matchers to LabelNames and LabesValues.
  - [FEATURE] API client: Add buildinfo call.
  - [BUGFIX] Fix build on riscv64.
* 1.9.0 / 2020-12-17
  - [FEATURE] NewPidFileFn helper to create process collectors for processes whose PID is read from a file.
  - [BUGFIX] promhttp: Prevent endless loop in InstrumentHandler... middlewares with invalid metric or label names.
* 1.8.0 / 2020-10-15
  - [CHANGE] API client: Use time.Time rather than string for timestamps in RuntimeinfoResult.
  - [FEATURE] Export MetricVec to facilitate implementation of vectors of custom Metric types.
  - [FEATURE] API client: Support /status/tsdb endpoint.
  - [ENHANCEMENT] API client: Enable GET fallback on status code 501.
  - [ENHANCEMENT] Remove Metric references after reslicing to free up more memory.

Additional transitive dependency updates:
* https://github.com/prometheus/common/compare/v0.10.0...v0.32.1
* https://github.com/prometheus/procfs/compare/v0.1.3...v0.7.3
* https://github.com/golang/appengine/compare/v1.6.5...v1.6.6
* cb27e3aa20...8632dd7979
* 0f9fa26af8...5a964db013
2022-03-30 09:49:40 -07:00
Aaron Gable dab8a71b0e
Use new RA methods from WFE revocation path (#5983)
Simplify the WFE `RevokeCertificate` API method in three ways:
- Remove most of the logic checking if the requester is authorized to
  revoke the certificate in question (based on who is making the
  request, what authorizations they have, and what reason they're
  requesting). That checking is now done by the RA. Instead, simply
  verify that the JWS is authenticated.
- Remove the hard-to-read `authorizedToRevoke` callbacks, and make the
  `revokeCertBySubscriberKey` (nee `revokeCertByKeyID`) and
  `revokeCertByCertKey` (nee `revokeCertByJWK`) helpers much more
  straight-line in their execution logic.
- Call the RA's new `RevokeCertByApplicant` and `RevokeCertByKey` gRPC
  methods, rather than the deprecated `RevokeCertificateWithReg`.

This change, without any flag flips, should be invisible to the
end-user. It will slightly change some of our log message formats.
However, by now relying on the new RA gRPC revocation methods, this
change allows us to change our revocation policies by enabling the
`AllowDoubleRevocation` and `MozRevocationReasons` feature flags, which
affect the behavior of those new helpers.

Fixes #5936
2022-03-28 14:14:11 -07:00
Jacob Hoffman-Andrews 6271a88ba6
rocsp-tool: don't log every 10th insert (#6017)
Logging every 10 is quite noisy; instead adopt the same strategy we use
for errors, and log all of them at first, fading out to fewer of them as
we get to bigger numbers.
2022-03-25 11:43:26 -07:00
Samantha 7c22b99d63
akamai-purger: Improve throughput and configuration safety (#6006)
- Add new configuration key `throughput`, a mapping which contains all
  throughput related akamai-purger settings.
- Deprecate configuration key `purgeInterval` in favor of `purgeBatchInterval` in
  the new `throughput` configuration mapping.
- When no `throughput` or `purgeInterval` is provided, the purger uses optimized
  default settings which offer 1.9x the throughput of current production settings.
- At startup, all throughput related settings are modeled to ensure that we
  don't exceed the limits imposed on us by Akamai.
- Queue is now `[][]string`, instead of `[]string`.
  - When a given queue entry is purged we know all 3 of it's URLs were purged.
  - At startup we know the size of a theoretical request to purge based on the
    number of queue entries included
- Raises the queue size from ~333-thousand cached OCSP responses to
  1.25-million, which is roughly 6 hours of work using the optimized default
  settings
- Raise `purgeInterval` in test config from 1ms, which violates API limits, to 800ms

Fixes #5984
2022-03-23 17:23:07 -07:00
Jacob Hoffman-Andrews dd8be8d7b0
Add regID to signing log messages (#6014) 2022-03-22 10:32:23 -07:00
Andrew Gabbitas 79048cffba
Support writing initial OCSP response to redis (#5958)
Adds a rocsp redis client to the sa if cluster information is provided in the
sa config. If a redis cluster is configured, all new certificate OCSP
responses added with sa.AddPrecertificate will attempt to be written to
the redis cluster, but will not block or fail on errors.

Fixes: #5871
2022-03-21 20:33:12 -06:00
Samantha 3dfe4efe5c
CI: Update make-deb.sh path in release action (#6015) 2022-03-21 16:50:52 -07:00
Aaron Gable c57b12436d
Remove update-master github workflow (#6013)
We've had this disabled for a long time now, and nothing
relies on keeping `master` up to date anymore.
2022-03-21 14:58:06 -07:00
Samantha 3e9eaf84ea
rocsp-tool: Add syslog support (#6010)
Add a logging stanza to rocsp-tool's config, and initialize a boulder
logger rather than using Go's default log facilities.

Fixes #5976
2022-03-21 14:51:56 -07:00
Jacob Hoffman-Andrews 7336f1acce
wfe: Log TLS version (#6001)
This will help inform deprecation of TLS 1.0 and TLS 1.1 for ACME API requests.
2022-03-21 14:01:52 -07:00
Jacob Hoffman-Andrews 7d00d9fbcf
Use go1.18 in CI, and fix up Docker image (#6002)
- Remove GOPATH-style path structure, which isn't needed with Go
modules.
- Remove check for existing of docker buildx builder instance, since it
was unreliable.
2022-03-21 12:24:13 -07:00
Jacob Hoffman-Andrews 42de50c7f2
Increment csrSignatureAlgorithms after finalize (#6009)
This gives us a more meaningful count: of CSR signature algorithms we
_accepted_. We aren't very interested in the count of ones we rejected.
2022-03-21 11:34:36 -07:00
Jacob Hoffman-Andrews 07cb1179d0
Add logging of "oldTLS" bit (#6008)
That causes the VA to emit ValidationRecords with the OldTLS bit set if
it observes a redirect to HTTPS that negotiates TLS < 1.2.

I've manually tested but there is not yet an integration test. I need
to make a parallel change in challtestsrv and then incorporate here.
2022-03-21 11:34:03 -07:00
Samantha 4cb3afc9a3
Grant release action permission to post packages to releases (#6004) 2022-03-17 15:36:21 -07:00
Jacob Hoffman-Andrews b073248c6c
Add fetch-and-verify-go.sh (#6005)
Go releases are PGP-signed with a key from
https://www.google.com/linuxrepositories/. We can improve our confidence
in the provenance of our Go binaries by verifying that signature. This
adds a script that encapsulates the public key, the fetch, and the
verification, outputting go.tar.gz once it's verified.

So far this only adds to the release workflow in CI. It needs a little
more thought about how to organize boulder-tools so it can consume
fetch-and-verify-go.sh (which is in a different directory and therefore
not part of the input to `docker build`).
2022-03-16 21:43:51 -07:00
Jacob Hoffman-Andrews d4336e5f4c
Make expiration-mailer clean exit on SIGTERM (#5998)
Plumb a context through everything, and cancel that context when we
catch a shutdown signal.

Fixes #5953
2022-03-16 10:58:18 -07:00
Aaron Gable 910dde95f6
Clean up goodkey configs (#5993)
Fixes https://github.com/letsencrypt/boulder/issues/5851
2022-03-15 15:26:19 -07:00
Aaron Gable 3f26ac0768
Remove false positive in go version comment (#5995) 2022-03-15 10:41:28 -07:00
Jacob Hoffman-Andrews 210735099f
Add documentation of our logging system (#5991) 2022-03-15 10:21:06 -07:00
alexzorin 73ed095f0d
docker: re-export OCSP ports (#5994) 2022-03-14 22:56:59 -07:00
Jacob Hoffman-Andrews edcf4ac18b
Unexport some ports in docker-compose.yml (#5990) 2022-03-14 11:54:02 -07:00
Aaron Gable 07d56e3772
Add new, simpler revocation methods to RA (#5969)
Add two new gRPC methods to the SA:
- `RevokeCertByKey` will be used when the API request was signed by the
  certificate's keypair, rather than a Subscriber keypair. If the
  request is for reason `keyCompromise`, it will ensure that the key is
  added to the blocked keys table, and will attempt to "re-revoke" a
  certificate that was already revoked for some other reason.
- `RevokeCertByApplicant` supports both the path where the original
  subscriber or another account which has proven control over all of the
  identifier in the certificate requests revocation via the API. It does
  not allow the requested reason to be `keyCompromise`, as these
  requests do not represent a demonstration of key compromise.

In addition, add a new feature flag `MozRevocationReasons` which
controls the behavior of these new methods. If the flag is not set, they
behave like they have historically (see above). If the flag is set to true,
then the new methods enforce the upcoming Mozilla policies around
revocation reasons, namely:
- Only the original Subscriber can choose the revocation reason; other
  clients will get a set reason code based on the method of requesting
  revocation. When the original Subscriber requests reason
  `keyCompromise`, this request will be honored, but the key will not be
  blocked and other certificates with that key will not also be revoked.
- Revocations signed with the certificate key will always get reason
  `keyCompromise`, because we do not know who is sending the request and
  therefore must assume that the use of the key in this way represents
  compromise. Because these requests will always be fore reason
  `keyCompromise`, they will always be added to the blocked keys table
  and they will always attempt "re-revocation".
- Revocations authorized via control of all names in the cert will
  always get reason `cessationOfOperation`, which is to be used when the
  original Subscriber does not control all names in the certificate
  anymore.

Finally, update the existing `AdministrativelyRevokeCertificate` method
to use the new helper functions shared by the two new methods.

Part of #5936
2022-03-14 08:58:17 -07:00
Andrew Gabbitas c931cf06e6
Fix typo in rocsp doc comment (#5989) 2022-03-11 11:42:53 -07:00
Jacob Hoffman-Andrews 1047c4cf7a
Remove chisel.py (#5986)
This was used for ACMEv1 and is not needed anymore.

Also update some outdated references in chisel2.py.
2022-03-11 08:39:06 -08:00
Samantha 21416b4d72
Post Debian package on release tag cut (#5977)
- Add a CI workflow which publishes a GitHub Release containing a Debian package
  when a release tag is pushed
- Add a script, called by the CI host, that installs all of the dependencies
  necessary to `make` a Debian package
- Remove the, now defunct, goreleaser config file

Fixes #5970
2022-03-10 16:00:39 -08:00
Jacob Hoffman-Andrews 6395701244
Update CI to test go1.18beta2 (#5982)
This requires using GODEBUG to enable a couple of thing turned off by go1.18 (TLS 1.0/1.1, SHA-1 CSRs).

Also add help for a failure mode of cross builds.
2022-03-09 13:42:15 -08:00
Aaron Gable 32973392de
Revert "Bump google.golang.org/grpc from 1.36.1 to 1.44.0" (#5981)
Reverts letsencrypt/boulder#5963

Turns out the tests are still flaky -- using the `grpc.WaitForReady(true)`
connection option results in sometimes seeing 9 entries added to the
purger queue, and sometimes 10 entries. Reverting because flakiness
on main should not be tolerated.
2022-03-08 10:32:30 -08:00
dependabot[bot] 2ec03b377b
Bump google.golang.org/grpc from 1.36.1 to 1.44.0 (#5963)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.36.1 to 1.44.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.36.1...v1.44.0)

Also update akamai-purger integration test to avoid experimental API.

The `conn.GetState()` API is marked experimental and may change behavior
at any time. It appears to have changed between v1.36.1 and v1.44.0,
and so the akamai-purger integration tests which rely on it break.

Rather than writing our own loop which polls `conn.GetState()`, just
use the stable `WaitForReady(true)` connection option, and apply it to
all connections by setting it as a default option in the dial options.
2022-03-07 17:00:20 -08:00
Samantha 3b665f8dbf
akamai-purger: Queue and response handling improvements (#5955)
- Make maximum queue size configurable via a new configuration key:
  'MaxQueueSize'.
- Default 'MaxQueueSize' to the previous value (1M) when 'MaxQueueSize'
  isn't specified.
- akamaiPurger.purge() will only place the URLs starting at the first entry of
  the failed batch where a failure was encountered instead of the entire set
  that was originally passed.
  - Add a test to ensure that these changes are working as intended.
- Make the purge batching easier to understand with some minor changes
  to variable names
- Responses whose HTTP status code is not 201 will no longer be unmarshaled
- Logs will explicitly call out if a response indicates that we've exceeded any
  rate limits imposed by Akamai.

Fixes #5917
2022-03-07 12:21:16 -08:00
Aaron Gable b19b79162f
Minor updates from review of the HTTP-01 method (#5975)
Make minor updates to our implementation of the HTTP-01 validation method based
on in-depth review of BRs Section 3.2.2.4.19 and RFC 8555 Section 8.3.
- Move the HTTP response code check above parsing the body.
- Explicitly check for 301, 302, 307, and 308 redirect codes, so that if the go
  stdlib updates to allow additional redirects we don't follow suit.
- Trim additional forms of white-space from the key authorization.
2022-03-03 11:23:10 -08:00
dependabot[bot] 8014fa1fb0
Bump github.com/letsencrypt/challtestsrv from 1.2.0 to 1.2.1 (#5964)
Bumps [github.com/letsencrypt/challtestsrv](https://github.com/letsencrypt/challtestsrv) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/letsencrypt/challtestsrv/releases)
- [Commits](https://github.com/letsencrypt/challtestsrv/compare/v1.2.0...v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/letsencrypt/challtestsrv
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-02 09:47:28 -08:00
Aaron Gable e15bb89347
Update miekg/dns from v1.1.30 to v1.1.45 (#5972)
Changelog: https://github.com/miekg/dns/compare/v1.1.30...v1.1.45

Brings in several small changes, but more importantly brings Boulder
up to (and past) the version used by pebble/challtestsrv.
2022-03-02 09:03:13 -08:00
Aaron Gable 745e69e7f9
Add UpdateRevokedCertificate method to SA (#5962)
Add a new gRPC method `UpdateRevokedCertificate` to the SA. This
method takes the same argument as the existing `RevokeCertificate` RPC,
but only operates on certificates that have already been revoked with a
reason other than keyCompromise (c.f. `RevokeCertificate`, which only
operates on certificates that have not been revoked).

One thing to be careful of here is that storing an updated revocation reason
should not also change the revocation date. To support this, add a new field
to the existing `RevokeCertificateRequest` that allows us to differentiate the
time at which the new OCSP response was created, and the time at which
the revocation went into effect.

Part of #5936
2022-02-28 14:22:12 -08:00
Jacob Hoffman-Andrews 5fe5859c38
Add GitHub Action for golangci-lint (#5973)
Adapted from https://github.com/golangci/golangci-lint-action#how-to-use.
Uses the same version we've been using in boulder-tools.

Part of #5946

Note: we will eventually want to go back to doing this in boulder-tools,
so it's easy to run the lints locally. But this is useful so we can
unblock testing on go 1.18beta2.
2022-02-28 09:55:26 -08:00
Samantha 80fe3aed54
akamai-purger: Cleanup (#5949)
Light cleanup of akamai-purger and the akamai cache-client. This does not make
any material changes to logic.

- Use `errors.New` and `errors.Is` instead of a custom `ErrFatal` type and
  `errors.As`
- Add whitespace to separate chunks of execution and error checking from one
  another
- Use `logger.Infof` and `logger.Errorf` instead of wrapped calls to
  `fmt.Sprintf`
- Remove capital letters from the beginning of error messages
- Additional comments and removal of some that are no longer accurate
2022-02-24 20:57:25 -08:00
Andrew Gabbitas d006588f46
Orphan finder: Fix redundant syslog config value (#5971)
Replace redundant stdoutlevel with a sysloglevel value in test configs.
2022-02-24 14:24:03 -08:00
Jacob Hoffman-Andrews 44ab348548
Update gRPC TLS handshake (#5968)
Use newly-available HandshakeContext instead of firing off a goroutine.
https://pkg.go.dev/crypto/tls#Conn.HandshakeContext

Don't set explicit min and max TLS versions. Go's defaults do a good job
protecting us here (and the max version prevented us from using TLS
1.3).
2022-02-24 14:23:32 -08:00
dependabot[bot] 4a75ca4227
Bump github.com/eggsampler/acme/v3 from 3.0.0 to 3.2.1 (#5959)
Update github.com/eggsampler/acme/v3 from 3.0.0 to 3.2.1.
- Release notes: https://github.com/eggsampler/acme/releases
- Commits: https://github.com/eggsampler/acme/compare/v3.0.0...v3.2.1
2022-02-22 16:02:30 -08:00
Aaron Gable c94a24897f
Remove go1.16 backwards compatibility hacks (#5952)
These were needed for the transition from go1.16 to go1.17. We
don't run go1.16 anywhere anymore, so they can be removed.
2022-02-22 14:23:28 -08:00
Aaron Gable ef9f0cdab5
Update go.mod go directive to 1.17 (#5950)
The `go` directive inside go.mod determines certain behaviors of
the go command. Since we're using go 1.17 everywhere, we should
update our module's go directive to reflect that, and update its contents
to match the new behavior.

Particularly, updating to 1.17 here means that all indirect dependencies
are listed directly inside go.mod (in a separate block, to keep things clean),
and the go.sum and go.mod files are deleted from vendored dependencies
so that the go tool can correctly find the root of the module even when run
from a vendored dependency's subdirectory.
2022-02-18 12:25:53 -08:00
Mads Jensen 4b3eb5066c
Add link for funding in GitHub repository (#5956)
Adds a Sponsor button to the GitHub repository page, which
points at our own donation site.
2022-02-18 11:22:00 -08:00
Mads Jensen 86d146e524
dependencies: Enable periodic PRs from dependabot (#5957)
Add dependabot configuration at .github/dependabot.yml which enables periodic
dependency checking and PRs from dependendabot. For more information see the
docs at: 

https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates
2022-02-18 11:01:08 -08:00
Aaron Gable b5b5033136
Simplify gosec linter configuration (#5948)
Replace regex-based filtering at the top-level with linter-specific
configuration.
2022-02-16 15:19:10 -08:00
Aaron Gable d1777c5fda
Fix shadowing assignments inside closures (#5944)
When inside a closure, it is important to not accidentally assign
to variables declared outside the scope of the closure. Doing so
causes static analysis tools (such as `errcheck`) to be unable to
evaluate the lifetime of the variable, and unable to determine if
it is appropriately read from before being assigned to again.

Fix two instances where we assign to a variable declared in the
closure's enclosing scope, rather than declaring a new variable
with the same name.
2022-02-16 14:33:17 -08:00
Andrew Gabbitas 3bb3421631
Remove go 1.17.5 from test matrix (#5940) 2022-02-12 12:03:23 -07:00
Aaron Gable 9fe411185a
Don't use magic strings for authz statuses (#5911)
We have the `core.AcmeStatus` type precisely so that we don't have to
rely on correctly typing the word `"pending"` everywhere. Use those values
instead of raw strings when converting between string-like Authz statuses
and the database bitfield.
2022-02-11 16:00:19 -08:00
Aaron Gable 9221692c35
ocsp-updater.New(): take args instead of conf struct (#5934)
Taking a config object as an argument to a constructor is an anti-
pattern we avoid elsewhere. Resolve the TODO previously put in
place when this first written by instead increasing the number of
arguments `New()` takes and pulling those values out of the config
ahead of time.

Fixes #5904
2022-02-11 15:59:57 -08:00
Andrew Gabbitas fcb817897c
Add go1.17.7 to the test matrix (#5939) 2022-02-10 17:50:46 -07:00
J.C. Jones c9b8eb5c6a
Make cert-checker's stderr less misleading (#5937)
The stderr, when reviewing logs, saying that the sample size was 0
implies that cert-checker sampled no certificates. In reality, it
is working fine; that zero indicates that the JSON array dumped to
stdout has 0 entries, which is actually good.

Let's just fix this stderr message (which isn't parsed by any tooling we
have) so that the next poor SRE that sees it doesn't freak out and file
tickets to start incident response.
2022-02-10 08:43:00 -08:00
Samantha ecd6d0118c
notify-mailer: Improve error checking during template execution (#5932)
- Break message body construction out into a testable method.
- Ensure that in the event of a missing key, an informative error is returned instead
  of allowing the message to be populated with the zero value of the key.
- Add message body construction tests for success, empty map, and missing key. 
- Comment the `recipient` struct and it's `Data` field to make it clear that SRE
  must be informed of any modifications.

Fixes #5921
2022-02-08 09:53:51 -08:00