This is primarily based on @rolandshoemaker's `readme-updates` branch and the closed PR: #3195
The README is restructured to be user-driven. Information that is strictly contributor focused (e.g. dep upgrades) is moved to CONTRIBUTING.md. The review feedback from #3195 from @jsha was all about text that we removed entirely in 29cdd78155
The README now links to the production deployment guide in the wiki and describes our general position on using Boulder.
The CONTRIBUTING.md guide links to the various docs/ pages and was updated.
The docs/acme-divergences page is updated for draft-15.
The DESIGN.md document is moved from the root of the repo into docs/. It has been updated to cover ACMEv2 and precertificates/SCT embedding.
Resolves https://github.com/letsencrypt/boulder/issues/3850
Does a simpler probe than compared to using a `blackbox_exporter`, but directly collects the info we think will aid debugging publisher outages.
Updates #3821.
Unit tests are confirmed to pass:
```
$> git rev-parse HEAD
34b7be2e59081f4bbe6970785e021e6bf0741f2a
$> go test ./...
ok github.com/zmap/zlint 0.224s
? github.com/zmap/zlint/cmd/zlint [no test files]
? github.com/zmap/zlint/cmd/zlint-gtld-update [no test files]
ok github.com/zmap/zlint/lints 0.121s
ok github.com/zmap/zlint/util 0.008s
```
When a retryable error occurs and there are multiple DNS servers configured it is prudent to change servers before retrying the query. This helps ensure that one dead DNS server won't result in queries failing.
Resolves https://github.com/letsencrypt/boulder/issues/3846
For HTTP-01 challenges that return incorrect responses, the
VA tries to put the first little bit of the HTTP response in the problem
detail.
However, VA needs to be able to serialize the problem detail as a
protobuf to send it to the RA, and protobufs require string types to be
UTF-8. Filter out any invalid UTF-8 sequences and replace them with
REPLACEMENT CHARACTER.
Sometimes two simultaneous challenge POSTs can result in a situation
where the challenges get updated to show one of them as "valid", then
later get updated to show them all "pending" again even though the
validation succeeded (and was audit logged).
Fixes#3833
Use a boulder error type to indicate duplicate rows instead of a normal untyped error (as gRPC mangles this type of error but understands how to properly handle a boulder error).
Allows the expired-authz-purger to be run in a daemon mode, by passing the `-daemon` flag on start up while still allowing it to run as it does now for deployment purposes. Checkpointing the last ID seen between invocations of the binary is left for a follow-up change as I'd like to get the major structural changes looked at alone first.
Updates #3840.
Required a little bit of rework of the RA issuance flow (to add parsing of the precert to determine the expiration date, and moving final cert parsing before final cert submission) and RA tests, but I think it shouldn't create any issues...
Fixes#3197.
In db01b0b we removed email validation from the RA. This was the only
use of the `bdns` package by the RA and so we can go one step further
and delete the remaining setup, configuration and `bdns` fields.
This updates our zlint dependency to 8093f211c4 - the tip of master at the time of writing. Notably this brings in a gTLD map that has effective periods so cert-checker can catch issuance for removed gTLDs after their removal date.
Unit tests are confirmed to pass:
```
$> git rev-parse HEAD
8093f211c43679b1ade744d238a02ba1f0c07371
$> go test ./...
ok github.com/zmap/zlint 0.284s
? github.com/zmap/zlint/cmd/zlint [no test files]
? github.com/zmap/zlint/cmd/zlint-gtld-update [no test files]
ok github.com/zmap/zlint/lints 0.165s
ok github.com/zmap/zlint/util 0.005s
```
Performing DNS lookups to check the A/AAAA/MX records of a provided contact e-mail address adds variability to the RA's NewRegistration/UpdateRegistration functions and requires that the RA be able to reach out to the EFN. Since this is simply a convenience to prevent some classes of registration errors we can remove it to improve performance and to tighten up our security posture slightly.
Resolves https://github.com/letsencrypt/boulder/issues/3849
This switches from whitespace to semi-colon separated tag/value parameters,
while implementing stricter checks on valid tag and value values (to match
the RFC). Test coverage is added for CAA value parameter parsing, along with
some additional tests for CAA records with multiple parameter values.
Fixes issue #3795.
If an order is expired the status is invalid and we don't need to get any of the order's authorizations. Its important to exit early in this case because expired authorizations may be purged from the DB. Fetching the authz's for an expired order may return less authz objects than expected, triggering a 500 error response.
Resolves https://github.com/letsencrypt/boulder/issues/3839
Retains the existing logging of orphaned certs until we are confident that this
solution can fully replace it (even then we may want to keep it just for auditing etc).
Fixes#3636.
Fixes#3836.
```
$ ./test.sh
ok github.com/cloudflare/cfssl/api 1.023s coverage: 81.1% of statements
ok github.com/cloudflare/cfssl/api/bundle 1.464s coverage: 87.2% of statements
ok github.com/cloudflare/cfssl/api/certadd 16.766s coverage: 86.8% of statements
ok github.com/cloudflare/cfssl/api/client 1.062s coverage: 51.9% of statements
ok github.com/cloudflare/cfssl/api/crl 1.075s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/gencrl 1.038s coverage: 72.5% of statements
ok github.com/cloudflare/cfssl/api/generator 1.478s coverage: 33.3% of statements
ok github.com/cloudflare/cfssl/api/info 1.085s coverage: 84.1% of statements
ok github.com/cloudflare/cfssl/api/initca 1.050s coverage: 90.5% of statements
ok github.com/cloudflare/cfssl/api/ocsp 1.114s coverage: 93.8% of statements
ok github.com/cloudflare/cfssl/api/revoke 3.063s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/scan 2.988s coverage: 62.1% of statements
ok github.com/cloudflare/cfssl/api/sign 2.680s coverage: 83.3% of statements
ok github.com/cloudflare/cfssl/api/signhandler 1.114s coverage: 26.3% of statements
ok github.com/cloudflare/cfssl/auth 1.010s coverage: 68.2% of statements
ok github.com/cloudflare/cfssl/bundler 22.078s coverage: 84.5% of statements
ok github.com/cloudflare/cfssl/certdb/dbconf 1.013s coverage: 84.2% of statements
ok github.com/cloudflare/cfssl/certdb/ocspstapling 1.302s coverage: 69.2% of statements
ok github.com/cloudflare/cfssl/certdb/sql 1.223s coverage: 70.5% of statements
ok github.com/cloudflare/cfssl/cli 1.014s coverage: 62.5% of statements
ok github.com/cloudflare/cfssl/cli/bundle 1.011s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cli/crl 1.086s coverage: 57.8% of statements
ok github.com/cloudflare/cfssl/cli/gencert 7.927s coverage: 83.6% of statements
ok github.com/cloudflare/cfssl/cli/gencrl 1.064s coverage: 73.3% of statements
ok github.com/cloudflare/cfssl/cli/gencsr 1.058s coverage: 70.3% of statements
ok github.com/cloudflare/cfssl/cli/genkey 2.718s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/cli/ocsprefresh 1.077s coverage: 64.3% of statements
ok github.com/cloudflare/cfssl/cli/revoke 1.033s coverage: 88.2% of statements
ok github.com/cloudflare/cfssl/cli/scan 1.014s coverage: 36.0% of statements
ok github.com/cloudflare/cfssl/cli/selfsign 2.342s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/cli/serve 1.076s coverage: 38.2% of statements
ok github.com/cloudflare/cfssl/cli/sign 1.070s coverage: 54.8% of statements
ok github.com/cloudflare/cfssl/cli/version 1.011s coverage: 100.0% of statements
ok github.com/cloudflare/cfssl/cmd/cfssl 1.028s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cmd/cfssljson 1.012s coverage: 3.4% of statements
ok github.com/cloudflare/cfssl/cmd/mkbundle 1.011s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/config 1.023s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/crl 1.054s coverage: 68.3% of statements
ok github.com/cloudflare/cfssl/csr 8.473s coverage: 89.6% of statements
ok github.com/cloudflare/cfssl/errors 1.014s coverage: 79.6% of statements
ok github.com/cloudflare/cfssl/helpers 1.216s coverage: 80.6% of statements
ok github.com/cloudflare/cfssl/helpers/derhelpers 1.017s coverage: 48.0% of statements
ok github.com/cloudflare/cfssl/helpers/testsuite 7.826s coverage: 65.8% of statements
ok github.com/cloudflare/cfssl/initca 151.314s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/log 1.013s coverage: 59.3% of statements
ok github.com/cloudflare/cfssl/multiroot/config 1.258s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/ocsp 1.353s coverage: 75.1% of statements
ok github.com/cloudflare/cfssl/revoke 1.149s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/scan 1.023s coverage: 1.1% of statements
skipped github.com/cloudflare/cfssl/scan/crypto/md5
skipped github.com/cloudflare/cfssl/scan/crypto/rsa
skipped github.com/cloudflare/cfssl/scan/crypto/sha1
skipped github.com/cloudflare/cfssl/scan/crypto/sha256
skipped github.com/cloudflare/cfssl/scan/crypto/sha512
skipped github.com/cloudflare/cfssl/scan/crypto/tls
ok github.com/cloudflare/cfssl/selfsign 1.098s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer 1.020s coverage: 19.4% of statements
ok github.com/cloudflare/cfssl/signer/local 4.886s coverage: 77.9% of statements
ok github.com/cloudflare/cfssl/signer/remote 2.500s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer/universal 2.228s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/transport 1.012s
ok github.com/cloudflare/cfssl/transport/ca/localca 1.046s coverage: 94.9% of statements
ok github.com/cloudflare/cfssl/transport/kp 1.050s coverage: 37.1% of statements
ok github.com/cloudflare/cfssl/ubiquity 1.037s coverage: 88.3% of statements
ok github.com/cloudflare/cfssl/whitelist 3.519s coverage: 100.0% of statements
...
$ go test ./... (master✱)
ok golang.org/x/crypto/acme 2.782s
ok golang.org/x/crypto/acme/autocert 2.963s
? golang.org/x/crypto/acme/autocert/internal/acmetest [no test files]
ok golang.org/x/crypto/argon2 0.047s
ok golang.org/x/crypto/bcrypt 4.694s
ok golang.org/x/crypto/blake2b 0.056s
ok golang.org/x/crypto/blake2s 0.050s
ok golang.org/x/crypto/blowfish 0.015s
ok golang.org/x/crypto/bn256 0.460s
ok golang.org/x/crypto/cast5 4.204s
ok golang.org/x/crypto/chacha20poly1305 0.560s
ok golang.org/x/crypto/cryptobyte 0.014s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
ok golang.org/x/crypto/curve25519 0.025s
ok golang.org/x/crypto/ed25519 0.073s
? golang.org/x/crypto/ed25519/internal/edwards25519 [no test files]
ok golang.org/x/crypto/hkdf 0.012s
ok golang.org/x/crypto/internal/chacha20 0.047s
ok golang.org/x/crypto/internal/subtle 0.011s
ok golang.org/x/crypto/md4 0.013s
ok golang.org/x/crypto/nacl/auth 9.226s
ok golang.org/x/crypto/nacl/box 0.016s
ok golang.org/x/crypto/nacl/secretbox 0.012s
ok golang.org/x/crypto/nacl/sign 0.012s
ok golang.org/x/crypto/ocsp 0.047s
ok golang.org/x/crypto/openpgp 8.872s
ok golang.org/x/crypto/openpgp/armor 0.012s
ok golang.org/x/crypto/openpgp/clearsign 16.984s
ok golang.org/x/crypto/openpgp/elgamal 0.013s
? golang.org/x/crypto/openpgp/errors [no test files]
ok golang.org/x/crypto/openpgp/packet 0.159s
ok golang.org/x/crypto/openpgp/s2k 7.597s
ok golang.org/x/crypto/otr 0.612s
ok golang.org/x/crypto/pbkdf2 0.045s
ok golang.org/x/crypto/pkcs12 0.073s
ok golang.org/x/crypto/pkcs12/internal/rc2 0.013s
ok golang.org/x/crypto/poly1305 0.016s
ok golang.org/x/crypto/ripemd160 0.034s
ok golang.org/x/crypto/salsa20 0.013s
ok golang.org/x/crypto/salsa20/salsa 0.013s
ok golang.org/x/crypto/scrypt 0.942s
ok golang.org/x/crypto/sha3 0.140s
ok golang.org/x/crypto/ssh 0.939s
ok golang.org/x/crypto/ssh/agent 0.529s
ok golang.org/x/crypto/ssh/knownhosts 0.027s
ok golang.org/x/crypto/ssh/terminal 0.016s
ok golang.org/x/crypto/tea 0.010s
ok golang.org/x/crypto/twofish 0.019s
ok golang.org/x/crypto/xtea 0.012s
ok golang.org/x/crypto/xts 0.016s
```
If the context provided to a remote VA's `PerformValidation` is
cancelled we should not treat the returned context cancelled error as an
unexpected error and should instead ignore it as an expected result.
If a handler doesn't explicitly call `WriteHeader` before `Write` then the status code is set to `http.StatusOK` but `measured_http.MeasuredHandler` doesn't handle this which results in reporting `0` as the response code.
A match of an OCSP request's serial number to *any* of the configured `reqSerialPrefixes` entries is sufficient for the request to be valid, not just the last `reqSerialPrefixes` entry.
Resolves https://github.com/letsencrypt/boulder/issues/3829
Notably this brings an [updated gTLD list](https://github.com/zmap/zlint/pull/233).
Tests are confirmed to pass:
```
go test ./...
ok github.com/zmap/zlint 0.157s
? github.com/zmap/zlint/cmd/zlint [no test files]
ok github.com/zmap/zlint/lints 0.130s
ok github.com/zmap/zlint/util 0.005s
```
The implementation of the `features.ACME13KeyRollover` flag was written
with the intent to allow client developers to send both the `"newKey"`
and `"oldKey"` and be interoperable between both feature flag states. We
should have an explicit unit test for this to be sure it works as
intended.
The previous update was just [9 days ago](https://github.com/letsencrypt/boulder/pull/3808). However, since we merged some changes into the PSL that are related to IANA TLDs I though about providing an immediate patch.
Tests are passing:
```
➜ ~ cd ~/go/src/github.com/weppos/publicsuffix-go
➜ publicsuffix-go git:(master) GOCACHE=off go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.021s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.034s
```
ACME draft-13 changed the inner JWS' body to contain the old key
that signed the outer JWS. Because this is a backwards incompatible
change with the draft-12 ACME v2 key rollover we introduce a new feature
flag `features.ACME13KeyRollover` and conditionally use the old or new
key rollover semantics based on its value.
None of the development team use this approach to running a dev env. It
no longer works without modifying `test/startservers.py` and the `test/`
configurations. Given that it has been broken for a month+ and has only
provoked one user issue I think we can be fairly confident that few others
are using this method of setting up a Boulder development environment
and should prioritize our time/docs accordingly.
This commit updates the vendored `publicsuffix-go` dependency to [b8c0530](b8c0530c1a), the tip of master at the time of writing.
Unit tests confirmed to pass:
```
~/go/src/github.com/weppos/publicsuffix-go/publicsuffix$ go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.007s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.024s
```
Resolves#3807
ctpolicy permutes logs before submitting to them, to give each log a
chance. The stagger feature was meant to sleep for an amount of time
proportional to a log's position in the permuted list. However, it was
actually using the log's position in the un-permuted list, so logs that
appear later in the config would always be submitted to later than logs
earlier in the config.
This fixes that, and does some minor variable renaming for clarity.
Daniel McCarney