This commit fixes `sa.GetAuthorizations` to ensure that both the valid
**and** the pending authorizations it finds for reuse by `ra.NewOrder`
are required to have rows in the order authz join table.
The `sa.GetAuthorizations` unit test is updated to ensure we test the
expected V2 reuse behaviour is enforced.
Resolves https://github.com/letsencrypt/boulder/issues/3565
Previously, each time we defined a new test case in integration-test.py, we had to explicitly call it.
This made it easy to leave out cases without realizing it. After this change, we will automatically
find all functions named "test_" and call them. As a result, I found that we weren't calling
`test_revoked_by_account`, and it was failing. So I fixed it as part of this PR.
Fixes#3518
We are soon enforcing that V2 POST requests have a `Content-Type` of
`application/jose+json` to match ACME specification requirement. For
CORS this requires adding the `Content-Type` to the
`Access-Control-Allow-Headers` CORS field because the JOSE content type
is not one that can be used by a "simple header" whitelisted by default.
This commit adds `Access-Control-Allow-Headers: Content-Type` where
required and updates unit tests accordingly.
We shipped this feature flag disabled because at the time Certbot's acme
module had a bug with revocation that sent the wrong content-type. That
has been fixed upstream and the current boulder-tools image has the fix.
We should be able to turn this flag on now for config-next without
breaking tests.
Prior to this commit, if the expiration-mailer database configuration is
invalid, or the database is unreachable,
`cmd/expiration-mailer/main.go`'s `main()` function tries to call
`sa.SetSQLDebug(dbMap, logger)` before erroring from the DB
initialization failure. This causes a nil panic.
This commit changes the order of two lines such that `sa.SetSQLDebug` is
only called when there was no db setup error.
The service used to do a linear scan for the lowest counter value in its
cross-off list, in order to expire that value. This took about 3ms. I added a
heap to make finding the lowest value more efficient. Now, redeeming a nonce
takes about 3μs.
```
benchmark old ns/op new ns/op delta
BenchmarkNonces-4 1597950 5100 -99.68%
```
Fixes#1219.
Adds SCT embedding to the certificate issuance flow. When a issuance is requested a precertificate (the requested certificate but poisoned with the critical CT extension) is issued and submitted to the required CT logs. Once the SCTs for the precertificate have been collected a new certificate is issued with the poison extension replace with a SCT list extension containing the retrieved SCTs.
Fixes#2244, fixes#3492 and fixes#3429.
By default the MariaDB/MySQL container starts with a global max
connections limit of 100. The SA is configured in `test/config` and
`test/config-next` to use 100 connections. This doesn't leave any
overhead for `ocsp-updater` connections and can be reached under load
pretty easily.
This commit adjusts the global max connections setting from
`test/create_db.sh`, setting it to a more generous `500` instead of the
default `100`.
This is in general the case, but occasionally when there are two inflight validations for
a challenge at once, the authz can get marked invalid while leaving the challenge
"pending". For clients that poll the challenge instead of the authz this can lead to infinite
polling. To stop those clients, we just ensure the challenge status is consistent with its
authorization object.
Fixes#3493.
Requesting a certificate with "*.example.com" and "www.example.com" as
separate SANs doesn't make sense, because "www.example.com" is covered
by the wildcard.
#3524
`boulder-publisher` and `ct-test-serv` make a number of assumptions about _what_ is
being submitted to a CT log (mainly that it's always a plain certificate) which prevents
proper submission of precertificates and verification/usage of the returned SCTs.
This change adds the ability for the publisher to submit both normal certificates and
precertificates and verify the returned SCTs. This change also adds the ability for
`ct-test-serv` to accept `/ct/v1/add-pre-chain` submissions and create correct SCTs
for precert entry types.
Right now we check safe browsing at new-authz time, which introduces a possible
external dependency when calling new-authz. This is usually fine, since most safe
browsing checks can be satisfied locally, but when requests have to go external,
it can create variance in new-authz timing.
Fixes#3491.
For a long time now the WFE has generated URLs based on the incoming
request rather than a hardcoded BaseURL. BaseURL is no longer set in the
prod configs.
This also allows factoring out relativeEndpoint into the web package.
Pulls in SCT list serialization fix, unblocks #3521.
```
ok github.com/cloudflare/cfssl/api/client 1.137s coverage: 52.2% of statements
ok github.com/cloudflare/cfssl/api/crl 1.110s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/gencrl 1.062s coverage: 72.5% of statements
ok github.com/cloudflare/cfssl/api/generator 1.304s coverage: 33.3% of statements
ok github.com/cloudflare/cfssl/api/info 1.133s coverage: 84.1% of statements
ok github.com/cloudflare/cfssl/api/initca 1.068s coverage: 90.5% of statements
ok github.com/cloudflare/cfssl/api/ocsp 1.152s coverage: 93.8% of statements
ok github.com/cloudflare/cfssl/api/revoke 2.574s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/scan 2.885s coverage: 62.1% of statements
ok github.com/cloudflare/cfssl/api/sign 3.188s coverage: 83.3% of statements
ok github.com/cloudflare/cfssl/api/signhandler 1.179s coverage: 26.3% of statements
ok github.com/cloudflare/cfssl/auth 1.012s coverage: 68.2% of statements
ok github.com/cloudflare/cfssl/bundler 15.700s coverage: 84.5% of statements
ok github.com/cloudflare/cfssl/certdb/dbconf 1.016s coverage: 84.2% of statements
ok github.com/cloudflare/cfssl/certdb/ocspstapling 1.415s coverage: 69.2% of statements
ok github.com/cloudflare/cfssl/certdb/sql 1.248s coverage: 70.5% of statements
ok github.com/cloudflare/cfssl/cli 1.013s coverage: 61.9% of statements
ok github.com/cloudflare/cfssl/cli/bundle 1.012s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cli/crl 1.091s coverage: 57.8% of statements
ok github.com/cloudflare/cfssl/cli/gencert 11.960s coverage: 83.6% of statements
ok github.com/cloudflare/cfssl/cli/gencrl 1.089s coverage: 73.3% of statements
ok github.com/cloudflare/cfssl/cli/gencsr 1.064s coverage: 70.3% of statements
ok github.com/cloudflare/cfssl/cli/genkey 6.415s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/cli/ocsprefresh 1.060s coverage: 64.3% of statements
ok github.com/cloudflare/cfssl/cli/revoke 1.033s coverage: 88.2% of statements
ok github.com/cloudflare/cfssl/cli/scan 1.013s coverage: 36.0% of statements
ok github.com/cloudflare/cfssl/cli/selfsign 2.029s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/cli/serve 1.073s coverage: 39.0% of statements
ok github.com/cloudflare/cfssl/cli/sign 1.054s coverage: 54.8% of statements
ok github.com/cloudflare/cfssl/cli/version 1.012s coverage: 100.0% of statements
ok github.com/cloudflare/cfssl/cmd/cfssl 1.036s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cmd/cfssljson 1.018s coverage: 3.4% of statements
ok github.com/cloudflare/cfssl/cmd/mkbundle 1.012s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/config 1.029s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/crl 1.056s coverage: 68.3% of statements
ok github.com/cloudflare/cfssl/csr 31.882s coverage: 89.6% of statements
ok github.com/cloudflare/cfssl/errors 1.016s coverage: 79.6% of statements
ok github.com/cloudflare/cfssl/helpers 1.251s coverage: 82.8% of statements
ok github.com/cloudflare/cfssl/helpers/testsuite 6.974s coverage: 65.8% of statements
ok github.com/cloudflare/cfssl/initca 207.580s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/log 1.010s coverage: 59.3% of statements
ok github.com/cloudflare/cfssl/multiroot/config 1.161s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/ocsp 1.230s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/revoke 1.336s coverage: 77.9% of statements
ok github.com/cloudflare/cfssl/scan 1.016s coverage: 1.1% of statements
ok github.com/cloudflare/cfssl/selfsign 1.059s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer 1.014s coverage: 19.4% of statements
ok github.com/cloudflare/cfssl/signer/local 3.355s coverage: 77.9% of statements
ok github.com/cloudflare/cfssl/signer/remote 2.371s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer/universal 2.163s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/transport 1.012s
ok github.com/cloudflare/cfssl/transport/ca/localca 1.043s coverage: 94.9% of statements
ok github.com/cloudflare/cfssl/transport/core 1.030s coverage: 90.9% of statements
ok github.com/cloudflare/cfssl/transport/kp 1.032s coverage: 37.1% of statements
ok github.com/cloudflare/cfssl/ubiquity 1.034s coverage: 88.3% of statements
ok github.com/cloudflare/cfssl/whitelist 2.879s coverage: 100.0% of statements
```
This commit adds a new WFE2 feature flag "EnforceV2ContentType". When
enabled, the WFE2's validPostRequest function will enforce that the
request carries a Content-Type header equal to
application/jose+json. This is required by ACME draft-10 per section
6.2 "Request Authentication".
This is behind a feature flag because it is likely to break
some number of existing ACMEv2 clients that may not be sending the
correct Content-Type.
We are defaulting to not setting the new feature flag in test/config-next
because it currently break's Certbot's acme module's revocation support
and we rely on this in our V2 integration tests.
Resolves#3529
Some of the pprof handlers have to be accessed through
pprof.Handler("string"), while some have to be accessed through an
exported var in pprof. We weren't doing the latter before, which meant
some key handlers like Profile weren't available.
The logEvent setup we had in issueCertificate depending on all error
handling code setting logEvent.Error in addition to returning the error.
Since this is not the common pattern in Go, there were a few places
where logEvent.Error wasn't set, making it hard to find the root cause
of errors in the RA logs (though the errors would get propagated to the
WFE for logging). This change wraps issueCertificate such that all
errors returned get logged.
This pulls in an updated Certbot repo with a fix to content-type for the
revoke method.
Also, adds a convenience replacement in tag_and_release.sh to update
docker-compose.yml.
This pulls in an upstream change that allows us to reference the Protected
header separately from the unprotected one (confusingly just called Header).
$ go test gopkg.in/square/go-jose.v2/...
ok gopkg.in/square/go-jose.v2 16.625s
ok gopkg.in/square/go-jose.v2/cipher 0.004s
? gopkg.in/square/go-jose.v2/jose-util [no test files]
ok gopkg.in/square/go-jose.v2/json 2.080s
? gopkg.in/square/go-jose.v2/jwk-keygen [no test files]
ok gopkg.in/square/go-jose.v2/jwt 0.128s
```
roland@catbus ~/code/go/src/github.com/cloudflare/cfssl master ./test.sh
BUILDING.md Gopkg.toml certdb crl helpers revoke test.sh
CHANGELOG LICENSE certinfo crypto info scan testdata
Dockerfile README.md cli csr initca script transport
Dockerfile.build api cmd doc log selfsign ubiquity
Dockerfile.minimal auth config errors multiroot signer vendor
Gopkg.lock bundler coverprofile.txt gopath ocsp test.prof whitelist
ok github.com/cloudflare/cfssl/api 1.043s coverage: 81.1% of statements
ok github.com/cloudflare/cfssl/api/bundle 1.570s coverage: 87.2% of statements
ok github.com/cloudflare/cfssl/api/certadd 12.607s coverage: 86.8% of statements
ok github.com/cloudflare/cfssl/api/client 1.070s coverage: 52.2% of statements
ok github.com/cloudflare/cfssl/api/crl 1.107s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/gencrl 1.057s coverage: 72.5% of statements
ok github.com/cloudflare/cfssl/api/generator 1.262s coverage: 33.3% of statements
ok github.com/cloudflare/cfssl/api/info 1.102s coverage: 84.1% of statements
ok github.com/cloudflare/cfssl/api/initca 1.073s coverage: 90.5% of statements
ok github.com/cloudflare/cfssl/api/ocsp 1.116s coverage: 93.8% of statements
ok github.com/cloudflare/cfssl/api/revoke 2.923s coverage: 75.0% of statements
ok github.com/cloudflare/cfssl/api/scan 17.178s coverage: 62.1% of statements
ok github.com/cloudflare/cfssl/api/sign 2.221s coverage: 83.3% of statements
ok github.com/cloudflare/cfssl/api/signhandler 1.145s coverage: 26.3% of statements
ok github.com/cloudflare/cfssl/auth 1.022s coverage: 68.2% of statements
ok github.com/cloudflare/cfssl/bundler 14.899s coverage: 84.5% of statements
ok github.com/cloudflare/cfssl/certdb/dbconf 1.040s coverage: 84.2% of statements
ok github.com/cloudflare/cfssl/certdb/ocspstapling 1.283s coverage: 69.2% of statements
ok github.com/cloudflare/cfssl/certdb/sql 1.092s coverage: 70.5% of statements
ok github.com/cloudflare/cfssl/cli 1.036s coverage: 61.9% of statements
ok github.com/cloudflare/cfssl/cli/bundle 1.034s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cli/crl 1.106s coverage: 57.8% of statements
ok github.com/cloudflare/cfssl/cli/gencert 6.106s coverage: 83.6% of statements
ok github.com/cloudflare/cfssl/cli/gencrl 1.081s coverage: 73.3% of statements
ok github.com/cloudflare/cfssl/cli/gencsr 1.075s coverage: 70.3% of statements
ok github.com/cloudflare/cfssl/cli/genkey 2.903s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/cli/ocsprefresh 1.074s coverage: 64.3% of statements
ok github.com/cloudflare/cfssl/cli/revoke 1.054s coverage: 88.2% of statements
ok github.com/cloudflare/cfssl/cli/scan 1.032s coverage: 36.0% of statements
ok github.com/cloudflare/cfssl/cli/selfsign 2.429s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/cli/serve 1.172s coverage: 39.0% of statements
ok github.com/cloudflare/cfssl/cli/sign 1.058s coverage: 54.8% of statements
ok github.com/cloudflare/cfssl/cli/version 1.028s coverage: 100.0% of statements
ok github.com/cloudflare/cfssl/cmd/cfssl 1.196s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/cmd/cfssljson 1.031s coverage: 3.4% of statements
ok github.com/cloudflare/cfssl/cmd/mkbundle 1.032s coverage: 0.0% of statements [no tests to run]
ok github.com/cloudflare/cfssl/config 1.054s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/crl 1.072s coverage: 68.3% of statements
ok github.com/cloudflare/cfssl/csr 20.657s coverage: 89.6% of statements
ok github.com/cloudflare/cfssl/errors 1.029s coverage: 79.6% of statements
ok github.com/cloudflare/cfssl/helpers 1.225s coverage: 82.8% of statements
ok github.com/cloudflare/cfssl/helpers/testsuite 6.558s coverage: 65.8% of statements
ok github.com/cloudflare/cfssl/initca 81.870s coverage: 73.2% of statements
ok github.com/cloudflare/cfssl/log 1.019s coverage: 59.3% of statements
ok github.com/cloudflare/cfssl/multiroot/config 1.190s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/ocsp 1.226s coverage: 77.4% of statements
ok github.com/cloudflare/cfssl/revoke 1.832s coverage: 77.9% of statements
ok github.com/cloudflare/cfssl/scan 1.042s coverage: 1.1% of statements
ok github.com/cloudflare/cfssl/selfsign 1.073s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer 1.030s coverage: 19.4% of statements
ok github.com/cloudflare/cfssl/signer/local 3.171s coverage: 78.1% of statements
ok github.com/cloudflare/cfssl/signer/remote 2.197s coverage: 70.0% of statements
ok github.com/cloudflare/cfssl/signer/universal 2.061s coverage: 67.7% of statements
ok github.com/cloudflare/cfssl/transport 1.031s
ok github.com/cloudflare/cfssl/transport/ca/localca 1.062s coverage: 94.9% of statements
ok github.com/cloudflare/cfssl/transport/core 1.054s coverage: 90.9% of statements
ok github.com/cloudflare/cfssl/transport/kp 1.059s coverage: 37.1% of statements
ok github.com/cloudflare/cfssl/transport/roots/system 1.384s coverage: 77.1% of statements
ok github.com/cloudflare/cfssl/ubiquity 1.057s coverage: 88.3% of statements
ok github.com/cloudflare/cfssl/whitelist 2.060s coverage: 100.0% of statements
```
This commit updates the RA to make the notion of submitting
a KeyAuthorization value as part of the ra.UpdateAuthorization call
optional. If set, the value is enforced against expected and an error is
returned if the provided authorization isn't correct. If it isn't set
the RA populates the field with the computed authorization for the VA to
enforce against the value it sees in challenges. This retains the legacy
behaviour of the V1 API. The V2 API will never unmarshal a provided
key authorization.
The ACMEv2/WFEv2 prepChallengeForDisplay function is updated to strip
the ProvidedKeyAuthorization field before sending the challenge object
back to a client. ACMEv1/WFEv1 continue to return the KeyAuthorization
in challenges to avoid breaking clients that are relying on this legacy
behaviour.
For deployability ease this commit retains the name of the
core.Challenge.ProvidedKeyAuthorization field even though it should
be called core.Challenge.ComputedKeyAuthorization now that it isn't
set based on the client's provided key authz. This will be easier as
a follow-up change.
Resolves#3514
Prior to this commit all of the `newOrder` requests generated by the
load generator had 1 DNS type identifier. In this commit the
`maxNamesPerCert` parameter that was present in the codebase is fixed
(it was never wired through to config before) and used to size the
orders randomly. Each order will have between 1 and `maxNamesPerCert`
identifiers.
For integration testing we set this at 20. 100 would be more realistic
but we'll start small for now.
This commit adds short 15s runs of the load generator against the V1 and
V2 APIs during the three integration test runs (v1 config, v1
config-next, and v2). 15s was selected because 30s caused too much
output and the build log to be truncated.
Presently the latency output is *not* being checked for errors. This was
too flaky in practice.
A fix for a race condition in the load-generator code itself related to
HTTP status code tracking is included in this commit.
The pending authz rate limit also needed to be adjusted to keep the
load-generator from failing requests after hitting 429s.
This race was uncovered by running the load generator as part of our CI.
Also, update ra_test.go. It was previously testing that the returned authz
and the stored authz should be identical, which is not actually a property
of UpdateAuthorization; in general, they will not be identical.
This code was never enabled in production. Our original intent was to
ship this as part of the ACMEv2 API. Before that could happen flaws were
identified in TLS-SNI-01|02 that resulted in TLS-SNI-02 being removed
from the ACME protocol. We won't ever be enabling this code and so we
might as well remove it.
There's a minor data race in UpdateAuthorization where the authorization can
be changed in the goroutine concurrently with returning it. This fixes it by
copying the authorization as an argument to the goroutine.
Fixes#3506.
Previously we introduced the concept of a "pending orders per account
ID" rate limit. After struggling with making an implementation of this
rate limit perform well we reevaluated the problem and decided a "new
orders per account per time window" rate limit would be a better fit for
ACMEv2 overall.
This commit introduces the new newOrdersPerAccount rate limit. The RA
now checks this before creating new pending orders in ra.NewOrder. It
does so after order reuse takes place ensuring the rate limit is only
applied in cases when a distinct new pending order row would be created.
To accomplish this a migration for a new orders field (created) and an
index over created and registrationID is added. It would be possible to
use the existing expires field for this like we've done in the past, but that
was primarily to avoid running a migration on a large table in prod. Since
we don't have that problem yet for V2 tables we can Do The Right Thing
and add a column.
For deployability the deprecated pendingOrdersPerAccount code & SA
gRPC bits are left around. A follow-up PR will be needed to remove
those (#3502).
Resolves#3410
Prior to this commit the comment on `pa.WillingToIssue` incorrectly
asserted that the input domain must not contain IDN labels. That's not
true anymore! 🎉
This commit updates `sa.getAllOrderAuthorizations` to not exclude
expired authorizations. The expired authorizations are used in
`sa.statusForOrder` to set the overall order status to invalid when one
or more authorizations are expired.
Fixes#3499
When we originally deployed the inline CT submission code, we wanted to
be conservative in case it increased our issuance error rate. However,
we've established that the success rate is quite good, so we'll remove
some complexity and make things more realistic by removing the code that
avoids returning errors when CT submission fails.
This change updates boulder-tools to use Go 1.10, and references a
newly-pushed image built using that new config.
Since boulder-tools pulls in the latest Certbot master at the time of
build, this also pulls in the latest changes to Certbot's acme module,
which now supports ACME v2. This means we no longer have to check out
the special acme-v2-integration branch in our integration tests.
This also updates chisel2.py to reflect some of the API changes that
landed in the acme module as it was merged to master.
Since we don't need additional checkouts to get the ACMEv2-compatible
version of the acme module, we can include it in the default RUN set for
local tests.
This puts puts the path for order finalization at the beginning of the URL, rather than the end, making it easier to generate stats about. It also allows us to split out GetOrder and FinalizeOrder from the previous "Order" function that combined some unrelated code.
Fixes#3440Fixes#3439 (order finalization now has its own stats like the other endpoints do)
Roland Bracewell Shoemaker