The existing RA `UpdateAuthorization` RPC needs replacing for
two reasons:
1. The name isn't accurate - `PerformValidation` better captures
the purpose of the RPC.
2. The `core.Challenge` argument is superfluous since Key
Authorizations are not sent in the initiation POST from the client
anymore. The corresponding unmarshal and verification is now
removed. Notably this means broken clients that were POSTing
the wrong thing and failing pre-validation will now likely fail
post-validation.
To remove `UpdateAuthorization` the new `PerformValidation`
RPC is added alongside the old one. WFE and WFE2 are
updated to use the new RPC when the perform validation
feature flag is enabled. We can remove
`UpdateAuthorization` and its associated wrappers once all
WFE instances have been updated.
Resolves https://github.com/letsencrypt/boulder/issues/3930
Removes superfluous usage of `UpdatePendingAuthorization` in the RA to update the key authorization and test if the authorization is pending and instead uses the result of the initial `GetAuthorization` call in the WFE.
Fixes#3923.
This allows POST-as-GET requests to Orders, Authorizations, Challenges, Certificates and Accounts. Legacy GET support remains for Orders, Authorizations, Challenges and Certificates. Legacy "POST {}" support for Accounts remains.
Resolves https://github.com/letsencrypt/boulder/issues/3871
Resolves https://github.com/letsencrypt/boulder/issues/3872
**Note to reviewers**: There's an outstanding bug that I've tracked down to the `--load` stage of the integration tests that results in one of the remote VA instances in the `test/config-next` configuration under Go 1.11.1 to fail to cleanly shut down. I'm working on finding the root cause but in the meantime I've disabled `--load` during CI so we can unblock moving forward with getting Go 1.11.1 in dev/CI. Tracking this in https://github.com/letsencrypt/boulder/issues/3889
Removes the checks for a handful of deployed feature flags in preparation for removing the flags entirely. Also moves all of the currently deprecated flags to a separate section of the flags list so they can be more easily removed once purged from production configs.
Fixes#3880.
The implementation of the `features.ACME13KeyRollover` flag was written
with the intent to allow client developers to send both the `"newKey"`
and `"oldKey"` and be interoperable between both feature flag states. We
should have an explicit unit test for this to be sure it works as
intended.
ACME draft-13 changed the inner JWS' body to contain the old key
that signed the outer JWS. Because this is a backwards incompatible
change with the draft-12 ACME v2 key rollover we introduce a new feature
flag `features.ACME13KeyRollover` and conditionally use the old or new
key rollover semantics based on its value.
This commit updates the `wfe2/verify.go` implementation of
`acctIDFromURL` to include the invalid `kid` value being rejected as
part of the rejection error message. This will hopefully help
users/client developers understand the problem faster. Unit tests are
updated accordingly.
While we intended to allow legacy ACME v1 accounts created through the WFE to work with the ACME v2 implementation and the WFE2 we neglected to consider that a legacy account would have a Key ID URL that doesn't match the expected for a V2 account. This caused `wfe2/verify.go`'s `lookupJWK` to reject all POST requests authenticated by a legacy account unless the ACME client took the extra manual step of "fixing" the URL.
This PR adds a configuration parameter to the WFE2 for an allowed legacy key ID prefix. The WFE2 verification logic is updated to allow both the expected key ID prefix and the configured legacy key ID prefix. This will allow us to specify the correct legacy URL in configuration for both staging/prod to allow unmodified V1 ACME accounts to be used with ACME v2.
Resolves https://github.com/letsencrypt/boulder/issues/3674
Remove various unnecessary uses of fmt.Sprintf - in particular:
- Avoid calls like t.Error(fmt.Sprintf(...)), where t.Errorf can be used directly.
- Use strconv when converting an integer to a string, rather than using
fmt.Sprintf("%d", ...). This is simpler and can also detect type errors at
compile time.
- Instead of using x.Write([]byte(fmt.Sprintf(...))), use fmt.Fprintf(x, ...).
Many of the probs.XYZ calls are of the form probs.XYZ(fmt.Sprintf(...)).
Convert these functions to take a format string and optional arguments,
following the same pattern used in the errors package. Convert the
various call sites to remove the now redundant fmt.Sprintf calls.
A very large number of the logger calls are of the form log.Function(fmt.Sprintf(...)).
Rather than sprinkling fmt.Sprintf at every logger call site, provide formatting versions
of the logger functions and call these directly with the format and arguments.
While here remove some unnecessary trailing newlines and calls to String/Error.
Splits out the old `Errors` slice into a public `Error` string and a `InternalErrors` slice. Also removes a number of occurrences of calling `logEvent.AddError` then immediately calling `wfe.sendError` with either the same internal error which caused the same error to be logged twice or no error which is slightly redundant as `wfe.sendError` calls `logEvent.AddError` internally.
Fixes#3664.
* SA: Add Order "Ready" status, feature flag.
This commit adds the new "Ready" status to `core/objects.go` and updates
`sa.statusForOrder` to use it conditionally for orders with all valid
authorizations that haven't been finalized yet. This state is used
conditionally based on the `features.OrderReadyStatus` feature flag
since it will likely break some existing clients that expect status
"Processing" for this state. The SA unit test for `statusForOrder` is
updated with a "ready" status test case.
* RA: Enforce order ready status conditionally.
This commit updates the RA to conditionally expect orders that are being
finalized to be in the "ready" status instead of "pending". This is
conditionally enforced based on the `OrderReadyStatus` feature flag.
Along the way the SA was changed to calculate the order status for the
order returned in `sa.NewOrder` dynamically now that it could be
something other than "pending".
* WFE2: Conditionally enforce order ready status for finalization.
Similar to the RA the WFE2 should conditionally enforce that an order's
status is either "ready" or "pending" based on the "OrderReadyStatus"
feature flag.
* Integration: Fix `test_order_finalize_early`.
This commit updates the V2 `test_order_finalize_early` test for the
"ready" status. A nice side-effect of the ready state change is that we
no longer invalidate an order when it is finalized too soon because we
can reject the finalization in the WFE. Subsequently the
`test_order_finalize_early` testcase is also smaller.
* Integration: Test classic behaviour w/o feature flag.
In the previous commit I fixed the integration test for the
`config/test-next` run that has the `OrderReadyStatus` feature flag set
but broke it for the `config/test` run without the feature flag.
This commit updates the `test_order_finalize_early` test to work
correctly based on the feature flag status in both cases.
This commit removes the possibility of unmarshaling the `Agreement`
field of a new account request. This is a legacy V1 concept and has
no bearing on ACMEv2. Instead a specific one-off struct that only
contains the fields we wish to allow update for (Contact and Status) is
used to unmarshal the new account request.
This commit updates the WFE2 to remove the "Agreement" value on V2 account objects before returning them to the user. This field is not defined in the V2 specification and we should not be returning it.
The V2 `TermsOfServiceAgreed` field is marked optional, and for Let's Encrypt purposes it doesn't make much sense to write it in returned Account objects because the value will necessarily be true 100% of the time. We never create an account unless the request has `TermsOfServiceAgreed: true`.
Resolves https://github.com/letsencrypt/boulder/issues/3590
This commit updates the WFE and WFE2 to have configuration support for
setting a value for the `/directory` object's "meta" field's
optional "caaIdentities" and "website" fields. The config-next wfe/wfe2
configuration are updated with values for these fields. Unit tests are
updated to check that they are sent when expected and not otherwise.
Bonus content: The `test.AssertUnmarshaledEquals` function had a bug
where it would consider two inputs equal when the # of keys differed.
This commit also fixes that bug.
This commit updates CTPolicy & the RA to return a distinct error when
the RA is unable to fetch the required SCTs for a certificate when
processing an issuance. This error type is plumbed up to the WFE/WFE2
where the `web/probs.go` code converts it into a server internal error
with a suitable user facing error.
In some places in the WFE/WFE2 we were calling `logEvent.AddError` and
adding a message that was ~= identical to the `detail` of
a `ProblemDetails` object returned through the API. For these cases this
commit removes the `.AddError` call. We can reference the information
from the API level error and this will save us log bytes overall.
This commit maintains instances where we call `logEvent.AddError` to add
a message with *more* detail than is returned through the API (e.g.
including ID #s or internal error strings).
Account objects contain email addresses, which the subscriber may choose to
change or delete, and which we protect under our privacy policy. Since audit
logs are retained much longer than regular logs, keeping email addresses out of
the audit logs improves the privacy properties of our email storage.
We are soon enforcing that V2 POST requests have a `Content-Type` of
`application/jose+json` to match ACME specification requirement. For
CORS this requires adding the `Content-Type` to the
`Access-Control-Allow-Headers` CORS field because the JOSE content type
is not one that can be used by a "simple header" whitelisted by default.
This commit adds `Access-Control-Allow-Headers: Content-Type` where
required and updates unit tests accordingly.
This is in general the case, but occasionally when there are two inflight validations for
a challenge at once, the authz can get marked invalid while leaving the challenge
"pending". For clients that poll the challenge instead of the authz this can lead to infinite
polling. To stop those clients, we just ensure the challenge status is consistent with its
authorization object.
Fixes#3493.
For a long time now the WFE has generated URLs based on the incoming
request rather than a hardcoded BaseURL. BaseURL is no longer set in the
prod configs.
This also allows factoring out relativeEndpoint into the web package.
This commit adds a new WFE2 feature flag "EnforceV2ContentType". When
enabled, the WFE2's validPostRequest function will enforce that the
request carries a Content-Type header equal to
application/jose+json. This is required by ACME draft-10 per section
6.2 "Request Authentication".
This is behind a feature flag because it is likely to break
some number of existing ACMEv2 clients that may not be sending the
correct Content-Type.
We are defaulting to not setting the new feature flag in test/config-next
because it currently break's Certbot's acme module's revocation support
and we rely on this in our V2 integration tests.
Resolves#3529
This commit updates the RA to make the notion of submitting
a KeyAuthorization value as part of the ra.UpdateAuthorization call
optional. If set, the value is enforced against expected and an error is
returned if the provided authorization isn't correct. If it isn't set
the RA populates the field with the computed authorization for the VA to
enforce against the value it sees in challenges. This retains the legacy
behaviour of the V1 API. The V2 API will never unmarshal a provided
key authorization.
The ACMEv2/WFEv2 prepChallengeForDisplay function is updated to strip
the ProvidedKeyAuthorization field before sending the challenge object
back to a client. ACMEv1/WFEv1 continue to return the KeyAuthorization
in challenges to avoid breaking clients that are relying on this legacy
behaviour.
For deployability ease this commit retains the name of the
core.Challenge.ProvidedKeyAuthorization field even though it should
be called core.Challenge.ComputedKeyAuthorization now that it isn't
set based on the client's provided key authz. This will be easier as
a follow-up change.
Resolves#3514
This puts puts the path for order finalization at the beginning of the URL, rather than the end, making it easier to generate stats about. It also allows us to split out GetOrder and FinalizeOrder from the previous "Order" function that combined some unrelated code.
Fixes#3440Fixes#3439 (order finalization now has its own stats like the other endpoints do)
This removes some fields from "Extra" that are logged on every poll event and
aren't necessarily. For instance, authorizationID and challengeID can easily be
derived from the endpoint, and AuthorizationRequesterID is a duplicate of
Requester.
This commit resolves the case where an error during finalization occurs.
Prior to this commit if an error (expected or otherwise) occurred after
setting an order to status processing at the start of order
finalization the order would be stuck processing forever.
The SA now has a `SetOrderError` RPC that can be used by the RA to
persist an error onto an order. The order status calculation can use
this error to decide if the order is invalid. The WFE is updated to
write the error to the order JSON when displaying the order information.
Prior to this commit the order protobuf had the error field as
a `[]byte`. It doesn't seem like this is the right decision, we have
a specific protobuf type for ProblemDetails and so this commit switches
the error field to use it. The conversion to/from `[]byte` is done with
the model by the SA.
An integration test is included that prior to this commit left an order
in a stuck processing state. With this commit the integration test
passes as expected.
Resolves https://github.com/letsencrypt/boulder/issues/3403
This commit implements a mapping from certificate AIA Issuer URL to PEM
encoded certificate chain. GET's to the V2 Certificate endpoint will
return a full PEM encoded certificate chain in addition to the leaf cert
using the AIA issuer URL of the leaf cert and the configured mapping.
The boulder-wfe2 command builds the chain mapping by reading the
"wfe" config section's 'certificateChains" field, specifying a list
of file paths to PEM certificates for each AIA issuer URL. At startup
the PEM file contents are ready, verified and separated by a newline.
The resulting populated AIA issuer URL -> PEM cert chain mapping is
given to the WFE for use with the Certificate endpoint.
Resolves#3291
This commit removes `CertCacheDuration`, `CertNoCacheExpirationWindow`,
`IndexCacheDuration` and `IssuerCacheDuration`. These were read from
config values that weren't set in config/config-next into WFE struct
fields that were never referenced in any code.
In two places of the WFE2 we had a non-nil error and produced a server
internal problem but did not pass along the non-nil error to be logged.
This leaves only the non-descriptive server internal error to work with.
This commit updates the WFE2 to pass the err along too.
Prior to this commit the WFE2 returned a HTTP 409 status with
a malformed problem body when a newAccount request arrived signed with
the same key as is used with an existing account. Per draft-09 this
should be a HTTP 200 OK response. This is what Pebble implements as
well.
This commit updates the WFE2 and tests to match draft-09 behaviour in
this regard.
An outdated TODO is also removed. The case where an error other
than berrors.NotFound is returned is handled correctly.
Resolves#3327
Prior to this commit we called `response.WriteHeader(http.StatusOK)` in
the wfe2's `Certificate` handler before
`response.Header().Set("Content-Type", "")`, which meant that the
response was served with a Content-Type of `application/json` instead of
the intended `application/pem-certificate-chain`. This commit changes
the order so the correct Content-Type header is written in the response.
This commit adds support for the "onlyReturnExisting" field of
new-account requests. If present & true then new-account will check if
an existing account for the provided key exists. If it does a Location
header is returned with the existing account's URL. If it does not exist
an error is returned. This is contrary to the "onlyReturnExisting=false"
behaviour that creates an account if an existing account with the same
key does not exist.
Resolves#3281
- Encode certificate as PEM.
- Use lowercase for field names.
- Use termsOfServiceAgreed instead of Agreement
- Use a different ToS URL for v2 that points at the v2 HTTPS port.
Resolves#3280
This PR updates the wfe2 to use the camelCase convention[0] from the
recently published ACME draft-09[1]. For the Boulder WFE2 that primarily
meant changing directory keys and updating the order "FinalizeURL" field
to be "Finalize".
[0] - 7bace61b89
[1] - https://tools.ietf.org/html/draft-ietf-acme-acme-09
The `combinations` field of an authorization is a relic of the V1 API
and not present in the current ACME draft specifications. This commit
updates the WFE2's `prepAuthorizationForDisplay` function to remove the
`combinations` field before returning an authorization object to be
displayed in an API response.
`terms-of-service-agreed` is checked at initial signup and doesn't need
to be rechecked. Worse, since the V2 registration only accepts a bool
the "Agreement" field is never set and checking it against != "" will
always fail for v2 accounts.
This was already done for Pebble[0] but was missed in the Boulder
WFE2.
[0] - 6d6e811c1d
This commit adds the "new-nonce" endpoint to the WFE2. A small unit test
is included. The existing /directory unit tests are updated for the new
endpoint.
When we implemented the new-order issuance flow for the WFE2 we forgot to include the endpoint in the /directory object. This commit adds it and updates associated tests.
This PR implements issuance for wildcard names in the V2 order flow. By policy, pending authorizations for wildcard names only receive a DNS-01 challenge for the base domain. We do not re-use authorizations for the base domain that do not come from a previous wildcard issuance (e.g. a normal authorization for example.com turned valid by way of a DNS-01 challenge will not be reused for a *.example.com order).
The wildcard prefix is stripped off of the authorization identifier value in two places:
When presenting the authorization to the user - ACME forbids having a wildcard character in an authorization identifier.
When performing validation - We validate the base domain name without the *. prefix.
This PR is largely a rewrite/extension of #3231. Instead of using a pseudo-challenge-type (DNS-01-Wildcard) to indicate an authorization & identifier correspond to the base name of a wildcard order name we instead allow the identifier to take the wildcard order name with the *. prefix.
This PR implements order finalization for the ACME v2 API.
In broad strokes this means:
* Removing the CSR from order objects & the new-order flow
* Adding identifiers to the order object & new-order
* Providing a finalization URL as part of orders returned by new-order
* Adding support to the WFE's Order endpoint to receive finalization POST requests with a CSR
* Updating the RA to accept finalization requests and to ensure orders are fully validated before issuance can proceed
* Updating the SA to allow finding order authorizations & updating orders.
* Updating the CA to accept an Order ID to log when issuing a certificate corresponding to an order object
Resolves#3123
There were two bugs in #3167:
All process-level stats got prefixed with "boulder", which broke dashboards.
All request_time stats got dropped, because measured_http was using the prometheus DefaultRegisterer.
To fix, this PR plumbs through a scope object to measured_http, and uses an empty prefix when calling NewProcessCollector().
* Move probs.go to web.
* Move probs_test.go
* Factor out probs.go from wfe
* Move context.go
* Extract context.go into web package.
* Add a constructor for TopHandler.
This is shared code between both packages. Better to have it in a single shared place.
In the process, remove the unexported signatureValidationError, which was unnecessary; all returned errors from checkAlgorithm get turned into Malformed.
In 7d04ea9 we introduced the notion of a requestEvent, which had an AddError method that could be called to log an error. In that change we also added an AddError call before every wfe.sendError, to ensure errors got logged. In dc58017, we made it so that sendError would automatically add its errors to the request event, so we wouldn't need to write AddError everywhere. However, we never cleaned up the existing AddError calls, and since then have tended to "follow local style" and add a redundant AddError before many of our sendError calls.
This change attempts to undo some of that, by removing all AddError calls that appear to be redundant with the sendError call immediately following. It also adds a section on error handling to CONTRIBUTING.md.
Previously, CAA problems were lumped in under "ConnectionProblem" or
"Unauthorized". This should make things clearer and easier to differentiate.
Fixes#3043
I thought there was a bug in NewRegistration when GetRegByKey returns an error, so I wrote a unittest... and discovered it works correctly. Oh well, now we have more tests!
We originally planned to use this to match up logs across multiple systems, but we don't currently use it, and it chews up a lot of space in our logs. We can add it back in later if/when we want to start doing that correlation.
Add a logging statement that fires when a remote VA fail causes
overall failure. Also change remoteValidationFailures into a
counter that counts the same thing, instead of a histogram. Since
the histogram had the default bucket sizes, it failed to collect
what we needed, and produced more metrics than necessary.
This commit adds a new boulder error type WrongAuthorizationState.
This error type is returned by the SA when UpdateAuthorization is
provided an authz that isn't pending. The RA and WFE are updated
accordingly such that this error percolates back through the API to the
user as a Malformed problem with a sensible description. Previously this
behaviour resulted in a ServerInternal error.
Resolves#3032
To support having problem types that use either the classic
"urn:acme:error" namespace or the new "urn:ietf:params:acme:error"
namespace as appropriate we need to prefix the problem type at runtime
right before returning it through the WFE to the user as JSON. This
commit updates the WFE/WFE2 to do this for both problems sent through
sendError as well as problems embedded in challenges. For the latter
we do not modify problems with a type that is already prefixed to
support backwards compatibility.
Resolves#2938
Note: We should cut a follow-up issue to devise a way to share some
common code between the WFE and WFE2. For example, the
prepChallengeForDisplay should probably be hoisted to a common
"web" package
* Remove all of the errors under core. Their purpose is now served by errors, and they were almost entirely unused. The remaining uses were switched to errors.
* Remove errors.NotSupportedError. It was used in only one place (ca.go), and that usage is more appropriately a ServerInternal error.
This PR implements certificate revocation for the WFE2. This
endpoint differs from others in that it supports *both* traditional key
ID based JWS request authentication in addition to embedded JWK based
JWS request authentication. The first is considered authenticated to
revoke a certificate if the signer account has valid authorizations for
all of the names in the certificate. The second is considered
authenticated if the embedded JWK that signs the request has the same
public key as the certificate being revoked.
Resolves#2952
Prior to this commit if the sa.GetAuthorization found no pending authz
rows and no authz rows for a given authz ID then sql.ErrNoRows
was returned to callers.
This commit changes the SA's GetAuthorization function to transform
sql.ErrNoRows into berrors.NotFound error. The wfe (and wfe2) are
updated to check for the GetAuthorization error being a berrors.NotFound
instance and now handle this correctly with a missing response instead of
a server internal error.
Resolves#3023
The ACME specification no longer describes "registrations" since this is
a fairly overloaded term. Instead the term used is "account". This
commit updates the WFE2 & tests throughout to replace occurrences of
"reg" and "registration" to use "acct" and "account".
NOTE: This change is strictly limited to the wfe2 package. E.g. the
RA/SA and many core objects still refer to registrations.
Resolves#2986
This commit implements certificate revocation for the WFE2. This
endpoint differs from others in that it supports *both* traditional key
ID based JWS request authentication in addition to embedded JWK based
JWS request authentication. The first is considered authenticated to
revoke a certificate if the signer account has valid authorizations for
all of the names in the certificate. The second is considered
authenticated if the embedded JWK that signs the request has the same
public key as the certificate being revoked.
Per #3001 we should not be adding new StatsD code for metrics anymore.
This commit updates all of the WFE2 to use 1st class Prometheus stats.
Unit tests are updated accordingly.
I have broken the error stats into two counts:
1. httpErrorCount for all of the http layer client request errors (e.g.
no POST body, no content-length)
2. joseErrorCount, for all of the JOSE layer client request errors (e.g.
malformed JWS, broken signature, invalid JWK)
This commit also removes the stubbed out `TestValidKeyRollover` function
from `wfe2/verify_test.go`. This was committed accidentally and the same
functionality is covered by the `wfe2/wfe_test.go` `TestKeyRollover`
function.
RFC 7515 section 7.2.1 "General JWS JSON Serialization Syntax" describes
an optional "signatures" field that contains an array of JSON objects,
each representing a signature or MAC. ACME only uses the mandatory
"signature" field that contains the BASE64URL of a signature.
We previously checked that the parsed JWS had only one signature and
rejected accordingly but in order to be safe and ensure that nothing is
read from this "signatures" array when we intended to be using the
"signature" field this commit updates the check to explicitly reject the
"signatures" field prior to parsing with go-jose similar to how the
unprotected header is handled.
This is the second half of a clean-up of the WFE2 unit tests that were copied over from the original WFE implementation.
This PR covers TestChallenge, TestAuthorization, and TestGetCertificate.
Resolves#2928
This is the first half of a clean-up of the WFE2 unit tests that were copied over from the original WFE implementation.
I will file a follow-up pt2 PR for TestChallenge, TestAuthorization, and TestGetCert which I think are the remaining tests that could use a 🛁.
The one non-test commit changed the WFE2 index to return a problem when the method is unsupported similar to the other API endpoints. This might be inappropriate since normally the index returns XHTML and not JSON. It made testing easier but I'm open to switching back to returning a "" body and special casing the index test.
Note to reviewers: The main diff is hidden by GH by default and needs to be expanded.
Updates #2928
Daniel