Commit Graph

704 Commits

Author SHA1 Message Date
Aaron Gable 2fe12cdf20
Unwrap SA Add/Revoke Certificate methods (#5598)
Make the gRPC wrappers for the SA's `AddCertificate`,
`AddPrecertificate`, `AddSerial`, and `RevokeCertificate`
methods simple pass-throughs.

Fixup a couple tests that were passing only because their
requests to in-memory SA objects were not passing through
the wrapper's consistency checks.

Part of #5532
2021-08-25 15:54:25 -07:00
Aaron Gable 22a4e2afec
Add IssuerID field to CertificateStatus proto (#5595)
Add a new field, `issuerID`, to the `core.CertificateStatus` proto
message. Also add this field to the list of fields carried over when
converting back and forth between core and proto objects.

This should be safe to deploy without risk of miscommunication
between components that update at different times. If the SA
updates first and begins sending IssuerIDs to clients, they'll simply
ignore them. If the clients update first and try to extract IssuerIDs
from messages that don't have them, they'll simply get the zero
value, which is what they were getting before anyway.

Part of #5152 and #5532
2021-08-20 13:21:44 -07:00
Aaron Gable 5fcabde592
Make CertificateStatus.IssuerID not a reference (#5594)
Change `CertificateStatus.IssuerID` from `*int64` to just an
`int64`. It might make sense for this field to be nillable in a world
where we want to distinguish between it being missing and it
being zero, but none of our code actually does that: we error
out either way.

Part of #5152
2021-08-20 13:19:29 -07:00
Aaron Gable f454892dd1
Unwrap SA Get[Pre]Certificate methods (#5588)
Make the gRPC wrappers for sa.GetCertificate and
sa.GetPrecertificate bare passthroughs. The latter of
these already took and returned appropriate protobufs,
so this change mostly just makes the former look like the
latter.

Part of #5532
2021-08-19 15:43:48 -07:00
Aaron Gable 3492a996ab
Update TODOs for Issue #5152 (#5591)
Changing how we're going to finally handle #5152: rather
than changing everything to use IssuerNameIDs, we're going
to change the meaning of IssuerID. This will allow us to avoid
renaming database columns and protobuf message fields.
2021-08-19 14:31:09 -07:00
Samantha 53b89707d5
GRPC: Unwrap ra.DeactivateAuthorization (#5567)
- Move `DeactivateAuthorization` logic from `grpc` to `ra` and `wfe`
- Update `ra` mocks in `wfe` tests
- Remove unnecessary marshalling between `core.Authorization` and
  `corepb.Authorization` in `ra` tests.

Fixes #5562
2021-08-12 11:30:57 -07:00
Aaron Gable b7ce627572
Remove SA Registration gRPC wrappers (#5551)
Remove all error checking and type transformation from the gRPC wrappers
for the following methods on the SA:
- GetRegistration
- GetRegistrationByKey
- NewRegistration
- UpdateRegistration
- DeactivateRegistration

Update callers of these methods to construct the appropriate protobuf
request messages directly, and to consume the protobuf response messages
directly. In many cases, this requires changing the way that clients
handle the `Jwk` field (from expecting a `JSONWebKey` to expecting a
slice of bytes) and the `Contacts` field (from expecting a possibly-nil
pointer to relying on the value of the `ContactsPresent` boolean field).

Implement two new methods in `sa/model.go` to convert directly between
database models and protobuf messages, rather than round-tripping
through `core` objects in between. Delete the older methods that
converted between database models and `core` objects, as they are no
longer necessary.

Update test mocks to have the correct signatures, and update tests to
not rely on `JSONWebKey` and instead use byte slices.

Fixes #5531
2021-08-04 13:33:41 -07:00
Samantha 2a5b9f651a
GRPC: Make ra.AdministrativelyRevokeCertificate a pass-through (#5558)
- Move `AdministrativelyRevokeCertificate` logic from `grpc` to `ra`
- Test new error conditions in `ra/ra_test.go`
- Update `ra` mocks in `wfe` tests

Fixes #5529
2021-08-02 13:52:00 -07:00
Andrew Gabbitas f599da27cc
Make ra.NewAuthorization wrapper passthrough (#5553)
Fixes: #5527
2021-08-02 13:09:59 -07:00
Andrew Gabbitas 1681c365aa
Make ra.NewCertificate a passthrough (#5557)
Fixes: #5528
2021-08-02 10:47:09 -07:00
Andrew Gabbitas 9133dba948
Make ra.DeactivateRegistration pass-through (#5522)
Fixes: #5521
2021-07-13 11:40:09 -06:00
Aaron Gable 4c581436a3
Add go1.17beta1 to CI (#5483)
Add go1.17beta1 docker images to the set of things we build,
and integrate go1.17beta1 into the set of environments CI runs.
Fix one test which breaks due to an underlying refactoring in
the `crypto/x509` stdlib package. Fix one other test which breaks
due to new guarantees in the stdlib's TLS ALPN implementation.

Also removes go1.16.5 from CI so we're only running 2 versions.

Fixes #5480
2021-07-13 10:00:04 -07:00
Aaron Gable 64c9ec350d
Unify protobuf generation (#5458)
Create script which finds every .proto file in the repo and correctly
invokes `protoc` for each. Create a single file with a `//go:generate`
directive to invoke the new script. Delete all of the other generate.go
files, so that our proto generation is unified in one place.

Fixes #5453
2021-06-07 08:49:15 -07:00
Aaron Gable 8be32d3312
Use google.protobuf.Empty instead of core.Empty (#5454)
Replace `core.Empty` with `google.protobuf.Empty` in all of our gRPC
methods which consume or return an empty protobuf. The golang core
proto libraries provide an empty message type, so there is no need
for us to reinvent the wheel.

This change is backwards-compatible and does not require a special
deploy. The protobuf message descriptions of `core.Empty` and
`google.protobuf.Empty` are identical, so their wire-formats are
indistinguishable and therefore interoperable / cross-compatible.

Fixes #5443
2021-06-03 14:17:41 -07:00
Andrew Gabbitas 5b235bd8eb
Change ra.UpdateRegistration sig to match grpc (#5449)
Change ra.UpdateRegistration sig to match grpc

Fixes: #5403
2021-06-01 11:55:33 -06:00
Aaron Gable 7455a8a32d
Make RevokeCertificateWithReg wrappers passthroughs (#5445)
Update the signature of the RA's RevokeCertificateWithReg
method to exactly match that of the gRPC method it implements.
Remove all logic from the `RevokeCertificateWithReg` client
and server wrappers. Move the small amount of checking they
were performing directly into the server implementation.

Fixes #5440
2021-06-01 08:42:32 -07:00
Andrew Gabbitas 6b45dce5f1
Make ra.UpdateRegistration wrapper pass-through (#5431)
Part of: #5403
Fixes: #5398
2021-05-28 15:18:54 -06:00
Andrew Gabbitas 59bab8bac4
Make core.Registration.CreatedAt a *time.Time (#5422)
* Make core.Registration.CreatedAt a *time.time

Fixes: #5421
2021-05-21 13:44:56 -06:00
Aaron Gable 7bf854fe03
Move OCSP gRPC service to separate file and struct (#5402)
Create a new `ocspImpl` struct which satisfies the interface required
by the `OCSPGenerator` gRPC service. Move the `GenerateOCSP`
method from the `certificateAuthorityImpl` to this new type. To support
existing gRPC clients, keep a reference to the new OCSP service in
the CA impl, and maintain a pass-through `GenerateOCSP` method.
Simplify some of the CA setup code, and make the CA implementation
non-exported because it doesn't need to be.

In order to maintain our existing signature and sign error metrics,
they now need to be initialized outside the CA and OCSP constructors.
This complicates the tests slightly, but seems like a worthwhile
tradeoff.

Fixes #5226
Fixes #5086
2021-04-29 14:20:39 -07:00
Andrew Gabbitas 5fdacbeaa6
grpc wrapper removal: Turn ra.NewRegistration into passthrough (#5397)
Turn ra.NewRegistration into passthrough

Fixes #5343
2021-04-23 13:53:22 -06:00
Jacob Hoffman-Andrews 7194624191
Update grpc and protobuf to latest. (#5369)
protoc now generates grpc code in a separate file from protobuf code.
Also, grpc servers are now required to embed an "unimplemented"
interface from the generated .pb.go file, which provides forward
compatibility.

Update the generate.go files since the invocation for protoc has changed
with the split into .pb.org and _grpc.pb.go.

Fixes #5368
2021-04-01 17:18:15 -07:00
Aaron Gable 91473b384b
Remove common config from publisher (#5353)
The old `config.Common.CT.IntermediateBundleFilename` format is no
longer used in any production configs, and can be removed safely.

Part of #5162
Part of #5242
Fixes #5269
2021-03-18 16:59:06 -07:00
Aaron Gable 95b77dbd25
Remove va gRPC wrapper (#5328)
Delete the ValidationAuthorityGRPCServer and ...GRPCClient structs,
and update references to instead reference the underlying vapb.VAClient
type directly. Also delete the core.ValidationAuthority interface.

Does not require updating interfaces elsewhere, as the client
wrapper already included the variadic grpc.CallOption parameter.

Fixes #5325
2021-03-11 15:38:50 -08:00
Aaron Gable 1f776ba768
Remove publisher gRPC wrapper (#5327)
Delete the PublisherClientWrapper and PublisherServerWrapper. Update
various structs and functions to expect a pubpb.PublisherClient instead
of a core.Publisher; these two interfaces differ only in that the
auto-generated PublisherClient takes a variadic CallOptions parameter.
Update all mock publishers in tests to match the new interface. Finally,
delete the now-unused core.Publisher interface and some already-unused
mock-generating code.

This deletes a single sanity check (for a nil SCT even when there is a
nil error), but that check was redundant with an identical check in the
only extant client code in ctpolicy.go.

Fixes #5323
2021-03-11 10:50:29 -08:00
Aaron Gable 993953bf81
Remove ca gRPC wrapper (#5330)
Delete the CertificateAuthorityClientWrapper, OCSPGeneratorClientWrapper,
and CertificateAuthorityServerWrapper structs, which provided no error
checking above and beyond their wrapped types. Replace them with the
corresponding auto-generated gRPC types in calling code. Update some
mocks to have the necessary variadic grpc.CallOption parameter. Finally,
delete the now-unused core.CertificateAuthority interface.

Fixes #5324
2021-03-11 10:45:46 -08:00
Jacob Hoffman-Andrews e042b9a47f
Remove some unused types from core. (#5292) 2021-02-22 10:09:10 -08:00
Andrew Gabbitas aa20bcaded
Add validated timestamp to challenges (#5253)
We do not present a validated timestamp in challenges where status = valid
as required by RFC8555.

This change is the first step to presenting challenge timestamps to the
client. It adds a timestamp to each place where we change a challenge to
valid. This only displays in the logs and will not display to the
subscriber because it is not yet stored somewhere retrievable. The next
step will be to store it in the database and then finally present it to
the client.

Part of #5198
2021-01-29 08:07:32 -08:00
Aaron Gable 400bf3a02a
Allow WFEv1 to specify which issuer to use (#5222)
We intend to delete the v1 API (i.e. `wfe` and its associated codepaths)
in the near future, and as such are not giving it new features or
capabilities. However, before then we intend to allow the v2 API to
provide issuance both from our RSA and from our ECDSA intermediates.
The v1 API cannot gain such capability at the same time.

The CA doesn't know which frontend originated any given issuance
request, so we can't simply gate the single- or double-issuer behavior
based on that. Instead, this change introduces the ability for the
WFE (and the RA, which sits between the WFE and the CA) to request
issuance from a specific intermediate. If the specified intermediate is
not available in the CA, issuance will fail. If no intermediate is
specified (as is the case in requests coming from wfe2), it falls back
to selecting the issuer based on the algorithm of the public key to
be signed.

Fixes #5216
2021-01-20 09:22:03 -08:00
Aaron Gable ebba443cad
Remove cmd.LoadCert in favor of core.LoadCert (#5165)
Having both of these very similar methods sitting around
only serves to increase confusion. This removes the last
few places which use `cmd.LoadCert` and replaces them
with `core.LoadCert`, and deletes the method itself.

Fixes #5163
2020-11-10 13:00:46 -08:00
Aaron Gable 96f9bfaab8
Fix IsAnyNilOrZero to correctly handle all numeric types (#5096)
The previous implementation of `IsAnyNilOrZero` did not in fact work,
and its tests did not catch this fact. Within the numeric clause, the
compiler would only instantiate the comparison literal 0 to be one
of the eight possible types. Comparisons against any of the other
seven types would always be false, no matter what value that type
held.

The tests did not catch this because they only tested two literal
values: `0` and `-12.345`, both of which can be `float64`s.

This change updates the utility function to use the `reflect` package,
to ensure that it works correctly. It also updates the test to test
multiple different kinds of numeric values, and removes the code
for handling pointer-to- types, as all of our proto2 code has been
removed.

Finally, it updates the SA wrapper's `RevokeCertificate` method to
correctly not require that `req.Reason` be non-zero: this field can
and often is zero, as that value represents `Unspecified`.

Using the reflect package is a conscious tradeoff. It will be slower
than manually writing out every single case, but it will also be less
prone to error.

Part of #5097
2020-09-23 09:01:20 -07:00
Jacob Hoffman-Andrews bf7c80792d
core: move to proto3 (#5063)
Builds on #5062
Part of #5050
2020-08-31 17:58:32 -07:00
Aaron Gable e2c8f6743a
Introduce new core.AcmeChallenge type (#5012)
ACME Challenges are well-known strings ("http-01", "dns-01", and
"tlsalpn-01") identifying which kind of challenge should be used
to verify control of a domain. Because they are well-known and
only certain values are valid, it is better to represent them as
something more akin to an enum than as bare strings. This also
improves our ability to ensure that an AcmeChallenge is not
accidentally used as some other kind of string in a different
context. This change also brings them closer in line with the
existing core.AcmeResource and core.OCSPStatus string enums.

Fixes #5009
2020-08-11 15:02:16 -07:00
Aaron Gable 0f5d2064a8
Remove logic from VA PerformValidation wrapper (#5003)
Updates the type of the ValidationAuthority's PerformValidation
method to be identical to that of the corresponding auto-generated
grpc method, i.e. directly taking and returning proto message
types, rather than exploded arguments.

This allows all logic to be removed from the VA wrappers, which
will allow them to be fully removed after the migration to proto3.

Also updates all tests and VA clients to adopt the new interface.

Depends on #4983 (do not review first four commits)
Part of #4956
2020-08-06 10:45:35 -07:00
Aaron Gable 634d57ce86
Use 2-space indents in all proto files (#5006)
Our proto files had a variety of indentation styles: 2 spaces,
4 spaces, 8 spaces, and tabs; sometimes mixed within the same
file. The proto3 style guide[1] says to use 2-space indents,
so this change standardizes on that.

[1] https://developers.google.com/protocol-buffers/docs/style
2020-08-05 10:38:19 -07:00
Aaron Gable ffdae2d338
Return proto from ca.IssueCertificateFromPrecertificate (#4982)
This is the only method on the ca which uses a non-proto
type as its request or response value. Changing this to
use a proto removes the last logic from the wrappers,
allowing them to be removed in a future CL. It also makes
the interface more uniform and easier to reason about.

Issue: #4940
2020-07-23 18:39:10 -07:00
Jacob Hoffman-Andrews f9fd977bd6
Remove SerialExists. (#4976)
It no longer... exists.

Fixes #4943
2020-07-22 18:59:34 -07:00
Aaron Gable 12d8674130
Move CA RPC non-nil checks out of wrappers (#4971)
Introduces a new generic helper utility to check that
fields of proto messages are non-nil and non-zero.

Uses this helper to simplify the ca RPC wrapper
methods, moving their completeness checks into
the underlying method handler. Also annotates the
completeness checks to justify which fields are or
are not being checked for future readers. Finally,
removes the similar non-nil checks from the client
wrappers, where they provide no marginal value.

Follow-up changes will do the same for other RPC
services, migrate said services to proto3, and change
the IssueCertificateForPrecertificate method to return
a corepb.Certificate instead of a core.Certificate, like
the other methods on the ca service.

Issues: #4955
2020-07-22 17:42:16 -07:00
Aaron Gable 3a03e86e89
Standardize all proto import names (#4970)
We previously used mixed case names for proto imports
(e.g. both `caPB` and `rapb`), sometimes in the same file.
This change standardizes on the all-lowercase spelling,
which was predominant throughout the codebase.
2020-07-20 16:29:17 -07:00
Aaron Gable 6f0016262f
Simplify database interactions (#4949)
Simplify database interactions

This change is a result of an audit of all places where
Go code directly constructs SQL queries and executes them
against a dbMap, with the goal of eliminating all instances
of constructing a well-known object type (such as a
core.CertificateStatus) from explicitly-listed database columns.
Instead, we should be relying on helper functions defined in the
sa itself to determine which columns are relevant for the
construction of any given object.

This audit did not find many places where this was occurring. It
did reveal a few simplifications, which are contained in this
change:
1) Greater use of existing SelectFoo methods provided by models.go
2) Streamlining of various SelectSingularFoo methods to always
   select by serial string, rather than user-provided WHERE clause
3) One spot (in ocsp-responder) where using a well-known type seemed
   better than using a more minimal custom type

Addresses #4899
2020-07-20 11:12:52 -07:00
Jacob Hoffman-Andrews 6f4966cc0f
Check email address validity in notify-mailer. (#4841)
This required a refactoring: Move validateEmail from the RA to ValidEmail
in the `policy` package. I also moved `ValidDomain` from a method on
PolicyAuthority to a standalone function so that ValidEmail can call it.

notify-mailer will now log invalid addresses and skip them without
attempting to send mail. Since @example.com addresses are invalid,
I updated the notify-mailer test, which used a lot of such addresses.

Also, now when notify-mailer receives an unrecoverable error sending
mail, it logs the email address and what offset within the list it was.
2020-06-04 18:28:02 -07:00
Phil Porada 348e4cb197
Update StorageAuthority interface comment (#4825)
The old comment referenced the StorageUpdater interface which no longer exists.
2020-05-27 11:57:00 -07:00
Jacob Hoffman-Andrews b1347fb3b3
Upgrade to latest protoc and protoc-gen-go (#4794)
There are some changes to the code generated in the latest version, so
this modifies every .pb.go file.

Also, the way protoc-gen-go decides where to put files has changed, so
each generate.go gets the --go_opt=paths=source_relative flag to
tell protoc to continue placing output next to the input.

Remove staticcheck from build.sh; we get it via golangci-lint now.

Pass --no-document to gem install fpm; this is recommended in the fpm docs.
2020-04-23 18:54:44 -07:00
Roland Bracewell Shoemaker 9df97cbf06
Add a blocked keys table, and use it (#4773)
Fixes #4712 and fixes #4711.
2020-04-15 13:42:51 -07:00
Jacob Hoffman-Andrews 0db7d9ff89
Block keys using hex(sha256(spki)). (#4745)
In addition to base64(sha256(spki)).

As part of that, change KeyDigest to return [32]byte, and add KeyDigestB64 which provides the base64-encoded output that KeyDigest used to provide. Also update all call sites.
2020-04-09 09:41:33 -07:00
Jacob Hoffman-Andrews b567adb9f5
Switch to godoc-style comments. (#4760)
* Switch to godoc-style comments.

staticcheck cleanup: https://staticcheck.io/docs/checks#ST1021

* Revert spurious diff.
2020-04-08 17:22:40 -07:00
Jacob Hoffman-Andrews bef02e782a
Fix nits found by staticcheck (#4726)
Part of #4700
2020-03-30 10:20:20 -07:00
Jacob Hoffman-Andrews 3a1a08a10b
Remove unused code. (#4722)
Found by staticcheck.
2020-03-27 11:55:42 -07:00
Roland Bracewell Shoemaker e1c057b84a
ocsp-updater looper simplification (#4659)
Replaces a cmd/ocsp-updater test of core.RetryBackoff with a core one, which is more appropriate in my view, since that is all that it is really testing.

Fixes #4596.
2020-01-31 10:08:38 -08:00
Daniel McCarney fde145ab96
RA: implement stricter email validation. (#4574)
Prev. we weren't checking the domain portion of an email contact address
very strictly in the RA. This updates the PA to export a function that
can be used to validate the domain the same way we validate domain
portions of DNS type identifiers for issuance.

This also changes the RA to use the `invalidEmail` error type in more
places.

A new Go integration test is added that checks these errors end-to-end
for both account creation and account update.
2019-11-22 13:39:31 -05:00
Roland Bracewell Shoemaker b557d870c7 CA/SA: Store issuer info in certificateStatus, use for OCSP generation (#4546)
This avoids needing to send the entire certificate in OCSP generation
RPCs.

Ended up including a few cleanups that made the implementation easier.

Initially I was struggling with how to derive the issuer identification info.
We could just stick the full SPKI hash in certificateStatus, but that takes a
significant amount of space, we could configure unique issuer IDs in the CA
config, but that would require being very careful about keeping the IDs
constant, and never reusing an ID, or we could store issuers in a table in the
database and use that as a lookup table, but that requires figuring out how to
get that info into the table etc. Instead I've just gone with what I found to
be the easiest solution, deriving a stable ID from the cert hash. This means we
don't need to remember to configure anything special and the CA config stays
the same as it is now.

Fixes #4469.
2019-11-18 09:15:29 -05:00
Roland Bracewell Shoemaker b8ee84da7b
Switch GenerateOCSP to directly use protos instead of wrapper (#4549) 2019-11-14 11:10:33 -08:00
Roland Bracewell Shoemaker e402156c1c Revert "Revert "Remove remaining old format authorization code from SA/… (#4502)" (#4524)
This reverts commit dc2ce4ca84.
2019-11-04 09:45:19 -05:00
Roland Bracewell Shoemaker dc2ce4ca84
Revert "Remove remaining old format authorization code from SA/… (#4502)
We need to apply some fixes for bugs introduced in #4476 before it can be deployed, as such we need to revert #4495 as there needs to be a full deploy cycle between these two changes.

This reverts commit 3ae1ae1.

😭
2019-10-23 10:45:29 -07:00
Roland Bracewell Shoemaker 3ae1ae1493 Remove remaining old format authorization code from SA/protos (#4495) 2019-10-23 09:08:38 -04:00
Roland Bracewell Shoemaker 46e0468220 Make authz2 the default storage format (#4476)
This change set makes the authz2 storage format the default format. It removes
most of the functionality related to the previous storage format, except for
the SA fallbacks and old gRPC methods which have been left for a follow-up
change in order to make these changes deployable without introducing
incompatibilities.

Fixes #4454.
2019-10-21 15:29:15 -04:00
Daniel McCarney 1cd9733c24
WFE2: allow revocation of precertificates. (#4433)
When the `features.PrecertificateRevocation` feature flag is enabled the WFE2
will allow revoking certificates for a submitted precertificate. The legacy WFE1
behaviour remains unchanged (as before (pre)certificates issued through the V1
API will be revocable with the V2 API).

Previously the WFE2 vetted the certificate from the revocation request by
looking up a final certificate by the serial number in the requested
certificate, and then doing a byte for byte comparison between the stored and
requested certificate.

Rather than adjust this logic to handle looking up and comparing stored
precertificates against requested precertificates (requiring new RPCs and an
additional round-trip) we choose to instead check the signature on the requested
certificate or precertificate and consider it valid for revocation if the
signature validates with one of the WFE2's known issuers. We trust the integrity
of our own signatures.

An integration test that performs a revocation of a precertificate (in this case
one that never had a final certificate issued due to SCT embedded errors) with
all of the available authentication mechanisms is included.

Resolves https://github.com/letsencrypt/boulder/issues/4414
2019-09-16 16:40:07 -04:00
Jacob Hoffman-Andrews 9906c93217
Generate and store OCSP at precertificate signing time (#4420)
This change adds two tables and two methods in the SA, to store precertificates
and serial numbers.

In the CA, when the feature flag is turned on, we generate a serial number, store it,
sign a precertificate and OCSP, store them, and then return the precertificate. Storing
the serial as an additional step before signing the certificate adds an extra layer of
insurance against duplicate serials, and also serves as a check on database availability.
Since an error storing the serial prevents going on to sign the precertificate, this decreases
the chance of signing something while the database is down.

Right now, neither table has read operations available in the SA.

To make this work, I needed to remove the check for duplicate certificateStatus entry
when inserting a final certificate and its OCSP response. I also needed to remove
an error that can occur when expiration-mailer processes a precertificate that lacks
a final certificate. That error would otherwise have prevented further processing of
expiration warnings.

Fixes #4412

This change builds on #4417, please review that first for ease of review.
2019-09-09 12:21:20 -07:00
Roland Bracewell Shoemaker cba4adac68 SA: Remove CountCertificatesByExactNames RPC (#4318) 2019-07-02 09:30:44 -04:00
Roland Bracewell Shoemaker 352899ba2f Remove RevokeAuthorizationsByDomain/2 functionality (#4302)
* Remove RevokeAuthorizationsByDomain/2 functionality
* Remove old integration test
2019-06-26 15:48:18 -04:00
Roland Bracewell Shoemaker 059112640b wfe: Use RawURLEncoding for authz2 challenge IDs (#4280) 2019-06-21 09:55:10 -04:00
Roland Bracewell Shoemaker acc44498d1 RA: Make RevokeAtRA feature standard behavior (#4268)
Now that it is live in production and is working as intended we can remove
the old ocsp-updater functionality entirely.

Fixes #4048.
2019-06-20 14:32:53 -04:00
Daniel McCarney 7dd176e9a4 Implement suberrors for policy blocked names. (#4234)
When validating a CSR's identifiers, or a new order's identifiers there may be more than one identifier that is blocked by policy. We should return an error that has suberrors identifying each bad identifier individually in this case.

Updates https://github.com/letsencrypt/boulder/issues/4193
Resolves https://github.com/letsencrypt/boulder/issues/3727
2019-05-31 15:00:17 -07:00
Roland Bracewell Shoemaker 6f93942a04 Consistently used stdlib context package (#4229) 2019-05-28 14:36:16 -04:00
Roland Bracewell Shoemaker e839042bae dns: Remove Authorities field from ValidationRecord (#4230) 2019-05-28 14:11:32 -04:00
Daniel McCarney ea9871de1e core: split identifier types into separate package. (#4225)
This will allow implementing sub-problems without creating a cyclic
dependency between `core` and `problems`.

The `identifier` package is somewhat small/single-purpose and in the
future we may want to move more "ACME" bits beyond the `identifier`
types into a dedicated package outside of `core`.
2019-05-23 13:24:41 -07:00
Jacob Hoffman-Andrews 76beffe074 Clean up must staple and precert options in CA (#4201)
Precertificate issuance has been the only supported mode for a while now. This
cleans up the remaining flags in the CA code. The same is true of must staple.

This also removes the IssueCertificate RPC call and its corresponding wrappers,
and removes a lot of plumbing in the CA unittests that was used to test the
situation where precertificate issuance was not enabled.
2019-05-21 15:34:28 -04:00
Daniel McCarney 277ef28eb5 core: remove unused types (#4192)
The core.Buffer type and the core.IdentifierData type seem to be relics from a bygone era.
2019-05-02 10:36:53 -07:00
Roland Bracewell Shoemaker d06c6a5285
New style authorizations: All SA methods (#4134)
This PR implements new SA methods for handling authz2 style authorizations and updates existing SA methods to count and retrieve them where applicable when the `NewAuthorizationSchema` feature is enabled.

Fixes #4093
Fixes #4082
Updates #4078 
Updates #4077
2019-04-24 09:40:38 -07:00
Roland Bracewell Shoemaker 7ff30cf857 Remove account ID in WFE2 if feature enabled (#4160)
This helps encourage ACME client developers to use the RFC 8555
specified `Location` header instead.

Fixes #4136.
2019-04-17 13:38:25 -04:00
Jacob Hoffman-Andrews 935df44851 Move "Combinations" support to WFE1. (#4155)
Early ACME drafts supported a notion of "combinations" of challenges
that had to be completed together. This was removed from subsequent
drafts. Boulder has only ever supported "combinations" that exactly map
to the list of challenges, 1 for 1.

This removes all the plumbing for combinations, and adds a list of
combinations to the authz JSON right before marshaling it in WFE1.
2019-04-16 11:31:51 -07:00
Jacob Hoffman-Andrews d2d5f0a328 Update miekg/dns and golang/protobuf. (#4150)
Precursor to #4116. Since some of our dependencies impose a minimum
version on these two packages higher than what we have in Godeps, we'll
have to bump them anyhow. Bumping them independently of the modules
update should keep things a little simpler.

In order to get protobuf tests to pass, I had to update protoc-gen-go in
boulder-tools. Now we download a prebuilt binary instead of using the
Ubuntu package, which is stuck on 3.0.0. This also meant I needed to
re-generate our pb.go files, since the new version generates somewhat
different output.

This happens to change the tag for pbutil, but it's not a substantive change - they just added a tagged version where there was none.

$ go test github.com/miekg/dns/...
ok      github.com/miekg/dns    4.675s
ok      github.com/miekg/dns/dnsutil    0.003s

ok      github.com/golang/protobuf/descriptor   (cached)
ok      github.com/golang/protobuf/jsonpb       (cached)
?       github.com/golang/protobuf/jsonpb/jsonpb_test_proto     [no test files]
ok      github.com/golang/protobuf/proto        (cached)
?       github.com/golang/protobuf/proto/proto3_proto   [no test files]
?       github.com/golang/protobuf/proto/test_proto     [no test files]
ok      github.com/golang/protobuf/protoc-gen-go        (cached)
?       github.com/golang/protobuf/protoc-gen-go/descriptor     [no test files]
ok      github.com/golang/protobuf/protoc-gen-go/generator      (cached)
ok      github.com/golang/protobuf/protoc-gen-go/generator/internal/remap       (cached)
?       github.com/golang/protobuf/protoc-gen-go/grpc   [no test files]
?       github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok      github.com/golang/protobuf/ptypes       (cached)
?       github.com/golang/protobuf/ptypes/any   [no test files]
?       github.com/golang/protobuf/ptypes/duration      [no test files]
?       github.com/golang/protobuf/ptypes/empty [no test files]
?       github.com/golang/protobuf/ptypes/struct        [no test files]
?       github.com/golang/protobuf/ptypes/timestamp     [no test files]
?       github.com/golang/protobuf/ptypes/wrappers      [no test files]
2019-04-09 09:27:28 -07:00
Jacob Hoffman-Andrews 677b9b88ad Remove GSB support. (#4115)
This is no longer enabled in prod; cleaning up the code.

https://community.letsencrypt.org/t/let-s-encrypt-no-longer-checking-google-safe-browsing/82168
2019-03-15 10:24:44 -07:00
Jacob Hoffman-Andrews d1e6d0f190 Remove TLS-SNI-01 (#4114)
* Remove the challenge whitelist
* Reduce the signature for ChallengesFor and ChallengeTypeEnabled
* Some unit tests in the VA were changed from testing TLS-SNI to testing the same behavior
  in TLS-ALPN, when that behavior wasn't already tested. For instance timeouts during connect 
  are now tested.

Fixes #4109
2019-03-15 09:05:24 -04:00
Roland Bracewell Shoemaker 317ea227d7 Remove UpdatePendingAuthorization (#4098)
This SA method/RPC is no longer actually used anywhere.

Fixes #3932.
2019-03-07 15:52:28 -05:00
Roland Bracewell Shoemaker 51f29b9953
Implement WFE retrieval logic for v2 authorizations (#4085)
This changeset implements the logic required for the WFE to retrieve v2 authorizations and their associated challenges while still maintaining the logic to retrieve old authorizations/challenges. Challenge IDs for v2 authorizations are obfuscated using a pretty simply scheme in order to prevent hard coding of indexes. A `V2` field is added to the `core.Authorization` object and populated using the existing field of the same name from the protobuf for convenience. v2 authorizations and challenges use a `v2` prefix in all their URLs in order to easily differentiate between v1 and v2 URLs (e.g. `/acme/authz/v2/asd` and `/acme/challenge/v2/asd/123`), once v1 authorizations cease to exist this prefix can be safely removed. As v2 authorizations use int IDs this change switches from string IDs to int IDs, this mainly only effects tests.

Integration tests are put off for #4079 as they really need #4077 and #4078 to be properly effective.

Fixes #4041.
2019-02-26 13:14:05 -08:00
Roland Bracewell Shoemaker 3e54cea295 Implement direct revocation at RA (#4043)
Implements a feature that enables immediate revocation instead of marking a certificate revoked and waiting for the OCSP-Updater to generate the OCSP response. This means that as soon as the request returns from the WFE the revoked OCSP response should be available to the user. This feature requires that the RA be configured to use the standalone Akamai purger service.

Fixes #4031.
2019-02-14 14:47:42 -05:00
Roland Bracewell Shoemaker 93ac7fbe9e
Modify authorization creation to allow for new style storage schema (#3998)
Adds a feature which gates creation of authorizations following the style required for the new schema (and which can be used for gating the reset of our new schema code later down the road).

There was an internal discussion about an issue this creates regarding a predictable ordering of challenges within a challenge due to sequential challenge IDs which will always be static for each challenge type. It was suggested we could add some kind of obfuscation to the challenge ID when presented to the user to prevent this. This hasn't been done in this PR as it would only be focused in the WFE and would be better suited as its own changeset.

Fixes #3981.
2019-01-17 17:09:38 -08:00
Daniel McCarney b0f407dcf0 RA: Remove deprecated UpdateAuthorization RPC. (#3993)
Staging and prod both deployed the PerformValidationRPC feature flag. All running WFE/WFE2 instances are using the more accurately named PerformValidation RPC and we can strip out the old UpdateAuthorization bits. The feature flag for PerformValidationRPC remains until we clean up the staging/prod configs.

Resolves #3947 and completes the last of #3930
2019-01-07 16:35:27 -08:00
Roland Bracewell Shoemaker 9afa0f7f19
Add models and store/load methods for authz2 (#3967)
Fixes #3965 and fixes #3949.

This change adds a model for the authz2 style authorization storage and implements methods to transform to/from the protobuf representations and SA methods to store and retrieve the new style authorizations.
2018-12-18 14:03:21 -08:00
Daniel McCarney 8f5de538c1
RA: Add PerformValidation RPC to replace UpdateAuthorization. (#3942)
The existing RA `UpdateAuthorization` RPC needs replacing for
two reasons:

1. The name isn't accurate - `PerformValidation` better captures
the purpose of the RPC.
2. The `core.Challenge` argument is superfluous since Key 
Authorizations are not sent in the initiation POST from the client 
anymore. The corresponding unmarshal and verification is now 
removed. Notably this means broken clients that were POSTing
the wrong thing and failing pre-validation will now likely fail 
post-validation.

To remove `UpdateAuthorization` the new `PerformValidation` 
RPC is added alongside the old one. WFE and WFE2 are 
updated to use the new RPC when the perform validation
feature flag is enabled. We can remove 
`UpdateAuthorization` and its associated wrappers once all 
WFE instances have been updated.

Resolves https://github.com/letsencrypt/boulder/issues/3930
2018-11-28 10:12:47 -05:00
Roland Bracewell Shoemaker e27f370fd3 Excise code relating to pre-SCT embedding issuance flow (#3769)
Things removed:

* features.EmbedSCTs (and all the associated RA/CA/ocsp-updater code etc)
* ca.enablePrecertificateFlow (and all the associated RA/CA code)
* sa.AddSCTReceipt and sa.GetSCTReceipt RPCs
* publisher.SubmitToCT and publisher.SubmitToSingleCT RPCs

Fixes #3755.
2018-06-28 08:33:05 -04:00
Maciej Dębski bb9ddb124e Implement TLS-ALPN-01 and integration test for it (#3654)
This implements newly proposed TLS-ALPN-01 validation method, as described in
https://tools.ietf.org/html/draft-ietf-acme-tls-alpn-01 This challenge type is disabled 
except in the config-next tree.
2018-06-06 13:04:09 -04:00
Kyle Spiers dd0e0249e5 core/util: ValidSerial should return false if the serial is not 32 or 36 (#3712)
The current check always fails because a length can't be simultaneously be both less than 32 and greater than 36.
2018-05-24 15:31:06 -04:00
Joel Sing 8ebdfc60b6 Provide formatting logger functions. (#3699)
A very large number of the logger calls are of the form log.Function(fmt.Sprintf(...)).
Rather than sprinkling fmt.Sprintf at every logger call site, provide formatting versions
of the logger functions and call these directly with the format and arguments.

While here remove some unnecessary trailing newlines and calls to String/Error.
2018-05-10 11:06:29 -07:00
Daniel McCarney 041cd26738
SA: Remove unused `CountCertificateRange` RPC. (#3676)
Now that #3638 has been deployed to all of the RA instances there are no
more RPC clients using the SA's `CountCertificatesRange` RPC.

This commit deletes the implementation, the RPC definition & wrappers,
and all the test code/mocks.
2018-05-01 15:39:45 -04:00
Jacob Hoffman-Andrews 4bea2d8a9a Fix ocsp-updater backoff jitter test. (#3670)
Previously this test assumed jitter could only be positive, but it can
be negative too. This fixes the expected bounds.
2018-04-26 08:11:33 -04:00
Daniel McCarney f8f9a158c7 orphan-finder: set cert issued date based on notbefore. (#3651)
The Boulder orphan-finder command uses the SA's AddCertificate RPC to add orphaned certificates it finds back to the DB. Prior to this commit this RPC always set the core.Certificate.Issued field to the
current time. For the orphan-finder case this meant that the Issued date would incorrectly be set to when the certificate was found, not when it was actually issued. This could cause cert-checker to alarm based on the unusual delta between the cert NotBefore and the core.Certificate.Issued value.

This PR updates the AddCertificate RPC to accept an optional issued timestamp in the request arguments. In the SA layer we address deployability concerns by setting a default value of the current time when none is explicitly provided. This matches the classic behaviour and will let an old RA communicate with a new SA.

This PR updates the orphan-finder to provide an explicit issued time to sa.AddCertificate. The explicit issued time is calculated using the found certificate's NotBefore and the configured backdate.
This lets the orphan-finder set the true issued time in the core.Certificate object, avoiding any cert-checker alarms.

Resolves #3624
2018-04-19 10:25:12 -07:00
Daniel ac6672bc71
Revert "Revert "V2: implement "ready" status for Order objects (#3614)" (#3643)"
This reverts commit 3ecf841a3a.
2018-04-12 13:20:47 -04:00
Jacob Hoffman-Andrews 3ecf841a3a Revert "V2: implement "ready" status for Order objects (#3614)" (#3643)
This reverts commit 1d22f47fa2.

According to
https://github.com/letsencrypt/boulder/pull/3614#issuecomment-380615172,
this broke Certbot's tests. We'll investigate, and then roll forward
once we understand what broke.
2018-04-12 10:46:57 -04:00
Daniel McCarney 1d22f47fa2 V2: implement "ready" status for Order objects (#3614)
* SA: Add Order "Ready" status, feature flag.

This commit adds the new "Ready" status to `core/objects.go` and updates
`sa.statusForOrder` to use it conditionally for orders with all valid
authorizations that haven't been finalized yet. This state is used
conditionally based on the `features.OrderReadyStatus` feature flag
since it will likely break some existing clients that expect status
"Processing" for this state. The SA unit test for `statusForOrder` is
updated with a "ready" status test case.

* RA: Enforce order ready status conditionally.

This commit updates the RA to conditionally expect orders that are being
finalized to be in the "ready" status instead of "pending". This is
conditionally enforced based on the `OrderReadyStatus` feature flag.
Along the way the SA was changed to calculate the order status for the
order returned in `sa.NewOrder` dynamically now that it could be
something other than "pending".

* WFE2: Conditionally enforce order ready status for finalization.

Similar to the RA the WFE2 should conditionally enforce that an order's
status is either "ready" or "pending" based on the "OrderReadyStatus"
feature flag.

* Integration: Fix `test_order_finalize_early`.

This commit updates the V2 `test_order_finalize_early` test for the
"ready" status. A nice side-effect of the ready state change is that we
no longer invalidate an order when it is finalized too soon because we
can reject the finalization in the WFE. Subsequently the
`test_order_finalize_early` testcase is also smaller.

* Integration: Test classic behaviour w/o feature flag.

In the previous commit I fixed the integration test for the
`config/test-next` run that has the `OrderReadyStatus` feature flag set
but broke it for the `config/test` run without the feature flag.

This commit updates the `test_order_finalize_early` test to work
correctly based on the feature flag status in both cases.
2018-04-11 10:31:25 -07:00
Daniel McCarney 8a03a6848e RA: Log Authz ID and solved-by Chal type at Issuance (#3601)
This PR updates the RA such that certificateRequestEvent objects created during issuance and written to the audit log as JSON also include a new Authorizations field. This field is a map of the form map[string]certificateRequestAuthz and can be used to map from an identifier name appearing in the associated certificate to a certificateRequestAuthz object. Each of the certificateRequestAuthz objects holds an authorization ID and the type of challenge that made the authorization valid.

Example Audit log output (with the JSON pulled out and pretty-printed):

{
 "ID":"0BjPk94KlxExRRIQ3kslRVSJ68KMaTh416chRvq0wyA",
 "Requester":666,
  "SerialNumber":"ff699d91cab5bc84f1bc97fc71e4e27abc1a",
  "VerifiedFields":["subject.commonName","subjectAltName"],
  "CommonName":"rand.44634cbf.xyz",
  "Names":["rand.44634cbf.xyz"],
  "NotBefore":"2018-03-28T19:50:07Z",
  "NotAfter":"2018-06-26T19:50:07Z",
  "RequestTime":"2018-03-28T20:50:07.234038859Z",
  "ResponseTime":"2018-03-28T20:50:07.278848954Z",
  "Authorizations":  {
    "rand.44634cbf.xyz" : {
      "ID":"jGt37Rnvfr0nhZn-wLkxrQxc2HBfV4t6TSraRiGnNBM",
      "ChallengeType":"http-01"
    }
  }
}

Resolves #3253
2018-04-04 20:59:38 +01:00
Roland Bracewell Shoemaker 9c9e944759 Add SCT embedding (#3521)
Adds SCT embedding to the certificate issuance flow. When a issuance is requested a precertificate (the requested certificate but poisoned with the critical CT extension) is issued and submitted to the required CT logs. Once the SCTs for the precertificate have been collected a new certificate is issued with the poison extension replace with a SCT list extension containing the retrieved SCTs.

Fixes #2244, fixes #3492 and fixes #3429.
2018-03-12 11:58:30 -07:00
Daniel McCarney 2612bf7168 Remove deprecated `sa.CountPendingOrders` cruft. (#3527)
#3501 made this code deprecated. We've deployed 3501 to the staging environment and can now pull out the old cruft.

Resolves #3502
2018-03-06 21:20:40 +00:00
Daniel McCarney 49d55d9ab5 Make POSTing KeyAuthorization optional, V2 don't echo it. (#3526)
This commit updates the RA to make the notion of submitting
a KeyAuthorization value as part of the ra.UpdateAuthorization call
optional. If set, the value is enforced against expected and an error is
returned if the provided authorization isn't correct. If it isn't set
the RA populates the field with the computed authorization for the VA to
enforce against the value it sees in challenges. This retains the legacy
behaviour of the V1 API. The V2 API will never unmarshal a provided
key authorization.

The ACMEv2/WFEv2 prepChallengeForDisplay function is updated to strip
the ProvidedKeyAuthorization field before sending the challenge object
back to a client. ACMEv1/WFEv1 continue to return the KeyAuthorization
in challenges to avoid breaking clients that are relying on this legacy
behaviour.

For deployability ease this commit retains the name of the
core.Challenge.ProvidedKeyAuthorization field even though it should
be called core.Challenge.ComputedKeyAuthorization now that it isn't
set based on the client's provided key authz. This will be easier as
a follow-up change.

Resolves #3514
2018-03-06 20:33:01 +00:00
Daniel McCarney 28cc969814 Remove TLS-SNI-02 implementation. (#3516)
This code was never enabled in production. Our original intent was to
ship this as part of the ACMEv2 API. Before that could happen flaws were
identified in TLS-SNI-01|02 that resulted in TLS-SNI-02 being removed
from the ACME protocol. We won't ever be enabling this code and so we
might as well remove it.
2018-03-02 10:56:13 -08:00
Daniel McCarney f2d3ad6d52 Enforce new orders per acct per window rate limit. (#3501)
Previously we introduced the concept of a "pending orders per account
ID" rate limit. After struggling with making an implementation of this
rate limit perform well we reevaluated the problem and decided a "new
orders per account per time window" rate limit would be a better fit for
ACMEv2 overall.

This commit introduces the new newOrdersPerAccount rate limit. The RA
now checks this before creating new pending orders in ra.NewOrder. It
does so after order reuse takes place ensuring the rate limit is only
applied in cases when a distinct new pending order row would be created.
To accomplish this a migration for a new orders field (created) and an
index over created and registrationID is added. It would be possible to
use the existing expires field for this like we've done in the past, but that
was primarily to avoid running a migration on a large table in prod. Since
we don't have that problem yet for V2 tables we can Do The Right Thing
and add a column.

For deployability the deprecated pendingOrdersPerAccount code & SA
gRPC bits are left around. A follow-up PR will be needed to remove
those (#3502).

Resolves #3410
2018-03-02 10:47:39 -08:00
Jacob Hoffman-Andrews 9003dd4522 Improve test-tools. (#3481)
Use `t.Helper` and `t.Fatalf` instead of our own versions.

Remove some unused or single-user helpers.

Make the output of `AssetUnmarshaledEquals` clearer by showing one line per field.
2018-02-27 10:19:24 -05:00
Daniel McCarney 04b2b17db3 Remove deprecated `sa.GetOrderAuthorizations`. (#3470)
It has been replaced by `sa.GetValidOrderAuthorizations`, the same RPC
with a clearer name.

Resolves #3424
2018-02-21 11:59:46 -08:00
Daniel McCarney d7bfb542c0
Handle order finalization errors. (#3404)
This commit resolves the case where an error during finalization occurs.
Prior to this commit if an error (expected or otherwise) occurred after
setting an order to status processing at the start of order
finalization the order would be stuck processing forever.

The SA now has a `SetOrderError` RPC that can be used by the RA to
persist an error onto an order. The order status calculation can use
this error to decide if the order is invalid. The WFE is updated to
write the error to the order JSON when displaying the order information.

Prior to this commit the order protobuf had the error field as
a `[]byte`. It doesn't seem like this is the right decision, we have
a specific protobuf type for ProblemDetails and so this commit switches
the error field to use it. The conversion to/from `[]byte` is done with
the model by the SA.

An integration test is included that prior to this commit left an order
in a stuck processing state. With this commit the integration test
passes as expected.

Resolves https://github.com/letsencrypt/boulder/issues/3403
2018-02-07 16:34:07 -05:00
Daniel McCarney 67ae7f75b4 `sa.GetOrderAuthorizations` -> `sa.GetValidOrderAuthorizations`. (#3411)
The SA RPC previously called `GetOrderAuthorizations` only returns
**valid, unexpired** authorizations. This commit updates the name to
emphasize that it only returns valid order authzs.
2018-02-07 11:54:18 -08:00
Roland Bracewell Shoemaker 62f3978f3b
Add inital CTPolicy impl (#3414)
Adds a package which implements group based SCT retrieval.

Fixes #3412.
2018-02-06 10:52:20 -08:00
Daniel McCarney eea049da40
Fix order reuse, calc order status by authz status (#3402)
This PR is a rework of what was originally https://github.com/letsencrypt/boulder/pull/3382, integrating the design feedback proposed by @jsha: https://github.com/letsencrypt/boulder/pull/3382#issuecomment-359912549 

This PR removes the stored Order status field and replaces it with a value that is calculated on-the-fly by the SA when fetching an order, based on the order's associated authorizations. 

In summary (and order of precedence):
* If any of the order's authorizations are invalid, the order is invalid.
* If any of the order's authorizations are deactivated, the order is deactivated.
* If any of the order's authorizations are pending, the order is pending.
* If all of the order's authorizations are valid, and there is a certificate serial, the order is valid.
* If all of the order's authorizations are valid, and we have began processing, but there is no certificate serial, the order is processing.
* If all of the order's authorizations are valid, and we haven't processing, then the order is pending waiting a finalization request.

This avoids having to explicitly update the order status when an associated authorization changes status.

The RA's implementation of new-order is updated to only reuse an existing order if the calculated status is pending. This avoids giving back invalid or deactivated orders to clients.

Resolves #3333
2018-02-01 16:33:42 -05:00
Roland Bracewell Shoemaker 8f168b909e Omit empty fields in JSON representation of core.ValidationRecord struct (#3399) 2018-01-29 10:51:57 -08:00
Jacob Hoffman-Andrews 8153b919be
Implement TLSSNIRevalidation (#3361)
This change adds a feature flag, TLSSNIRevalidation. When it is enabled, Boulder
will create new authorization objects with TLS-SNI challenges if the requesting
account has issued a certificate with the relevant domain name, and was the most
recent account to do so*. This setting overrides the configured list of
challenges in the PolicyAuthority, so even if TLS-SNI is disabled in general, it
will be enabled for revalidation.

Note that this interacts with EnforceChallengeDisable. Because
EnforceChallengeDisable causes additional checked at validation time and at
issuance time, we need to update those two places as well. We'll send a
follow-up PR with that.

*We chose to make this work only for the most recent account to issue, even if
there were overlapping certificates, because it significantly simplifies the
database access patterns and should work for 95+% of cases.

Note that this change will let an account revalidate and reissue for a domain
even if the previous issuance on that account used http-01 or dns-01. This also
simplifies implementation, and fits within the intent of the mitigation plan: If
someone previously issued for a domain using http-01, we have high confidence
that they are actually the owner, and they are not going to "steal" the domain
from themselves using tls-sni-01.

Also note: This change also doesn't work properly with ReusePendingAuthz: true.
Specifically, if you attempted issuance in the last couple days and failed
because there was no tls-sni challenge, you'll still have an http-01 challenge
lying around, and we'll reuse that; then your client will fail due to lack of
tls-sni challenge again.

This change was joint work between @rolandshoemaker and @jsha.
2018-01-12 11:00:06 -08:00
Maciej Dębski 44984cd84a Implement regID whitelist for allowed challenge types. (#3352)
This updates the PA component to allow authorization challenge types that are globally disabled if the account ID owning the authorization is on a configured whitelist for that challenge type.
2018-01-10 13:44:53 -05:00
Roland Shoemaker dcd2b438f4 Fix previous impl, add valid authz reuse fix and existing authz validation fix 2018-01-09 19:53:48 -08:00
Roland Shoemaker 5ca646c5dd Disallow the use of valid authorizations that used currently disabled challenges for issuance 2018-01-09 18:52:29 -08:00
Daniel McCarney 7bb16ff21e ACMEv2: Add pending order reuse (#3290)
This commit adds pending order reuse. Subsequent to this commit multiple
add-order requests from the same account ID for the same set of order
names will result in only one order being created. Orders are only
reused while they are not expired. Finalized orders will not be reused
for subsequent new-order requests allowing for duplicate order issuance.

Note that this is a second level of reuse, building on the pending
authorization reuse that's done between separate orders already.

To efficiently find an appropriate order ID given a set of names,
a registration ID, and the current time a new orderFqdnSets table is
added with appropriate indexes and foreign keys.

Resolves #3258
2018-01-02 13:27:16 -08:00
Daniel McCarney ca54d9758f More small WFE2 fixes (#3296)
- Fixes two places WFE2 returned http.StatusAccepted when it should return http.StatusOK
- Removes the legacy challenge "URI" field in place of "URL"
2017-12-20 10:34:03 -08:00
Jacob Hoffman-Andrews 52bb0aa7ba Use lowercase for the "status" field of registrations (#3293) 2017-12-19 15:48:13 -08:00
Daniel McCarney 0684d5fc73
Add pending orders rate limit to new-order. (#3257)
This commit adds a new rate limit to restrict the number of outstanding
pending orders per account. If the threshold for this rate limit is
crossed subsequent new-order requests will return a 429 response.

Note: Since this the rate limit object itself defines an `Enabled()`
test based on whether or not it has been configured there is **not**
a feature flag for this change.

Resolves https://github.com/letsencrypt/boulder/issues/3246
2017-12-04 16:36:48 -05:00
Daniel McCarney 1c99f91733 Policy based issuance for wildcard identifiers (Round two) (#3252)
This PR implements issuance for wildcard names in the V2 order flow. By policy, pending authorizations for wildcard names only receive a DNS-01 challenge for the base domain. We do not re-use authorizations for the base domain that do not come from a previous wildcard issuance (e.g. a normal authorization for example.com turned valid by way of a DNS-01 challenge will not be reused for a *.example.com order).

The wildcard prefix is stripped off of the authorization identifier value in two places:

When presenting the authorization to the user - ACME forbids having a wildcard character in an authorization identifier.
When performing validation - We validate the base domain name without the *. prefix.
This PR is largely a rewrite/extension of #3231. Instead of using a pseudo-challenge-type (DNS-01-Wildcard) to indicate an authorization & identifier correspond to the base name of a wildcard order name we instead allow the identifier to take the wildcard order name with the *. prefix.
2017-12-04 12:18:10 -08:00
Daniel McCarney 2f263f8ed5 ACME v2 Finalize order support (#3169)
This PR implements order finalization for the ACME v2 API.

In broad strokes this means:

* Removing the CSR from order objects & the new-order flow
* Adding identifiers to the order object & new-order
* Providing a finalization URL as part of orders returned by new-order
* Adding support to the WFE's Order endpoint to receive finalization POST requests with a CSR
* Updating the RA to accept finalization requests and to ensure orders are fully validated before issuance can proceed
* Updating the SA to allow finding order authorizations & updating orders.
* Updating the CA to accept an Order ID to log when issuing a certificate corresponding to an order object

Resolves #3123
2017-11-01 12:39:44 -07:00
Roland Bracewell Shoemaker b7bca87134 Batch fetching of existing authorizations and creation of pending authorizations (#3058)
For the new-order endpoint only. This does some refactoring of the order of operations in `ra.NewAuthorization` as well in order to reduce the duplication of code relating to creating pending authorizations, existing tests still seem to work as intended... A close eye should be given to this since we don't have integration tests yet that test it end to end. This also changes the inner type of `grpc.StorageAuthorityServerWrapper` to `core.StorageAuthority` so that we can avoid a circular import that is created by needing to import `grpc.AuthzToPB` and `grpc.PBToAuthz` in `sa/sa.go`.

This is a big change but should considerably improve the performance of the new-order flow.

Fixes #2955.
2017-09-25 09:10:59 -07:00
Roland Bracewell Shoemaker e91349217e Switch to using go 1.9 (#3047)
* Switch to using go 1.9

* Regenerate with 1.9

* Manually fix import path...

* Upgrade mockgen and regenerate

* Update github.com/golang/mock
2017-09-06 16:30:13 -04:00
Jacob Hoffman-Andrews 18f15b2b3d Remove unused error types (#3041)
* Remove all of the errors under core. Their purpose is now served by errors, and they were almost entirely unused. The remaining uses were switched to errors.
* Remove errors.NotSupportedError. It was used in only one place (ca.go), and that usage is more appropriately a ServerInternal error.
2017-09-05 16:51:32 -07:00
Roland Bracewell Shoemaker 191a043585 Implement handler for retrieving an order object and SA RPC (#3016)
Fixes #2984 and fixes #2985.
2017-09-01 15:26:36 -07:00
Roland Bracewell Shoemaker c560fa4fc5 Remove features checks from wfe2 (#2982) 2017-08-16 10:53:47 -04:00
Brian Smith e670e6e6b5 CA: Stub IssueCertificateForPrecertificate(). (#2973)
Stub out IssueCertificateForPrecertificate() enough so that we can continue with the PRs that implement & test it in parallel with PRs that implement and test the calling side (via mock implementations of the CA side).
2017-08-15 16:50:21 -07:00
Roland Bracewell Shoemaker 90ba766af9 Add NewOrder RPCs + methods to SA and RA (#2907)
Fixes #2875, #2900 and #2901.
2017-08-11 14:24:25 -04:00
Brian Smith d2291f6c5a CA: Implement IssuePrecertificate. (#2946)
* CA: Stub IssuePrecertificate gPRC method.

* CA: Implement IssuePrecertificate.

* CA: Test Precertificate flow in TestIssueCertificate().

move verification of certificate storage

IssuePrecertificate tests

Add CT precertificate poison extension to CFSSL whitelist.

CFSSL won't allow us to add an extension to a certificate unless that
certificate is in the whitelist.

According to its documentation, "Extensions requested in the CSR are
ignored, except for those processed by ParseCertificateRequest (mainly
subjectAltName)." Still, at least we need to add tests to make sure a
poison extension in a CSR isn't copied into the final certificate.

This allows us to avoid making invasive changes to CFSSL.

* CA: Test precertificate issuance in TestInvalidCSRs().

* CA: Only support IssuePrecertificate() if it is explicitly enabled.

* CA: Test that we produce CT poison extensions in the valid form.

The poison extension must be critical in order to work correctly. It probably wouldn't
matter as much what the value is, but the spec requires the value to be ASN.1 NULL, so
verify that it is.
2017-08-09 21:05:39 -07:00
Roland Bracewell Shoemaker fcef38f78c Performance and cleanup database migration (#2882)
Switch certificates and certificateStatus to use autoincrement primary keys to avoid performance problems with clustered indexes (fixes #2754).

Remove empty externalCerts and identifierData tables (fixes #2881).

Make progress towards deleting unnecessary LockCol and subscriberApproved fields (#856,  #873) by making them NULLable and not including them in INSERTs and UPDATEs.
2017-07-26 15:18:28 -07:00
Jacob Hoffman-Andrews 8bc1db742c Improve recycling of pending authzs (#2896)
The existing ReusePendingAuthz implementation had some bugs:

It would recycle deactivated authorizations, which then couldn't be fulfilled. (#2840)
Since it was implemented in the SA, it wouldn't get called until after the RA checks the Pending Authorizations rate limit. Which means it wouldn't fulfill its intended purpose of making accounts less likely to get stuck in a Pending Authorizations limited state. (#2831)
This factors out the reuse functionality, which used to be inside an "if" statement in the SA. Now the SA has an explicit GetPendingAuthorization RPC, which gets called from the RA before calling NewPendingAuthorization. This happens to obsolete #2807, by putting the recycling logic for both valid and pending authorizations in the RA.
2017-07-26 14:00:30 -07:00
Daniel McCarney 2a84bc2495 Replace go-jose v1 with go-jose v2. (#2899)
This commit replaces the Boulder dependency on
gopkg.in/square/go-jose.v1 with gopkg.in/square/go-jose.v2. This is
necessary both to stay in front of bitrot and because the ACME v2 work
will require a feature from go-jose.v2 for JWS validation.

The largest part of this diff is cosmetic changes:

Changing import paths
jose.JsonWebKey -> jose.JSONWebKey
jose.JsonWebSignature -> jose.JSONWebSignature
jose.JoseHeader -> jose.Header
Some more significant changes were caused by updates in the API for
for creating new jose.Signer instances. Previously we constructed
these with jose.NewSigner(algorithm, key). Now these are created with
jose.NewSigner(jose.SigningKey{},jose.SignerOptions{}). At present all
signers specify EmbedJWK: true but this will likely change with
follow-up ACME V2 work.

Another change was the removal of the jose.LoadPrivateKey function
that the wfe tests relied on. The jose v2 API removed these functions,
moving them to a cmd's main package where we can't easily import them.
This function was reimplemented in the WFE's test code & updated to fail
fast rather than return errors.

Per CONTRIBUTING.md I have verified the go-jose.v2 tests at the imported
commit pass:

ok      gopkg.in/square/go-jose.v2      14.771s
ok      gopkg.in/square/go-jose.v2/cipher       0.025s
?       gopkg.in/square/go-jose.v2/jose-util    [no test files]
ok      gopkg.in/square/go-jose.v2/json 1.230s
ok      gopkg.in/square/go-jose.v2/jwt  0.073s

Resolves #2880
2017-07-26 10:55:14 -07:00
Brian Smith ac63c78313 CA: Have IssueCertificate use IssueCertificateRequest directly. (#2886)
This is a step towards the long-term goal of eliminating wrappers and a step
towards the short-term goal of making it easier to refactor ca/ca_test.go to
add testing of precertificate-based issuance.
2017-07-25 13:35:25 -04:00
Roland Bracewell Shoemaker 764658ab84 Remove CAA distributed resolver (#2804)
We never used it and it's been superseded by the multi-VA design.
2017-06-15 13:35:50 -04:00
Daniel McCarney fbd87b1757 Splits CountRegistrationsByIP to exact-match and by /48. (#2782)
Prior to this PR the SA's `CountRegistrationsByIP` treated IPv6
differently than IPv4 by counting registrations within a /48 for IPv6 as
opposed to exact matches for IPv4. This PR updates
`CountRegistrationsByIP` to treat IPv4 and IPv6 the
same, always matching exactly. The existing RegistrationsPerIP rate
limit policy will be applied against this exact matching count.

A new `CountRegistrationsByIPRange` function is added to the SA that
performs the historic matching process, e.g. for IPv4 it counts exactly
the same as `CountRegistrationsByIP`, but for IPv6 it counts within
a /48.

A new `RegistrationsPerIPRange` rate limit policy is added to allow
configuring the threshold/window for the fuzzy /48 matching registration
limit. Stats for the "Exceeded" and "Pass" events for this rate limit are
separated into a separate `RegistrationsByIPRange` stats scope under
the `RateLimit` scope to allow us to track it separate from the exact 
registrations per IP rate limit.

Resolves https://github.com/letsencrypt/boulder/issues/2738
2017-05-30 15:12:20 -07:00
Daniel McCarney 47452d6c6c Prefer IPv6 addresses, fall back to IPv4. (#2715)
This PR introduces a new feature flag "IPv6First". 

When the "IPv6First" feature is enabled the VA's HTTP dialer and TLS SNI
(01 and 02) certificate fetch requests will attempt to automatically
retry when the initial connection was to IPv6 and there is an IPv4
address available to retry with.

This resolves https://github.com/letsencrypt/boulder/issues/2623
2017-05-08 13:00:16 -07:00
Daniel McCarney 361e7d4caa Clean up `berrors` (#2724)
This PR removes two berrors that aren't used anywhere in the codebase:

TooManyRequests , a holdover from AMQP, and is no longer used.
UnsupportedIdentifier, used just for rejecting IDNs, which we no longer do.
In addition, the SignatureValidation error was only used by the WFE so it is moved there and unexported.

Note for reviewers: To remove berrors.UnsupportedIdentifierError I replaced the errIDNNotSupported error in policy/pa.go with a berrors.MalformedError with the same name. This allows removing UnsupportedIdentifierError ahead of #2712 which removes the IDNASupport feature flag. This seemed OK to me, but I can restore UnsupportedIdentifierError and clean it up after 2712 if that's preferred.

Resolves #2709
2017-05-04 10:56:26 +01:00
Daniel McCarney 1ed34a4a5d Fixes cert count rate limit for exact PSL matches. (#2703)
Prior to this PR if a domain was an exact match to a public suffix
list entry the certificates per name rate limit was applied based on the
count of certificates issued for that exact name and all of its
subdomains.

This PR introduces an exception such that exact public suffix
matches correctly have the certificate per name rate limit applied based
on only exact name matches.

In order to accomplish this a new RPC is added to the SA
`CountCertificatesByExactNames`. This operates similar to the existing
`CountCertificatesByNames` but does *not* include subdomains in the
count, only exact matches to the names provided. The usage of this new
RPC is feature flag gated behind the "CountCertificatesExact" feature flag.

The RA unit tests are updated to test the new code paths both with and
without the feature flag enabled.

Resolves #2681
2017-05-02 13:43:35 -07:00
David Calavera cc5ee3906b Refactor IsSane and IsSane* to return useful errors. (#2685)
This change changes the returning values from boolean to error.

It makes `checkConsistency` an internal function and removes the
optional argument in favor of making checks explicit where they are
used.

It also renames those functions to CheckConsistency* to not
give the impression of still returning boolean values.

Signed-off-by: David Calavera <david.calavera@gmail.com>
2017-04-19 12:08:47 -04:00
Roland Bracewell Shoemaker fd561ef842 Block issuance on first OCSP response generation (#2633)
Generate first OCSP response in ca.IssueCertificate instead of ocsp-updater.newCertificateTick
if features.GenerateOCSPEarly is enabled. Adds a new field to the sa.AddCertiifcate RPC for
the OCSP response and only adds it to the certificate status + sets ocspLastUpdated if it is a
non-empty slice. ocsp-updater.newCertificateTick stays the same so we can catch certificates
that were successfully signed + stored but a OCSP response couldn't be generated (for whatever
reason).

Fixes #2477.
2017-04-04 11:28:09 -07:00
Roland Bracewell Shoemaker e2b2511898 Overhaul internal error usage (#2583)
This patch removes all usages of the `core.XXXError` and almost all usages of `probs` outside of the WFE and VA and replaces them with a unified internal error type. Since the VA uses `probs.ProblemDetails` quite extensively in challenges, and currently stores them in the DB I've saved this change for another change (it'll also require a migration). Since `ProblemDetails` should only ever be exposed to end-users all of its related logic should be moved into the `WFE` but since it still needs to be exposed to the VA and SA I've left it in place for now.

The new internal `errors` package offers the same convenience functions as `probs` does as well as a new simpler type testing method. A few small changes have also been made to error messages, mainly adding the library and function name to internal server errors for easier debugging (i.e. where a number of functions return the exact same errors and there is no other way to distinguish which method threw the error).

Also adds proper encoding of internal errors transferred over gRPC (the current encoding scheme is kept for `core` and `probs` errors since it'll be ideally be removed after we deploy this and follow-up changes) using `grpc/metadata` instead of the gRPC status codes.

Fixes #2507. Updates #2254 and #2505.
2017-03-22 23:27:31 -07:00
David Calavera c71c3cff80 Implement TLS-SNI-02 challenge validations. (#2585)
I think these are all the necessary changes to implement TLS-SNI-02 validations, according to the section 7.3 of draft 05:

https://tools.ietf.org/html/draft-ietf-acme-acme-05#section-7.3

I don't have much experience with this code, I'll really appreciate your feedback.

Signed-off-by: David Calavera <david.calavera@gmail.com>
2017-03-22 10:17:59 -07:00
David Calavera 0dc2513d2d
Generate GRPC objects with Go 1.8.
Signed-off-by: David Calavera <david.calavera@gmail.com>
2017-02-21 12:11:17 +01:00
Daniel McCarney a2b8faea1e Only resubmit missing SCTs. (#2342)
This PR introduces the ability for the ocsp-updater to only resubmit certificates to logs that we are missing SCTs from. Prior to this commit when a certificate was missing one or more SCTs we would submit it to every log, causing unnecessary overhead for us and the log operator.

To accomplish this a new RPC endpoint is added to the Publisher service "SubmitToSingleCT". Unlike the existing "SubmitToCT" this RPC endpoint accepts a log URI and public key in addition to the certificate DER bytes. The certificate is submitted directly to that log, and a cache of constructed resources is maintained so that subsequent submissions to the same log can reuse the stat name, verifier, and submission client.

Resolves #1679
2016-12-05 13:54:02 -08:00
Roland Bracewell Shoemaker 03fdd65bfe Add gRPC server to SA (#2374)
Adds a gRPC server to the SA and SA gRPC Clients to the WFE, RA, CA, Publisher, OCSP updater, orphan finder, admin revoker, and expiration mailer.

Also adds a CA gRPC client to the OCSP Updater which was missed in #2193.

Fixes #2347.
2016-12-02 17:24:46 -08:00
Daniel 3abc2ee896
Fixes unused digest 2016-11-30 13:43:41 -05:00
Daniel 5f7ec2217e
Spelling fixes 2016-11-30 13:17:56 -05:00
Roland Bracewell Shoemaker a87379bc6e Add gRPC server to RA (#2350)
Fixes #2348.
2016-11-29 15:34:35 -08:00
Daniel McCarney 95e231b91b Moves `core/reverse-name.go` to SA, adds test. (#2360)
The only consumer of `core/reverse-name.go` and the `core.ReverseName`
function was the SA. In keeping with minimizing the junk that collects
up in `core` this commit moves `core.ReverseName` into the SA as the
unexported `reverseName` function. There was no unit test for
`ReverseName` so this commit also adds a unit test to `sa_test.go`.
2016-11-28 12:22:26 -08:00
Daniel McCarney a6f2b0fafb Updates `go-jose` dep to v1.1.0 (#2314)
This commit updates the `go-jose` dependency to [v1.1.0](https://github.com/square/go-jose/releases/tag/v1.1.0) (Commit: aa2e30fdd1fe9dd3394119af66451ae790d50e0d). Since the import path changed from `github.com/square/...` to `gopkg.in/square/go-jose.v1/` this means removing the old dep and adding the new one.

The upstream go-jose library added a `[]*x509.Certificate` member to the `JsonWebKey` struct that prevents us from using a direct equality test against two `JsonWebKey` instances. Instead we now must compare the inner `Key` members.

The `TestRegistrationContactUpdate` function from `ra_test.go` was updated to populate the `Key` members used in testing instead of only using KeyID's to allow the updated comparisons to work as intended.

The `Key` field of the `Registration` object was switched from `jose.JsonWebKey` to `*jose.JsonWebKey ` to make it easier to represent a registration w/o a Key versus using a value with a nil `JsonWebKey.Key`.

I verified the upstream unit tests pass per contributing.md:
```
daniel@XXXXX:~/go/src/gopkg.in/square/go-jose.v1$ git show
commit aa2e30fdd1fe9dd3394119af66451ae790d50e0d
Merge: 139276c e18a743
Author: Cedric Staub <cs@squareup.com>
Date:   Thu Sep 22 17:08:11 2016 -0700

    Merge branch 'master' into v1
    
    * master:
      Better docs explaining embedded JWKs
      Reject invalid embedded public keys
      Improve multi-recipient/multi-sig handling

daniel@XXXXX:~/go/src/gopkg.in/square/go-jose.v1$ go test ./...
ok  	gopkg.in/square/go-jose.v1	17.599s
ok  	gopkg.in/square/go-jose.v1/cipher	0.007s
?   	gopkg.in/square/go-jose.v1/jose-util	[no test files]
ok  	gopkg.in/square/go-jose.v1/json	1.238s
```
2016-11-08 13:56:50 -05:00
Roland Bracewell Shoemaker 313bc75c05 Switch to Golang 1.7.3 in travis (#2305)
Protobuf files need to be regenerated because (I think) Golang 1.7.3 uses a somewhat different method of ordering fields in a struct when marshaling to bytes.
2016-11-07 10:49:23 -08:00
Roland Bracewell Shoemaker ce679bad41 Implement key rollover (#2231)
Fixes #503.

Functionality is gated by the feature flag `AllowKeyRollover`. Since this functionality is only specified in ACME draft-03 and we mostly implement the draft-02 style this takes some liberties in the implementation, which are described in the updated divergences doc. The `key-change` resource is used to side-step draft-03 `url` requirement.
2016-10-27 10:22:09 -04:00
Daniel McCarney 4c9cf065a8 `certificateStatus` table optimizations (Part One) (#2177)
This PR adds a migration to create two new fields on the `certificateStatus` table: `notAfter` and `isExpired`. The rationale for these fields is explained in #1864. Usage of these fields is gated behind `features.CertStatusOptimizationsMigrated` per [CONTRIBUTING.md](https://github.com/letsencrypt/boulder/blob/master/CONTRIBUTING.md#gating-migrations). This flag should be set to true **only** when the `20160817143417_CertStatusOptimizations.sql` migration has been applied.

Points of difference from #2132 (the initial preparatory "all-in-one go" PR):
**Note 1**: Updating the `isExpired` field in the OCSP updater can not be done yet, the `notAfter` field needs to be fully populated first - otherwise a separate query or a messy `JOIN` would have to be used to determine if a certStatus `isExpired` by using the `certificates` table's `expires` field. 
**Note 2**: Similarly we can't remove the `JOIN` on `certificates` from the `findStaleOCSPResponse` query yet until all DB rows have `notAfter` populated. This will happen in a separate **Part Two** PR.
2016-09-30 14:52:19 -04:00
Roland Bracewell Shoemaker c6e3ef660c Re-apply 2138 with proper gating (#2199)
Re-applies #2138 using the new style of feature-flag gated migrations. Account deactivation is gated behind `features.AllowAccountDeactivation`.
2016-09-29 17:16:03 -04:00
Roland Bracewell Shoemaker 2c966c61b2 Revert "Allow account deactivation (#2138)" (#2188)
This reverts commit 6f3d078414, reversing
changes made to c8f1fb3e2f.
2016-09-19 11:20:41 -07:00
Roland Shoemaker dbf9afa7d6 Review fixes pt. 1 2016-08-25 16:28:58 -07:00
Roland Shoemaker aa7f85d3f5 Merge branch 'master' into reg-deact 2016-08-24 11:51:15 -07:00
Roland Bracewell Shoemaker 51ee04e6a9 Allow authorization deactivation (#2116)
Implements `valid` and `pending` authz deactivation.
2016-08-23 16:25:06 -04:00