Since these two containers were using the same entrpoint.sh, they were
competing to run migrations and bind ports when run with `docker-compose
up`. Since we don't need the netaccess container when doing
`docker-compose up`, give it a separate entrypoint that exits
immediately by default, but does the normal migrations when run with
`docker-compose run`.
Things removed:
* features.EmbedSCTs (and all the associated RA/CA/ocsp-updater code etc)
* ca.enablePrecertificateFlow (and all the associated RA/CA code)
* sa.AddSCTReceipt and sa.GetSCTReceipt RPCs
* publisher.SubmitToCT and publisher.SubmitToSingleCT RPCs
Fixes#3755.
There are two main queries we do against pendingAuthorizations: counting pending authzs
for rate limiting, and looking up existing authorizations for reuse. Neither of these is served
perfectly by our current (regID, expires) index. The index works well in some conditions but
not others. This change adds two more specific indexes to replace the existing ones.
The below set of EXPLAINs demonstrates the new indexes in use. Note that in order to
get MariaDB to make proper use of the new index, I had to create a local account and
fill it with pending authzs using a modified chisel.py.
The `registrationID`, `status`, `expires` index is used for the case when an account creates
and then deactivates a lot of authzs, since deactivated authzs stay in the pending table
and create performance issues. Adding an index that includes status can fix those performance
issues. The last section of statements logs below, after I `UPDATE` all the pending authorizations
to be `deactivated`, demonstrates the speed difference in the count query (check the `rows` count).
```
mysql> SHOW CREATE TABLE pendingAuthorizations \G *************************** 1. row ***************************
Table: pendingAuthorizations
Create Table: CREATE TABLE `pendingAuthorizations` (
`id` varchar(255) NOT NULL,
`identifier` varchar(255) NOT NULL,
`registrationID` bigint(20) NOT NULL,
`status` varchar(255) NOT NULL,
`expires` datetime DEFAULT NULL,
`combinations` varchar(255) NOT NULL,
`LockCol` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
KEY `regId_expires_idx` (`registrationID`,`expires`),
CONSTRAINT `regId_pending_authz` FOREIGN KEY (`registrationID`) REFERENCES `registrations` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE status = 'pending' AND expires > NOW() \G *************************** 1. row *************************** id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 270
Extra: Using where
1 row in set (0.01 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> ADD INDEX `identifier_registrationID_status_expires_idx` (
-> `identifier`, `registrationID`, `status`, `expires`),
-> ADD INDEX `registrationID_status_expires_idx` (
-> `registrationID`, `status`, `expires`),
-> DROP INDEX `regId_expires_idx`;
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 150
Extra: Using where; Using index
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: identifier_registrationID_status_expires_idx,registrationID_status_expires_idx
key: identifier_registrationID_status_expires_idx
key_len: 1548
ref: NULL
rows: 1
Extra: Using index condition
1 row in set (0.00 sec)
mysql> UPDATE pendingAuthorizations SET status = 'deactivated' WHERE registrationID = 2003;
Query OK, 150 rows affected (0.02 sec)
Rows matched: 150 Changed: 150 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 1
Extra: Using where; Using index
1 row in set (0.01 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> DROP INDEX `identifier_registrationID_status_expires_idx`,
-> DROP INDEX `registrationID_status_expires_idx`,
-> ADD INDEX `regId_expires_idx` (`registrationID`,`expires`);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
```
Fix: When any of OCSPURL, CRLURL or IssuerURL in the CertProfile are empty, the relevant fields are encoded in ASN.1 as empty arrays.
Fix: KeyUsage is a bitmask requiring bitwise OR.
Previously, if a remote VA returned an error that is not a ProblemDetail, the
primary VA would log a ServerInternalProblem but not the underlying error.
This commit updates performRemoteValidation to always return the full error it
receives from a remote VA.
This commit also adds a unittest that checks that the VA still returns a
ServerInternalProblem to the RA, and that the VA audit logs the underlying
error.
Resolves https://github.com/letsencrypt/boulder/issues/3753
We've migrated the production/staging Boulder instances to builds using
Go 1.10.3 and can now remove the Go 1.10.2 builds from the travis matrix
and the `tag_and_upload.sh` Boulder tools script.
In each of GetAuthorization, DeactivateAuthorization, and SetOrderError, there
was an error case that could cause us to not rollback the transaction,
leaving it open. This could cause database performance problems.
Tested against relevant hardware for generating both RSA and ECDSA roots and intermediates with keys generated using `gen-key`.
Also this makes a few changes to the `gen-key` tool after further experience with the HSM and more reading of the PCKS#11 specification. Main change is the removal of `compatMode`, which was intended to provide support for two naming schemes for EC used in subsequent PKCS#11 drafts. It turns out these schemes were changes in name only and the underlying structs/ints were the exact same (i.e. `CKA_ECDSA_PARAMS == CKA_EC_PARAMS` and `CKM_ECDSA_KEY_PAIR_GEN == CKM_EC_KEY_PAIR_GEN`) and just allowed using one of the two names based on preference. This meant with `compatMode` enabled or disabled the tool did the exact same thing.
Fixes#3697.
This adds support for the account-uri CAA parameter as specified by
section 3 of https://tools.ietf.org/html/draft-ietf-acme-caa-04, allowing
issuance to be restricted to one or more ACME accounts as specified by CAA
records.
We see a fair number of ACME accounts/registrations with contact
addresses for the RFC2606 Section 3 "Reserved Example Second Level
Domain Names" (`example.com`, `example.net`, `example.org`). These are
not real contact addresses and are likely the result of the user
copy-pasting example configuration. These users will miss out on
expiration emails and other subscriber communications :-(
This commit updates the RA's `validateEmail` function to reject any
contact addresses for reserved example domain names. The corresponding
unit test is updated accordingly.
Resolves https://github.com/letsencrypt/boulder/issues/3719
Switch linting library to zmap/zlint.
```
github.com/zmap/zlint$ go test ./...
ok github.com/zmap/zlint 0.190s
? github.com/zmap/zlint/cmd/zlint [no test files]
ok github.com/zmap/zlint/lints 0.216s
ok github.com/zmap/zlint/util (cached)
```
This commit updates the VA's checkCAA function to include the provided
challengeType parameter as part of the audit log line for the CAA
check result. If challengeType == nil then the value "none" is logged.
Unit tests are updated accordingly.
This change will make it easier to distinguish cases where "Valid for
issuance" is false because of a validation-methods restriction.
Resolves#3740
In preparation of `vgo` it seems many upstream projects are adding new
tags. This is causing variations in the `godep-restore` phase of CI when
the CI Godeps picks up a new `Comment` field that isn't present in the
committed JSON. This has broken master ~3 times now and so it seems
prudent to ignore the Comment field entirely for the short term. We
continue to wait for the glorious day in which we can switch to `vgo`
and never muck with `Godeps.json` again.
This commit updates the `wfe2/verify.go` implementation of
`acctIDFromURL` to include the invalid `kid` value being rejected as
part of the rejection error message. This will hopefully help
users/client developers understand the problem faster. Unit tests are
updated accordingly.
We landed this feature flag disabled pending Certbot's acme library
supporting this status value. That work has landed and so we can enable
this feature in `config-next` ahead of a staging/prod rollout.
We're still on the same commit, which Godep previously commented as
"v1.0.0-2-gc12348c" (in other words, commit `c12348c`, which is slightly
ahead of the v1.0.0 tag). The upstream repo recently tagged a v1.0.1
release, at the same commit we were using. This caused Godep in our
tests to use a simplified comment referencing only the tag, which caused
a spurious diff and failed the test.
This commit updates the comment in Godeps.json.
The "v1.0.0-2-gc12348c" tag referenced in the Godeps.json comment for
the "github.com/matttproud/golang_protobuf_extensions/pbutil" import
doesn't seem to exist in the upstream repo anymore.
The "v1.0.1" comment being flagged as a diff in Godeps restore during CI
_does_ exist and it points to the same commit
(c12348ce28de40eed0136aa2b644d0ee0650e56c) we are using.
This commit fixes the comment to match upstream & expected.
(ugh Godeps....)
This is one of our slowest unittests, clocking in at 23 seconds in a recent run.
This was largely due to generating keys. Note that performance is significantly
worse under the race detector.
When performing CAA checking respect the validation-methods parameter (if
present) and restrict the allowed authorization methods to those specified.
This allows a domain to restrict authorization methods that can be used with
Let's Encrypt.
This is largely based on PR #3003 (by @lukaslihotzki), which was landed and
then later reverted due to issue #3143. The bug the resulted in the previous
code being reverted has been addressed (likely inadvertently) by 76973d0f.
This implementation also includes integration tests for CAA validation-methods.
Fixes issue #3143.
We'd like to start using the DNS load balancer in the latest version of gRPC. That means putting all IPs for a service under a single hostname (or using a SRV record, but we're not taking that path). This change adds an sd-test-srv to act as our service discovery DNS service. It returns both Boulder IP addresses for any A lookup ending in ".boulder". This change also sets up the Docker DNS for our boulder container to defer to sd-test-srv when it doesn't know an answer.
sd-test-srv doesn't know how to resolve public Internet names like `github.com`. Resolving public names is required for the `godep-restore` test phase, so this change breaks out a copy of the boulder container that is used only for `godep-restore`.
This change implements a shim of a DNS resolver for gRPC, so that we can switch to DNS-based load balancing with the currently vendored gRPC, then when we upgrade to the latest gRPC we won't need a simultaneous config update.
Also, this change introduces a check at the end of the integration test that each backend received at least one RPC, ensuring that we are not sending all load to a single backend.
This is a quick first pass at audit logging the *dns.CAA records in JSON format from within the VA's IsCAAValid function. This will provide more information for post-hoc analysis of CAA decisions.
Per RFC 6844 Section 5.1 the matching of CAA tag identifiers (e.g.
"Issue") is case insensitive. This commit updates the CAA tag processing
to be case insensitive as required by the RFC.
To exercise the fix this commit adds a test case to the `caaMockDNS`
`LookupCAA` implementation for a hostname (`mixedcase.com`) that has
a CAA record with a mixed case `Issue` tag. Prior to the fix from this
branch being included in `va/caa.go` the test fails:
```
--- FAIL: TestCAAChecking/Bad_(Reserved,_Mixed_case_Issue) (0.00s)
caa_test.go:292: checkCAARecords validity mismatch for
mixedcase.com: got true expected false
```
With the fix applied, the test passes.
This commit updates the `github.com/weppos/publicsuffix-go` dependency to
67ec7c1, the tip of master at the time of writing.
Unit tests are verified to pass:
```
$ go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix (cached)
ok github.com/weppos/publicsuffix-go/publicsuffix (cached)
```
Prior to this PR the builds of master are failing in Travis with an error during the `godep-restore` phase of our CI:
```
[ godep-restore] Starting
godep restore
rm -rf Godeps/ vendor/
godep save ./...
diff /dev/fd/63 /dev/fd/62
254c254
< "Comment": "v1.3-28-g3955978",
---
> "Comment": "v1.3.0-28-g3955978",
> git diff --exit-code -- ./vendor/
>
```
The root cause here is the upstream `github.com/go-sql-driver/mysql` project adding a
new tag for the 3955978caca48c1658a4bb7a9c6a0f084e326af3 commit. The new tag
is named to match semvar requirements and Godeps in CI is preferring that tag.
Updating the file to point to this new tag is a sensible fix from our side.
This commit updates the "Comment" filed to match what CI expects.
Prior to this commit the builds of master are failing in Travis with an
error during the `godep-restore` phase of our CI:
```
[ godep-restore] Starting
godep restore
rm -rf Godeps/ vendor/
godep save ./...
diff /dev/fd/63 /dev/fd/62
254c254
< "Comment": "v1.3-28-g3955978",
---
> "Comment": "v1.3.0-28-g3955978",
> git diff --exit-code -- ./vendor/
>
```
This seems to be a mysterious difference in the "Comment" field of the
`github.com/go-sql-driver/mysql` dependency. This dep hasn't changed
versions so given the general level of frustration involved with
debugging Godep it seems like the easiest path forward is to mimick the
diff.
This commit updates the "Comment" filed to match what CI expects.
Two fixes that I found while doing work on the gen-cert tool and setting up the HSM again
* Accept an empty PIN argument, this allows purely using the PED for login if not using challenge mode
* Generate 4 byte key ID for public/private key pairs during key gen, the HSM doesn't generate this field itself and `letsencrypt/pkcs11key` relies on this attribute to function
When rechecking CAA, the existing code maps all failures to a CAAError.
This means that any other non-CAA failure (for example, an internal server
error) gets hidden.
Avoid this by reworking recheckCAA to return errors and if we find a
non-CAAError, we return that directly. Revise tests to cover both
situations.
Updates issue #3143.
The existing CAA tests only test the CAA checks on the validation path and
not the CAA rechecking in the case where an existing authorization is present
(but older than the 8 hour window).
This extends the CAA integration tests to also cover the CAA rechecking
code path, by reusing older authorizations and rejecting issuance via CAA.
Production/staging have been updated to a release built with Go 1.10.2.
This allows us to remove the Go 1.10.1 builds from the travis matrix and
default to Go 1.10.2.
While we intended to allow legacy ACME v1 accounts created through the WFE to work with the ACME v2 implementation and the WFE2 we neglected to consider that a legacy account would have a Key ID URL that doesn't match the expected for a V2 account. This caused `wfe2/verify.go`'s `lookupJWK` to reject all POST requests authenticated by a legacy account unless the ACME client took the extra manual step of "fixing" the URL.
This PR adds a configuration parameter to the WFE2 for an allowed legacy key ID prefix. The WFE2 verification logic is updated to allow both the expected key ID prefix and the configured legacy key ID prefix. This will allow us to specify the correct legacy URL in configuration for both staging/prod to allow unmodified V1 ACME accounts to be used with ACME v2.
Resolves https://github.com/letsencrypt/boulder/issues/3674
So I took a quick stab at one possible solution for the authz expiration variance discussed over at community.letsencrypt.org/t/inconsistent-datetime-format-in-some-responses/61452
Golang's nanosecond precision results in newly created pending authz having one expires datetime, but subsequent requests have a different expires datetime due to database storage throwing away fractional second information.
"expires": "2018-04-14T04:43:00.105818284Z",
...
"expires": "2018-04-14T04:43:00Z",
I am not a Go expert so there might be some more widely accepted approach to accomplishing the same thing, please let me know if you would prefer a different solution.
Jacob Hoffman-Andrews