The previous update was just [9 days ago](https://github.com/letsencrypt/boulder/pull/3808). However, since we merged some changes into the PSL that are related to IANA TLDs I though about providing an immediate patch.
Tests are passing:
```
➜ ~ cd ~/go/src/github.com/weppos/publicsuffix-go
➜ publicsuffix-go git:(master) GOCACHE=off go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.021s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.034s
```
ACME draft-13 changed the inner JWS' body to contain the old key
that signed the outer JWS. Because this is a backwards incompatible
change with the draft-12 ACME v2 key rollover we introduce a new feature
flag `features.ACME13KeyRollover` and conditionally use the old or new
key rollover semantics based on its value.
None of the development team use this approach to running a dev env. It
no longer works without modifying `test/startservers.py` and the `test/`
configurations. Given that it has been broken for a month+ and has only
provoked one user issue I think we can be fairly confident that few others
are using this method of setting up a Boulder development environment
and should prioritize our time/docs accordingly.
This commit updates the vendored `publicsuffix-go` dependency to [b8c0530](b8c0530c1a), the tip of master at the time of writing.
Unit tests confirmed to pass:
```
~/go/src/github.com/weppos/publicsuffix-go/publicsuffix$ go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.007s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.024s
```
Resolves#3807
ctpolicy permutes logs before submitting to them, to give each log a
chance. The stagger feature was meant to sleep for an amount of time
proportional to a log's position in the permuted list. However, it was
actually using the log's position in the un-permuted list, so logs that
appear later in the config would always be submitted to later than logs
earlier in the config.
This fixes that, and does some minor variable renaming for clarity.
Since these two containers were using the same entrpoint.sh, they were
competing to run migrations and bind ports when run with `docker-compose
up`. Since we don't need the netaccess container when doing
`docker-compose up`, give it a separate entrypoint that exits
immediately by default, but does the normal migrations when run with
`docker-compose run`.
Things removed:
* features.EmbedSCTs (and all the associated RA/CA/ocsp-updater code etc)
* ca.enablePrecertificateFlow (and all the associated RA/CA code)
* sa.AddSCTReceipt and sa.GetSCTReceipt RPCs
* publisher.SubmitToCT and publisher.SubmitToSingleCT RPCs
Fixes#3755.
There are two main queries we do against pendingAuthorizations: counting pending authzs
for rate limiting, and looking up existing authorizations for reuse. Neither of these is served
perfectly by our current (regID, expires) index. The index works well in some conditions but
not others. This change adds two more specific indexes to replace the existing ones.
The below set of EXPLAINs demonstrates the new indexes in use. Note that in order to
get MariaDB to make proper use of the new index, I had to create a local account and
fill it with pending authzs using a modified chisel.py.
The `registrationID`, `status`, `expires` index is used for the case when an account creates
and then deactivates a lot of authzs, since deactivated authzs stay in the pending table
and create performance issues. Adding an index that includes status can fix those performance
issues. The last section of statements logs below, after I `UPDATE` all the pending authorizations
to be `deactivated`, demonstrates the speed difference in the count query (check the `rows` count).
```
mysql> SHOW CREATE TABLE pendingAuthorizations \G *************************** 1. row ***************************
Table: pendingAuthorizations
Create Table: CREATE TABLE `pendingAuthorizations` (
`id` varchar(255) NOT NULL,
`identifier` varchar(255) NOT NULL,
`registrationID` bigint(20) NOT NULL,
`status` varchar(255) NOT NULL,
`expires` datetime DEFAULT NULL,
`combinations` varchar(255) NOT NULL,
`LockCol` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
KEY `regId_expires_idx` (`registrationID`,`expires`),
CONSTRAINT `regId_pending_authz` FOREIGN KEY (`registrationID`) REFERENCES `registrations` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE status = 'pending' AND expires > NOW() \G *************************** 1. row *************************** id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 270
Extra: Using where
1 row in set (0.01 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> ADD INDEX `identifier_registrationID_status_expires_idx` (
-> `identifier`, `registrationID`, `status`, `expires`),
-> ADD INDEX `registrationID_status_expires_idx` (
-> `registrationID`, `status`, `expires`),
-> DROP INDEX `regId_expires_idx`;
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 150
Extra: Using where; Using index
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: identifier_registrationID_status_expires_idx,registrationID_status_expires_idx
key: identifier_registrationID_status_expires_idx
key_len: 1548
ref: NULL
rows: 1
Extra: Using index condition
1 row in set (0.00 sec)
mysql> UPDATE pendingAuthorizations SET status = 'deactivated' WHERE registrationID = 2003;
Query OK, 150 rows affected (0.02 sec)
Rows matched: 150 Changed: 150 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 1
Extra: Using where; Using index
1 row in set (0.01 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> DROP INDEX `identifier_registrationID_status_expires_idx`,
-> DROP INDEX `registrationID_status_expires_idx`,
-> ADD INDEX `regId_expires_idx` (`registrationID`,`expires`);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
```
Fix: When any of OCSPURL, CRLURL or IssuerURL in the CertProfile are empty, the relevant fields are encoded in ASN.1 as empty arrays.
Fix: KeyUsage is a bitmask requiring bitwise OR.
Previously, if a remote VA returned an error that is not a ProblemDetail, the
primary VA would log a ServerInternalProblem but not the underlying error.
This commit updates performRemoteValidation to always return the full error it
receives from a remote VA.
This commit also adds a unittest that checks that the VA still returns a
ServerInternalProblem to the RA, and that the VA audit logs the underlying
error.
Resolves https://github.com/letsencrypt/boulder/issues/3753
We've migrated the production/staging Boulder instances to builds using
Go 1.10.3 and can now remove the Go 1.10.2 builds from the travis matrix
and the `tag_and_upload.sh` Boulder tools script.
In each of GetAuthorization, DeactivateAuthorization, and SetOrderError, there
was an error case that could cause us to not rollback the transaction,
leaving it open. This could cause database performance problems.
Tested against relevant hardware for generating both RSA and ECDSA roots and intermediates with keys generated using `gen-key`.
Also this makes a few changes to the `gen-key` tool after further experience with the HSM and more reading of the PCKS#11 specification. Main change is the removal of `compatMode`, which was intended to provide support for two naming schemes for EC used in subsequent PKCS#11 drafts. It turns out these schemes were changes in name only and the underlying structs/ints were the exact same (i.e. `CKA_ECDSA_PARAMS == CKA_EC_PARAMS` and `CKM_ECDSA_KEY_PAIR_GEN == CKM_EC_KEY_PAIR_GEN`) and just allowed using one of the two names based on preference. This meant with `compatMode` enabled or disabled the tool did the exact same thing.
Fixes#3697.
This adds support for the account-uri CAA parameter as specified by
section 3 of https://tools.ietf.org/html/draft-ietf-acme-caa-04, allowing
issuance to be restricted to one or more ACME accounts as specified by CAA
records.
We see a fair number of ACME accounts/registrations with contact
addresses for the RFC2606 Section 3 "Reserved Example Second Level
Domain Names" (`example.com`, `example.net`, `example.org`). These are
not real contact addresses and are likely the result of the user
copy-pasting example configuration. These users will miss out on
expiration emails and other subscriber communications :-(
This commit updates the RA's `validateEmail` function to reject any
contact addresses for reserved example domain names. The corresponding
unit test is updated accordingly.
Resolves https://github.com/letsencrypt/boulder/issues/3719
Switch linting library to zmap/zlint.
```
github.com/zmap/zlint$ go test ./...
ok github.com/zmap/zlint 0.190s
? github.com/zmap/zlint/cmd/zlint [no test files]
ok github.com/zmap/zlint/lints 0.216s
ok github.com/zmap/zlint/util (cached)
```
This commit updates the VA's checkCAA function to include the provided
challengeType parameter as part of the audit log line for the CAA
check result. If challengeType == nil then the value "none" is logged.
Unit tests are updated accordingly.
This change will make it easier to distinguish cases where "Valid for
issuance" is false because of a validation-methods restriction.
Resolves#3740
In preparation of `vgo` it seems many upstream projects are adding new
tags. This is causing variations in the `godep-restore` phase of CI when
the CI Godeps picks up a new `Comment` field that isn't present in the
committed JSON. This has broken master ~3 times now and so it seems
prudent to ignore the Comment field entirely for the short term. We
continue to wait for the glorious day in which we can switch to `vgo`
and never muck with `Godeps.json` again.
This commit updates the `wfe2/verify.go` implementation of
`acctIDFromURL` to include the invalid `kid` value being rejected as
part of the rejection error message. This will hopefully help
users/client developers understand the problem faster. Unit tests are
updated accordingly.
We landed this feature flag disabled pending Certbot's acme library
supporting this status value. That work has landed and so we can enable
this feature in `config-next` ahead of a staging/prod rollout.
We're still on the same commit, which Godep previously commented as
"v1.0.0-2-gc12348c" (in other words, commit `c12348c`, which is slightly
ahead of the v1.0.0 tag). The upstream repo recently tagged a v1.0.1
release, at the same commit we were using. This caused Godep in our
tests to use a simplified comment referencing only the tag, which caused
a spurious diff and failed the test.
This commit updates the comment in Godeps.json.
The "v1.0.0-2-gc12348c" tag referenced in the Godeps.json comment for
the "github.com/matttproud/golang_protobuf_extensions/pbutil" import
doesn't seem to exist in the upstream repo anymore.
The "v1.0.1" comment being flagged as a diff in Godeps restore during CI
_does_ exist and it points to the same commit
(c12348ce28de40eed0136aa2b644d0ee0650e56c) we are using.
This commit fixes the comment to match upstream & expected.
(ugh Godeps....)
This is one of our slowest unittests, clocking in at 23 seconds in a recent run.
This was largely due to generating keys. Note that performance is significantly
worse under the race detector.
When performing CAA checking respect the validation-methods parameter (if
present) and restrict the allowed authorization methods to those specified.
This allows a domain to restrict authorization methods that can be used with
Let's Encrypt.
This is largely based on PR #3003 (by @lukaslihotzki), which was landed and
then later reverted due to issue #3143. The bug the resulted in the previous
code being reverted has been addressed (likely inadvertently) by 76973d0f.
This implementation also includes integration tests for CAA validation-methods.
Fixes issue #3143.
We'd like to start using the DNS load balancer in the latest version of gRPC. That means putting all IPs for a service under a single hostname (or using a SRV record, but we're not taking that path). This change adds an sd-test-srv to act as our service discovery DNS service. It returns both Boulder IP addresses for any A lookup ending in ".boulder". This change also sets up the Docker DNS for our boulder container to defer to sd-test-srv when it doesn't know an answer.
sd-test-srv doesn't know how to resolve public Internet names like `github.com`. Resolving public names is required for the `godep-restore` test phase, so this change breaks out a copy of the boulder container that is used only for `godep-restore`.
This change implements a shim of a DNS resolver for gRPC, so that we can switch to DNS-based load balancing with the currently vendored gRPC, then when we upgrade to the latest gRPC we won't need a simultaneous config update.
Also, this change introduces a check at the end of the integration test that each backend received at least one RPC, ensuring that we are not sending all load to a single backend.
Roland Bracewell Shoemaker