There are two main queries we do against pendingAuthorizations: counting pending authzs
for rate limiting, and looking up existing authorizations for reuse. Neither of these is served
perfectly by our current (regID, expires) index. The index works well in some conditions but
not others. This change adds two more specific indexes to replace the existing ones.
The below set of EXPLAINs demonstrates the new indexes in use. Note that in order to
get MariaDB to make proper use of the new index, I had to create a local account and
fill it with pending authzs using a modified chisel.py.
The `registrationID`, `status`, `expires` index is used for the case when an account creates
and then deactivates a lot of authzs, since deactivated authzs stay in the pending table
and create performance issues. Adding an index that includes status can fix those performance
issues. The last section of statements logs below, after I `UPDATE` all the pending authorizations
to be `deactivated`, demonstrates the speed difference in the count query (check the `rows` count).
```
mysql> SHOW CREATE TABLE pendingAuthorizations \G *************************** 1. row ***************************
Table: pendingAuthorizations
Create Table: CREATE TABLE `pendingAuthorizations` (
`id` varchar(255) NOT NULL,
`identifier` varchar(255) NOT NULL,
`registrationID` bigint(20) NOT NULL,
`status` varchar(255) NOT NULL,
`expires` datetime DEFAULT NULL,
`combinations` varchar(255) NOT NULL,
`LockCol` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
KEY `regId_expires_idx` (`registrationID`,`expires`),
CONSTRAINT `regId_pending_authz` FOREIGN KEY (`registrationID`) REFERENCES `registrations` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE status = 'pending' AND expires > NOW() \G *************************** 1. row *************************** id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 270
Extra: Using where
1 row in set (0.01 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> ADD INDEX `identifier_registrationID_status_expires_idx` (
-> `identifier`, `registrationID`, `status`, `expires`),
-> ADD INDEX `registrationID_status_expires_idx` (
-> `registrationID`, `status`, `expires`),
-> DROP INDEX `regId_expires_idx`;
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 150
Extra: Using where; Using index
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: identifier_registrationID_status_expires_idx,registrationID_status_expires_idx
key: identifier_registrationID_status_expires_idx
key_len: 1548
ref: NULL
rows: 1
Extra: Using index condition
1 row in set (0.00 sec)
mysql> UPDATE pendingAuthorizations SET status = 'deactivated' WHERE registrationID = 2003;
Query OK, 150 rows affected (0.02 sec)
Rows matched: 150 Changed: 150 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 1
Extra: Using where; Using index
1 row in set (0.01 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> DROP INDEX `identifier_registrationID_status_expires_idx`,
-> DROP INDEX `registrationID_status_expires_idx`,
-> ADD INDEX `regId_expires_idx` (`registrationID`,`expires`);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
```
Both the `20160818140745_AddRegStatus.sql` and
`20160914105917_RemoveChallengesAcctKeyAndTLS.sql` migrations have been
applied in production and can be moved out of `sa/_db-next/` to reflect
this fact.
This PR adds a migration to create two new fields on the `certificateStatus` table: `notAfter` and `isExpired`. The rationale for these fields is explained in #1864. Usage of these fields is gated behind `features.CertStatusOptimizationsMigrated` per [CONTRIBUTING.md](https://github.com/letsencrypt/boulder/blob/master/CONTRIBUTING.md#gating-migrations). This flag should be set to true **only** when the `20160817143417_CertStatusOptimizations.sql` migration has been applied.
Points of difference from #2132 (the initial preparatory "all-in-one go" PR):
**Note 1**: Updating the `isExpired` field in the OCSP updater can not be done yet, the `notAfter` field needs to be fully populated first - otherwise a separate query or a messy `JOIN` would have to be used to determine if a certStatus `isExpired` by using the `certificates` table's `expires` field.
**Note 2**: Similarly we can't remove the `JOIN` on `certificates` from the `findStaleOCSPResponse` query yet until all DB rows have `notAfter` populated. This will happen in a separate **Part Two** PR.
As detailed in issue #1872 the getSerialsIssuedSince function of the ocsp-updater cmd performs with poor runtime, likely due to a filesort and lack of index on the issued field.
This commit adds a migration to create a new index on the issued field.
* Add index to certificates table issued field.
* Rename index to use _idx suffix
Use bridged networking.
Add some files to .dockerignore to shrink the build state sent to Docker
daemon.
Use specific hostnames to contact services, rather than localhost.
Add instructions for adding those hostnames to /etc/hosts in non-Docker config.
Use DSN-style connect strings for DBs.
Remove localhost / 127.0.0.1 rewrite hack from create_db.sh.
Add hosts section with new hostnames.
Remove bin from .dockerignore.
SQL grants go to %
Short-circuit DB creation if already existing.
Make `go install` a part of Docker image build so that Docker run is much
faster.
Bind to 0.0.0.0 for OCSP responders so they can be reached from host, and
publish / expose their ports.
Remove ToSServerThread and test.js' fetch of ToS.
Increase the registrationsPerIP rate limit threshold. When issuing from a Docker
host, the 127.0.0.1 override doesn't apply, so the limit is quickly hit.
Update docker-compose for bridged networking. Note: docker-compose doesn't currently work, but should be close.
https://github.com/letsencrypt/boulder/pull/1639
Adds a new rate limit, certficatesPerFQDNSet, which counts certificates
with the same set of FQDNS using a table containing the hash of the dNSNames
mapped to a certificate serial. A new method is added to the SA in AddCertificate
to add this hash to the fqdnSets table, which is gated by a config bool.
Previously, when requesting a new certificate, we might churn through
many authz attempts to find the latest authz that was valid for a given
domain. There was an index on registration id, but a given registration
may have very many authzs. Think CDNs with one registration.
Updates #959
Fixes https://github.com/letsencrypt/boulder/issues/898
Also removes currently-unused 'development' DB, and do initial migrations in
parallel, which shortens create_db.sh from 20 seconds to 10 seconds.
Changes ResetTestDatabase into two functions, one each for SA and Policy DBs,
which take care of setting up the DB connection using a special higher-privileged
user called test_setup.
OCSP-Responder attempts to read the OCSP response from the certificateStatus table,
if it cannot find a response there it reads the ocspResponses table to try to find a
response, if neither contains a response the not found bool is passed back to the
Responder.
* Moves revocation from the CA to the OCSP-Updater, the RA will mark certificates as
revoked then wait for the OCSP-Updater to create a new (final) revoked response
* Merges the ocspResponses table with the certificateStatus table and only use UPDATES
to update the OCSP response (vs INSERT-only since this happens quite often and will
lead to an extremely large table)
also moves the first OCSP responses generation from the CA to the OCSP updater. This patch lays the
ground work for moving CT submission and adding CT backfill to the OCSP updater.
Previously, if we did an up migration, generated some certs, then did a
down migration, we would truncate some of the new-style serial numbers in a way
that would be hard to reverse.
Previously, 32 bytes were used for serial numbers but now we make
certificates with 36 bytes (see #823 and #813). Going forward, we want
them to be consistent, so we update our current ones by prepending 4
zeros to the them.
For the two existing certs, their leading datacenter id will be 0 in
production, but we take this hit because we can't adjust the serial
inside the certificates themselves, and can only pad them with zeroes in
the database.
Fixes#834
Adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table.
The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts.
This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs.
* A fake CT server for integration testing
* A external tool to search the database for certificates lacking a full set of SCT receipts
* A method to construct X.509 v3 extensions containing receipts for the OCSP responder
* Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients)
Resolves#95.
Challenge URIs should be determined by the WFE at fetch time, rather than stored
alongside the challenge in the DB. This simplifies a lot of the logic, and
allows to to remove a code path in NewAuthorization where we create an
authorization, then immediately save it with modifications to the challenges.
This change also gives challenges their own endpoint, which contains the
challenge id rather than the challenge's offset within its parent authorization.
This is also a first step towards replacing UpdateAuthorization with
UpdateChallenge: https://github.com/letsencrypt/boulder/issues/760.
Jacob Hoffman-Andrews