First commit adding support for tooling to aid in the tracking and remediation
of incidents.
- Add new SA method `IncidentsForSerial`
- Add database models for `incident`s and `incidentCert`s
- Add protobuf type for `incident`
- Add database migrations for `incidents`, `incident_foo`, and `incident_bar`
- Give db user `sa` permissions to `incidents`, `incident_foo`, and
`incident_bar`
Part Of #5947
In `sa.checkFQDNSetExists`, query both the normal `fqdnSets` and the
`fqdnSets_old` tables. The `fqdnSets` table was recently truncated to
only have 7 days worth of data, but this helper function is used to
bypass other rate limits if there exists a prior certificate for the
exact same set of names, and that functionality cares about at least
90 days worth of data. Therefore we need to query both tables, at least
until `fqdnSets` contains 90 days worth of data again.
Also make a variety of other changes to support this change: creating
the `fqdnSets_old` table in our test environment, documenting various
places where it needs to be cleaned up, and removing some unused code.
Fixes#5671
Remove two database migration files which were added immediately
prior to 5305, and which should have been removed by it, but were
left in place due to weird git merge semantics.
Remove five years of built-up database goose migration files.
Replace them with a single file which contains a snapshot of
the current database schema, which has been confirmed to
be identical to the current schema live in production.
Part of #5254
This commit consists of three classes of changes:
1) Changing various command main.go files to always behave as they
would have when features.BlockedKeyTable was true. Also changing
one test in the same manner.
2) Removing the BlockedKeyTable flag from configuration in config-next,
because the flag is already live.
3) Moving the BlockedKeyTable flag to the "deprecated" section of
features.go, and regenerating featureflag_strings.go.
A future change will remove the BlockedKeyTable flag (and other
similarly deprecated flags) from features.go entirely.
Fixes#4873
In the process, rename generateOCSPAndStoreCertificate to just
storeCertificate, because that function doesn't generate OCSP anymore;
instead the OCSP is generated (and stored) at precertificate issuance
time.
To make this work, I changed the twenty_days_ago setup to use
`config-next` when the main test phase is running `config`. That, in
turn, made the recheck_caa test fail, so I added a tweak to that.
I also moved the authzv2 migrations into `db`. Without that change,
the integration test would fail during the twenty_days_ago setup because
Boulder would attempt to create authzv2 objects but the table wouldn't
exist yet.
There are two main queries we do against pendingAuthorizations: counting pending authzs
for rate limiting, and looking up existing authorizations for reuse. Neither of these is served
perfectly by our current (regID, expires) index. The index works well in some conditions but
not others. This change adds two more specific indexes to replace the existing ones.
The below set of EXPLAINs demonstrates the new indexes in use. Note that in order to
get MariaDB to make proper use of the new index, I had to create a local account and
fill it with pending authzs using a modified chisel.py.
The `registrationID`, `status`, `expires` index is used for the case when an account creates
and then deactivates a lot of authzs, since deactivated authzs stay in the pending table
and create performance issues. Adding an index that includes status can fix those performance
issues. The last section of statements logs below, after I `UPDATE` all the pending authorizations
to be `deactivated`, demonstrates the speed difference in the count query (check the `rows` count).
```
mysql> SHOW CREATE TABLE pendingAuthorizations \G *************************** 1. row ***************************
Table: pendingAuthorizations
Create Table: CREATE TABLE `pendingAuthorizations` (
`id` varchar(255) NOT NULL,
`identifier` varchar(255) NOT NULL,
`registrationID` bigint(20) NOT NULL,
`status` varchar(255) NOT NULL,
`expires` datetime DEFAULT NULL,
`combinations` varchar(255) NOT NULL,
`LockCol` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
KEY `regId_expires_idx` (`registrationID`,`expires`),
CONSTRAINT `regId_pending_authz` FOREIGN KEY (`registrationID`) REFERENCES `registrations` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE status = 'pending' AND expires > NOW() \G *************************** 1. row *************************** id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 270
Extra: Using where
1 row in set (0.01 sec)
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> ADD INDEX `identifier_registrationID_status_expires_idx` (
-> `identifier`, `registrationID`, `status`, `expires`),
-> ADD INDEX `registrationID_status_expires_idx` (
-> `registrationID`, `status`, `expires`),
-> DROP INDEX `regId_expires_idx`;
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 150
Extra: Using where; Using index
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT id, identifier, registrationID, status, expires, combinations, LockCol FROM pendingAuthorizations WHERE registrationID = 2003 AND identifier = '{\"type\":\"dns\",\"value\":\"23.com\"}' AND status = 'pending' AND expires > NOW() ORDER BY expires ASC LIMIT 1 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: identifier_registrationID_status_expires_idx,registrationID_status_expires_idx
key: identifier_registrationID_status_expires_idx
key_len: 1548
ref: NULL
rows: 1
Extra: Using index condition
1 row in set (0.00 sec)
mysql> UPDATE pendingAuthorizations SET status = 'deactivated' WHERE registrationID = 2003;
Query OK, 150 rows affected (0.02 sec)
Rows matched: 150 Changed: 150 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G *************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: range
possible_keys: registrationID_status_expires_idx
key: registrationID_status_expires_idx
key_len: 781
ref: NULL
rows: 1
Extra: Using where; Using index
1 row in set (0.01 sec)
mysql>
mysql> ALTER TABLE `pendingAuthorizations`
-> DROP INDEX `identifier_registrationID_status_expires_idx`,
-> DROP INDEX `registrationID_status_expires_idx`,
-> ADD INDEX `regId_expires_idx` (`registrationID`,`expires`);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT COUNT(*) FROM pendingAuthorizations WHERE registrationID = 2003 AND status = 'pending' AND expires > NOW() \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pendingAuthorizations
type: ref
possible_keys: regId_expires_idx
key: regId_expires_idx
key_len: 8
ref: const
rows: 150
Extra: Using index condition; Using where
1 row in set (0.00 sec)
```
Both the `20160818140745_AddRegStatus.sql` and
`20160914105917_RemoveChallengesAcctKeyAndTLS.sql` migrations have been
applied in production and can be moved out of `sa/_db-next/` to reflect
this fact.
This PR adds a migration to create two new fields on the `certificateStatus` table: `notAfter` and `isExpired`. The rationale for these fields is explained in #1864. Usage of these fields is gated behind `features.CertStatusOptimizationsMigrated` per [CONTRIBUTING.md](https://github.com/letsencrypt/boulder/blob/master/CONTRIBUTING.md#gating-migrations). This flag should be set to true **only** when the `20160817143417_CertStatusOptimizations.sql` migration has been applied.
Points of difference from #2132 (the initial preparatory "all-in-one go" PR):
**Note 1**: Updating the `isExpired` field in the OCSP updater can not be done yet, the `notAfter` field needs to be fully populated first - otherwise a separate query or a messy `JOIN` would have to be used to determine if a certStatus `isExpired` by using the `certificates` table's `expires` field.
**Note 2**: Similarly we can't remove the `JOIN` on `certificates` from the `findStaleOCSPResponse` query yet until all DB rows have `notAfter` populated. This will happen in a separate **Part Two** PR.
As detailed in issue #1872 the getSerialsIssuedSince function of the ocsp-updater cmd performs with poor runtime, likely due to a filesort and lack of index on the issued field.
This commit adds a migration to create a new index on the issued field.
* Add index to certificates table issued field.
* Rename index to use _idx suffix
Use bridged networking.
Add some files to .dockerignore to shrink the build state sent to Docker
daemon.
Use specific hostnames to contact services, rather than localhost.
Add instructions for adding those hostnames to /etc/hosts in non-Docker config.
Use DSN-style connect strings for DBs.
Remove localhost / 127.0.0.1 rewrite hack from create_db.sh.
Add hosts section with new hostnames.
Remove bin from .dockerignore.
SQL grants go to %
Short-circuit DB creation if already existing.
Make `go install` a part of Docker image build so that Docker run is much
faster.
Bind to 0.0.0.0 for OCSP responders so they can be reached from host, and
publish / expose their ports.
Remove ToSServerThread and test.js' fetch of ToS.
Increase the registrationsPerIP rate limit threshold. When issuing from a Docker
host, the 127.0.0.1 override doesn't apply, so the limit is quickly hit.
Update docker-compose for bridged networking. Note: docker-compose doesn't currently work, but should be close.
https://github.com/letsencrypt/boulder/pull/1639
Adds a new rate limit, certficatesPerFQDNSet, which counts certificates
with the same set of FQDNS using a table containing the hash of the dNSNames
mapped to a certificate serial. A new method is added to the SA in AddCertificate
to add this hash to the fqdnSets table, which is gated by a config bool.
Previously, when requesting a new certificate, we might churn through
many authz attempts to find the latest authz that was valid for a given
domain. There was an index on registration id, but a given registration
may have very many authzs. Think CDNs with one registration.
Updates #959
Fixes https://github.com/letsencrypt/boulder/issues/898
Also removes currently-unused 'development' DB, and do initial migrations in
parallel, which shortens create_db.sh from 20 seconds to 10 seconds.
Changes ResetTestDatabase into two functions, one each for SA and Policy DBs,
which take care of setting up the DB connection using a special higher-privileged
user called test_setup.
OCSP-Responder attempts to read the OCSP response from the certificateStatus table,
if it cannot find a response there it reads the ocspResponses table to try to find a
response, if neither contains a response the not found bool is passed back to the
Responder.
* Moves revocation from the CA to the OCSP-Updater, the RA will mark certificates as
revoked then wait for the OCSP-Updater to create a new (final) revoked response
* Merges the ocspResponses table with the certificateStatus table and only use UPDATES
to update the OCSP response (vs INSERT-only since this happens quite often and will
lead to an extremely large table)
also moves the first OCSP responses generation from the CA to the OCSP updater. This patch lays the
ground work for moving CT submission and adding CT backfill to the OCSP updater.
Jacob Hoffman-Andrews