Previously, if we did an up migration, generated some certs, then did a
down migration, we would truncate some of the new-style serial numbers in a way
that would be hard to reverse.
Previously, 32 bytes were used for serial numbers but now we make
certificates with 36 bytes (see #823 and #813). Going forward, we want
them to be consistent, so we update our current ones by prepending 4
zeros to the them.
For the two existing certs, their leading datacenter id will be 0 in
production, but we take this hit because we can't adjust the serial
inside the certificates themselves, and can only pad them with zeroes in
the database.
Fixes#834
To keep the change small, I have not yet completely removed the
GetCertificateByShortSerial method from interfaces and the RPC. I will do taht
in a follow up change.
Adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table.
The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts.
This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs.
* A fake CT server for integration testing
* A external tool to search the database for certificates lacking a full set of SCT receipts
* A method to construct X.509 v3 extensions containing receipts for the OCSP responder
* Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients)
Resolves#95.
Challenge URIs should be determined by the WFE at fetch time, rather than stored
alongside the challenge in the DB. This simplifies a lot of the logic, and
allows to to remove a code path in NewAuthorization where we create an
authorization, then immediately save it with modifications to the challenges.
This change also gives challenges their own endpoint, which contains the
challenge id rather than the challenge's offset within its parent authorization.
This is also a first step towards replacing UpdateAuthorization with
UpdateChallenge: https://github.com/letsencrypt/boulder/issues/760.
The RA did not have any code to test what occurred when a challenge
failed. This let in the authz schema change in #705.
This change sets the expires column in authz back to NULLable and fixes
the RA tests (including, using clock.Clocks in the RA).
Fixes#744.
The ca's TestRevoke was failing occasionally.
The test was saying "has the certificate's OCSPLastUpdated been set to a
time within the last second?" as a way to see if the revocation updated
the OCSPLastUpdated. OCSPLastUpdated was not being set on revocation,
but the test still passed most of the time.
The test still passed most of the time because the creation of the
certificate (which also sets the OCSPLastUpdated) has usually happened
within the last second. So, even without revocation, the OCSPLastUpdated
was set to something in the last second because the test is fast.
Threading a clock.FakeClock through the CA induced the test to fail
consistently. Debugging and threading a FakeClock through the SA caused
changes in times reported but did not fix the test because the
OCSPLastUpdated was simply not being updated. There were not tests for
the sa.MarkCertificateRevoked API that was being called by
ca.RevokeCertificate.
Now the SA has tests for its MarkCertificateRevoked method. It uses a
fake clock to ensure not just that OCSPLastUpdated is set correctly, but
that RevokedDate is, as well. The test also checks for the
CertificateStatus's status and RevocationCode changes.
The SA and CA now use Clocks throughout instead of time.Now() allowing
for more reliable and expansive testing in the future.
The CA had to gain a public Clock field in order for the RA to use the
CertificateAuthorityImpl struct without using its constructor
function. Otherwise, the field would be nil and cause panics in the RA
tests.
The RA tests are similarly also panicking when the CAImpl attempts to
log something with its private, nil-in-those-tests log field but we're
getting "lucky" because the RA tests only cause the CAImpl to log when
they are broken.
There is a TODO there to make the CAImpl's constructor function take
just what it needs to operate instead of taking large config objects and
doing file IO and such. The Clk field should be made private and the log
field filled in for the RA tests.
Fixes#734.
Avoids silently allowing bad things to happen (like mismatched data
types, strings being truncated, etc.).
Ensures that MySQL/MariaDB warnings are treated as errors. This avoids a
number of nasty edge conditions we could wander into. Common things this
discovers includes places where data being sent had a different type
than what is in the schema, strings being truncated, writes of null to a
NOT NULL column, and so on. See
<https://dev.mysql.com/doc/refman/5.0/en/sql-mode.html#sql-mode-strict>.
MySQL is very funny.
Fixes#623
This has required some substantive changes to the tests. Where
previously the foreign key constraints did not exist in the tests, now
that we use the actual production schema, they do. This has mostly led
to having to create real Registrations in the sa, ca, and ra tests. Long
term, it would be nice to fake this out better instead of needing a real
sa in the ca and ra tests.
The "goose" being referred to is <https://bitbucket.org/liamstask/goose>.
Database migrations are stored in a _db directory inside the relevant
owner service (namely, ca/_db, and sa/_db, today).
An example of migrating up with goose:
goose -path ./sa/_db -env test up
An example of creating a new migration with goose:
goose -path ./sa/_db -env test create NameOfNewMigration sql
Notice the "sql" at the end. It would be easier for us to manage sql
migrations. I would like us to stick to only them. In case we do use Go
migrations in the future, the underscore at the beginning of "_db" will
at least prevent build errors when using "..." with goose-created Go
files. Goose-created Go migrations do not compile with the go tool but
only with goose.
Fixes#111
Unblocks #623
There's no reason to not just enforce it in the code completely and not
worry about it elsewhere. If we need to extract code out of NewDbMap,
we'd need recombineURLForDB in it, anyhow.
This changes moves from using SQLite in the integration tests and in the
test/boulder-config.json.
It does not port the unit tests over, unfortunately. That's a much more
invasive change.
This also updates the Dockerfile to include the MariaDB and RabbitMQ
requirements of start.py as well as adjusts the CMD to expose the
boulder server to the host machine. The Dockerfile also needed to have
its Go version bumped and the test.sh had to grow some explict
"function"s.
Updates #132
Fixes#579 (which blocks #132).
This changes the SA to use a unique index on the sha256 of a
Registration's JWK's public key data instead of on the full serialized
JSON of the JWK. This corrects multiple problems:
1. MySQL/Mariadb no longer complain about key's being larger than the
largest allowed key size in an index
2. We no longer have to worry about large keys not being seen as unique
3. We no longer have to worry about the JWK's JSON being serialized with its inner keys in different orders and causing incorrectly empty queries or non-unique writes.
This change also hides the details of how Registrations are stored in
the database from the other services outside of SA. This will give us
greater flexibility if we need to move them to another database, or
change their schema, etc.
Also, adds some tests for NoSuchRegistration in the SA.
We move the admin-revoker and expiration-mailer to using the
SA.GetRegistration RPC method instead of digging into the database
itself.
This allows the hiding of the registration model layer inside of SA, so
we can do fancy things with sha256 for the unique index inside of
it. This will happen in a later commit. See #579.
By exposing fewer details about how Registration is stored, we gain more
flexibility to fix up how its stored.
In the expiration-mailer, the performance hit for the early filtering of
mailto is likely neglibible and possibly even a benefit given the cost
of joins to the memory of MySQL.
If need be, we can built a bulk RPC layer for SA that provides the data
we need in findExpiringCertificates. It'll be easier than trying to
scale and change the storage layer underneath for each consumer.
Fixes https://github.com/letsencrypt/boulder/issues/539.
Passes a pointer to tx.Update() in the SA, resolving the gorp error we were
previously receiving in UpdateOCSP.
Fixes CA code to properly receive the error from UpdateOCSP, so future errors
will be logged correctly.
This is the result of `godep save -r ./...` and
`git rm -r -f Godeps/_workspace/src/github.com/square`
Our fork is currently at the head of go-jose when Richard made the local nonce
changes, with the nonce changes added on top. In other words, the newly created
files are exactly equal to the deleted files.
In a separate commit I will bring our own go-jose fork up to the remote head,
then update our deps.
Also note: Square's go-jose repo contains a `cipher` package. Since we don't
make any changes to that package, we leave it imported as-is.
This approach performs a best-effort generation of the first OCSP response during
certificate issuance. In the event that OCSP generation fails, it logs a warning at
the Boulder-CA console, but returns successfully since the Certificate was itself
issued.
- Correct the Database implementation to match current schema & GORP
- Add some debug logging
- Correct CA Key identification
- Move SQL Debugging into the Database object and file, so that things that don't use the Storage Authority can toggle it
- NewPendingAuthorization now uses a core.Authorization object, so
that foreign key constraints are followed
- core.Authorization now serializes RegistrationID to JSON, so it has to get
blanked out in WFE before transmission to client.
- Remove ParsedCertificate from core.Certificate, as type x509.Certificate cannot
be marshaled.
- Added AssertDeepEquals and AssertMarhsaledEquals to test-tools.go
- Caught several overloaded and misleadingly named errors in WFE
- Fix a bunch of typos in rpc-wrappers.go
- Unblank `id` in core.Registration JSON:
- It's not spec, but it's not hurting anything, and we reveal it to clients anyway.
- We need knowledge of the ID in RPC, so if we don't want to include this in the object, we need to make a transfer object to wrap it.
- Make the RPC logs much clearer as to who's talking to who
- Typo in WFE where we called a registration an authz
- Move dbMap construction and type converter into individual files in the sa package.
- Add DB configuration for the OCSP tool to the boulder config:
- left to the user if they want to use different boulder-config.json files
for different purposes.
- Added updater to Makefile
- Fix trailing ',' in the Boulder config, add more panic logging
- Ignore .pem files produced by the integration test
- Change RPC to use per-instance named reply-to queues.
- Finish OCSP Updater logic
- Rework RPC for OCSP to use a transfer object (due to serialization problems of x509.Certificate)
- First steps towards ocsp updater.
- Move the type converter into its own file in "sa" rather than core, to keep
Gorp from leaking out of the SA package.
- Add SQL configuration options
- Increase the width of the authz and pending_authz tables' challenges field
- Make it configurable whether CREATE TABLE commands should run
- Added SQL debug logging (SA option: "SQLDebug")
- Added timestamps to the log prints to stdout
- Ignore *.pem in test/js
- Modified start.sh to support environment overrides for BOULDER_CONFIG, like the AMQP mode
- Changed boulder-test-config to open the server on the loopback device, so as to not cause firewall prompts on each integration test run for those of us being restrictive
- Renamed "key" column to "jwk" in DB, to avoid keyword conflict
- Set MaxLength on "jwk" column to 512
Jacob Hoffman-Andrews