Adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table.
The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts.
This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs.
* A fake CT server for integration testing
* A external tool to search the database for certificates lacking a full set of SCT receipts
* A method to construct X.509 v3 extensions containing receipts for the OCSP responder
* Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients)
Resolves#95.
The changes to use an existing key depended on Node 0.12, but Travis provides
Node 0.10 (unless you specify language: node and a version, but we have
language: go).
Add an easy script to build and run the Docker instance.
Update some out-of-date information in the README.
Add goose to the Docker image.
Remove unnecessary go install step from Dockerfile.
Allow dns-test-srv to return a hardcoded address other than localhost. This was
preventing a Dockerized Boulder from answering requests from a letsencrypt
client on the host.
Change allowLoopbackAddresses to allowRestrictedAddresses and make it cover all
the private IPv4 ranges. The host IP in Docker is commonly in the 172.* range.
Fix a couple of references to lets-encrypt-preview.
This was inspired by investigation into https://github.com/letsencrypt/boulder/issues/756.
To try and reproduce, I tried running Boulder inside a container, and found some
broken things.
This gets us down to 5-6 minute build times.
A new script had to be added to run make conditionally without allowing
builds with make failures to pass.
Always pass Content-Length header
Always use tls: true
Don't write terms of service contents to terminal: it might be a PDF
Separate out signing, posting, and printing code into acme.js
Use acme.js in revoke.js for better error reporting.
Check for PEM passed to revoke.js.
This is in response to some trouble JC had using the client to test in prod. In
particular, Akamai insists on Content-Length (we turned this off for a while to
match Boulder, but now it's back on), and the PEM / DER distinction was a
stumbling block.
This makes the test terminate after the assertion failure. Especially in the
case of AssertNotError, this can avoid triggering an immediate nil dereference
panic that obscures the error message.
This removes TestMode from the boulder-va command, from ca.Config
(it was only used in the VA) and gets the integration config to specify
the ports it should use explicitly.
(It also removes a DBDriver field from ca.Config that was left over from
letsencrypt/boulder#624.)
Fixes#627.
Changes this to use just communicate(), not the subprocess.PIPE stuff (which
apparently can do Weird Things)
Also rename the install variable to cmd in the install function
This eases the CPU and thread requirements of our tests (by forking
less, not doing everything at once). It should also speed up the tests
by avoiding certain repetitive work.
Updates https://github.com/letsencrypt/letsencrypt/issues/712
This has required some substantive changes to the tests. Where
previously the foreign key constraints did not exist in the tests, now
that we use the actual production schema, they do. This has mostly led
to having to create real Registrations in the sa, ca, and ra tests. Long
term, it would be nice to fake this out better instead of needing a real
sa in the ca and ra tests.
The "goose" being referred to is <https://bitbucket.org/liamstask/goose>.
Database migrations are stored in a _db directory inside the relevant
owner service (namely, ca/_db, and sa/_db, today).
An example of migrating up with goose:
goose -path ./sa/_db -env test up
An example of creating a new migration with goose:
goose -path ./sa/_db -env test create NameOfNewMigration sql
Notice the "sql" at the end. It would be easier for us to manage sql
migrations. I would like us to stick to only them. In case we do use Go
migrations in the future, the underscore at the beginning of "_db" will
at least prevent build errors when using "..." with goose-created Go
files. Goose-created Go migrations do not compile with the go tool but
only with goose.
Fixes#111
Unblocks #623
If two OCSP responses were generated in the same second, the earlier would
previously take priority sometimes, leading to a "good" response for revoked
certificates and causing the OCSP integration test to be flaky.
There's no reason to not just enforce it in the code completely and not
worry about it elsewhere. If we need to extract code out of NewDbMap,
we'd need recombineURLForDB in it, anyhow.
This changes moves from using SQLite in the integration tests and in the
test/boulder-config.json.
It does not port the unit tests over, unfortunately. That's a much more
invasive change.
This also updates the Dockerfile to include the MariaDB and RabbitMQ
requirements of start.py as well as adjusts the CMD to expose the
boulder server to the host machine. The Dockerfile also needed to have
its Go version bumped and the test.sh had to grow some explict
"function"s.
Updates #132
Run builds in parallell as well as starting servers in parallel.
Wait for the servers to come up, so tests don't start running too early.
Enable race detection only for the integration test, not for start.py.
Previously I'd suggested it should always be on, but after running with it for a
while I'm convinced it's too slow for start.py (but still very valuable for
integration tests!).
The race detector has found at least one race in our current code. See
issue #465. Turn it on for the unit and integration tests running in
TravisCI.
Also, allow the local user to add new test flags with the `GOTESTFLAGS`
environment variable.
To ease speed of debugging issues, the ability to skip the unit or
integration tests is also provided.
Currently, the debug http server in every service contains just the
net/http/pprof handlers. This allows us to get CPU, blocking, and memory
profiling remotely.
Along the way, remove all the places we use http.DefaultServeMux (which
includes use of http.Handle and http.HandlerFunc) and use a NewServeMux
for each place.
Fixes#457
amqp-integration-test.py gains a way to print out what processes
failed to start.
test.sh gains:
* the ability to continue the build correctly if
LETSENCRYPT_PATH was provided but does not exist on disk.
* an explanatory exit message if the LETSENCRYPT_PATH does
exist, but there is not finished build in it.
* a working detection for a python 2.7 binary on OS X
This uses a node.js module to post `status` updates to Github, and uses a Travis
secret to authenticate.
- Post comments from static analysis tools
- Change to posting from LetsEncryptBot
- For integration testing, only fail if the compile fails, or
the NodeJS-client fails. Log if the Python client fails.
Travis:
* Downloads the Let's Encrypt client
* Installs system requirements for client
* Sets up virtualenv
Dockerfile:
* Buildout for development
* Includes numerous pacakges needed for integration testing
(including all of the above in Travis)
test.sh:
* If no path is defined for the LE client
* Download the Let's Encrypt client
* Set up virtualenv
test/amqp-integration-test.py:
* Run client test with sensible defaults
* One test: auth for foo.com
Also, remove dependency on cfssl CLI binary, and transitive dependency cf-tls.
These are no longer necessary now that we use the local signer. And the cf-tls
dependency had drifted out of date, causing build issues when I updated cfssl to
master.
This allows us to use the same PKCS#11 key for both cert signing and OCSP
signing, and simplifies config and startup.
This also starts building with -tags pkcs11 in all scripts, which is required
now that the CA can choose between pkcs11 and non-pkcs11.
In order to successfully issue using a pkcs11 key, you'll need to run a version
of Go built off the master branch. The released versions are missing this
commit:
fe40cdd756,
which is necessary for PKCS#11 signing.
- Moved the transaction handling up to the `certificate-authority.go` file
- Simplified `certificate-authority-data.go`
- Created a mocks file in `test/` and reworked RA and CA to use it
- More audit logging to CA
Clean up tempfiles on exit.
Print exceptions instead of hiding them.
Exit early if a build fails, and clean up processes that are running at the time.
Update README to reflect RabbitMQ requirement.
- NewPendingAuthorization now uses a core.Authorization object, so
that foreign key constraints are followed
- core.Authorization now serializes RegistrationID to JSON, so it has to get
blanked out in WFE before transmission to client.
- Remove ParsedCertificate from core.Certificate, as type x509.Certificate cannot
be marshaled.
- Added AssertDeepEquals and AssertMarhsaledEquals to test-tools.go
- Caught several overloaded and misleadingly named errors in WFE
- Move dbMap construction and type converter into individual files in the sa package.
- Add DB configuration for the OCSP tool to the boulder config:
- left to the user if they want to use different boulder-config.json files
for different purposes.
- Added updater to Makefile
- Fix trailing ',' in the Boulder config, add more panic logging
- Ignore .pem files produced by the integration test
- Change RPC to use per-instance named reply-to queues.
- Finish OCSP Updater logic
- Rework RPC for OCSP to use a transfer object (due to serialization problems of x509.Certificate)
- Add SQL configuration options
- Increase the width of the authz and pending_authz tables' challenges field
- Make it configurable whether CREATE TABLE commands should run
- Added SQL debug logging (SA option: "SQLDebug")
- Added timestamps to the log prints to stdout
- Ignore *.pem in test/js
- Modified start.sh to support environment overrides for BOULDER_CONFIG, like the AMQP mode
- Changed boulder-test-config to open the server on the loopback device, so as to not cause firewall prompts on each integration test run for those of us being restrictive
- Renamed "key" column to "jwk" in DB, to avoid keyword conflict
- Set MaxLength on "jwk" column to 512
Include updates to test.js to make its output more useful as a diagnostic.
It remains a future TODO to do integration testing with the real letsencrypt
client.
Also, work around a go vet bug.
Per the spec, authenticated requests must be signed by an account key, and
GET requests can't be signed under the current protocol. If the account holder
wishes to fetch their current registration, they can do so by posting a signed,
empty update to their registration resource.
Also fix a bug in generating registration URLs.
- Auditing for general errors in executables
- Auditing for improper messages received by WFE
- Automatic audit wlogging of software errors
- Audit logging for mis-routed messages
- Audit logging for certificate requests
- Auditing for improper messages received by WFE
- Add audit events table
- Expect more details in TestRegistration in web-front-end_test.go
- Remove "extra" debug details from web-front-end.go per Issue #174
- Improve test coverage of web-front-end.go
- WFE audit updates for revocation support rebase
- Add audit messages to RPC for Improper Messages and Error Conditions
- Also note misrouted messages
Use request instead of http so it works with either HTTP or HTTPS URLs.
Write DER certificate rather than PEM certificate. I was getting some bytewise
mismatches when checking the output PEM against a downloaded DER.
- Upstream CFSSL update broke a test in certificate-authority_test.go
See https://github.com/cloudflare/cfssl/issues/156
- Added "SerialPrefix" to config, letting you set a prefix to the generated
serial numbers.
This fixes the problem Kuba reported on IRC of receiving messages like:
[123 34 100 101 116 97 105 108 34 58 34 77 101 116 104 111 100 32 110 111 116 32
97 108 108 111 119 101 100 34 125]
from Boulder.
This changelist also adds the beginning of a test to WFE, but much more is
needed.
Jacob Hoffman-Andrews