B64enc and B64dec can be replaced by base64.RawURLEncoding.
Thumbprint is now implemented in go-jose, and we have the relevant version
imported already, so we can use that.
SyntaxError isn't used anywhere and can be deleted.
Adds a dns-01 type validation to test.js and reworks dns-test-srv to allow changing TXT record values.
Also makes some changes to how integration-test.py works in order to reduce complexity now the
ct-test-srv is working again.
This uses ProblemDetails throughout the wfe. This is the last step in
allowing the backend services to pass ProblemDetails from RPCs through
to the user.
Updates #1153.
Fixes#1161.
Moves the DNS code from core to dns and renames the dns package to bdns
to be clearer.
Fixes#1260 and will be good to have while we add retries and such.
This is a change to ValidationRecord. This case is unlikely to be
trigged by code, but was allowing tests to pass in a branch that deleted
the simpleHttp and dvsni challenge types and is a good check to have in
place.
Updates #894
Generally Dial will be very fast because our resolver is local, so there's no
need to override its default of 2s. However, since our resolver recurses more or
less every time, getting the answer back is very slow. So we want to be able to
set a high ReadTimeout.
Since Boulder always requests DNSSEC records, in practice DNS responses often
exceed the IP MTU.
Boulder installations expect to have a local DNS resolver, and all modern DNS
resolvers support TCP connections. Since miekg/dns does not perform an
"attempt udp, timeout, retry via tcp" approach, it's simpler and more reliable
to always use TCP for internal DNS resolution. This makes failures more
obvious as well.
Also change the integration test DNS server to TCP.
This allows us to call the Google Safe Browsing calls through the VA.
If the RA config's boolean UseIsSafeDomain is true, the RA will make the RPC
call to the VA during its NewAuthorization.
If the VA config's GoogleSafeBrowsingConfig struct is not nil, the VA
will check the Google Safe Browsing API in
VA.IsSafeDomain. If the GoogleSafeBrowsingConfig struct is nil, it will
always return true.
In order to actually make requests, the VA's GoogleSafeBrowsingConfig
will need to have a directory on disk it can store the local GSB hashes
it will check first and a working Google API key for the GSB API.
Fixes#1058
If a ServiceUnavailableError is returned from GenerateOCSP backoff before
attempting to retry the call as to not to overwhelm the CA with calls that
may instantly fail.
* Moves revocation from the CA to the OCSP-Updater, the RA will mark certificates as
revoked then wait for the OCSP-Updater to create a new (final) revoked response
* Merges the ocspResponses table with the certificateStatus table and only use UPDATES
to update the OCSP response (vs INSERT-only since this happens quite often and will
lead to an extremely large table)
This change lowercases domains before they are stored in the database
and makes policy.WillingToIssue reject any domains with uppercase
letters.
Fixes#927.
B64enc makes some nasty allocations through its use of strings.Replace
in unpad. This changes that strings.Replace into a simple for-loop.
B64enc gets used in many places, including the rpc library on every
request and response. While we should probably not use it in the rpc
library (#909), there are enough other places it's used (now or in the near
future) that make this valuable.
Was a performance problem found during early load-testing (#20) of the
CA. More to come.
Currently, the whitelisted registration ID is one that is impossible for the
database to return. Once the partner's registration is in place, we can
deploy a change to it.
Fixes#810
instead of submitted key. This minimizes the chances of unexpected JWK fields in
the submitted key altering its interpretation without altering the lookup in the
registrations table.
In the process, fix handling of NoSuchRegistration responses.
Fixes https://github.com/letsencrypt/boulder/issues/865.
* Rewrite JSONDuration as ConfigDuration that can handle both JSON and YAML unmarshaling
* Factor out RPC certificate count request struct
* Return 429 to WFE on rate limit exceeded
* Fix wonky RateLimitPolicy comment
To keep the change small, I have not yet completely removed the
GetCertificateByShortSerial method from interfaces and the RPC. I will do taht
in a follow up change.
Adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table.
The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts.
This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs.
* A fake CT server for integration testing
* A external tool to search the database for certificates lacking a full set of SCT receipts
* A method to construct X.509 v3 extensions containing receipts for the OCSP responder
* Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients)
Resolves#95.
The WFE test relies on a pre-generated cert. Since there are some sanity checks
on the dates in certs, we were getting errors during the test.
One quick fix is to have those sanity checks rely on RA's clock object, which
can be replaced with a fake for testing. In order to do that, I had to move the
sanity check (MatchesCSR) into the registration authority package, where it
makes more sense anyhow.
I also removed a handful of equality testing functions in objects.go that were
only used by MatchesCSR and whose purpose is better served by reflect.DeepEqual.
This was to avoid having to also move those equality testing functions into the
registration authority.
Challenge URIs should be determined by the WFE at fetch time, rather than stored
alongside the challenge in the DB. This simplifies a lot of the logic, and
allows to to remove a code path in NewAuthorization where we create an
authorization, then immediately save it with modifications to the challenges.
This change also gives challenges their own endpoint, which contains the
challenge id rather than the challenge's offset within its parent authorization.
This is also a first step towards replacing UpdateAuthorization with
UpdateChallenge: https://github.com/letsencrypt/boulder/issues/760.
Add an easy script to build and run the Docker instance.
Update some out-of-date information in the README.
Add goose to the Docker image.
Remove unnecessary go install step from Dockerfile.
Allow dns-test-srv to return a hardcoded address other than localhost. This was
preventing a Dockerized Boulder from answering requests from a letsencrypt
client on the host.
Change allowLoopbackAddresses to allowRestrictedAddresses and make it cover all
the private IPv4 ranges. The host IP in Docker is commonly in the 172.* range.
Fix a couple of references to lets-encrypt-preview.
This was inspired by investigation into https://github.com/letsencrypt/boulder/issues/756.
To try and reproduce, I tried running Boulder inside a container, and found some
broken things.
This has required some substantive changes to the tests. Where
previously the foreign key constraints did not exist in the tests, now
that we use the actual production schema, they do. This has mostly led
to having to create real Registrations in the sa, ca, and ra tests. Long
term, it would be nice to fake this out better instead of needing a real
sa in the ca and ra tests.
The "goose" being referred to is <https://bitbucket.org/liamstask/goose>.
Database migrations are stored in a _db directory inside the relevant
owner service (namely, ca/_db, and sa/_db, today).
An example of migrating up with goose:
goose -path ./sa/_db -env test up
An example of creating a new migration with goose:
goose -path ./sa/_db -env test create NameOfNewMigration sql
Notice the "sql" at the end. It would be easier for us to manage sql
migrations. I would like us to stick to only them. In case we do use Go
migrations in the future, the underscore at the beginning of "_db" will
at least prevent build errors when using "..." with goose-created Go
files. Goose-created Go migrations do not compile with the go tool but
only with goose.
Fixes#111
Unblocks #623
Refactor DNS problem details use
Actually store and log resolved addresses
Less convuluted get adresses function/usage
Store redirects, reconstruct transport on redirect, add redirect + lookup tests
Add another test
Review fixes
Initial bulk of review fixes (cleanups inc)
Comment cleanup
Add some more tests
Cleanups
Give addrFilter a type and add the config wiring
Expose filters
LookupHost cleanups
Remove Resolved Addresses and Redirect chain from replies to client without breaking RPC layer
Switch address/redirect logging method, add redirect loop checking + test
Review fixes + remove IPv6
Remove AddressFilter remnant + constant-ize the VA timeout
Review fixes pt. 1
Initialize validation record
Don't blank out validation reocrds
Add validation record sanity checking
Switch to shared struct
Check port is in valid range
Review fixes
Fixes#579 (which blocks #132).
This changes the SA to use a unique index on the sha256 of a
Registration's JWK's public key data instead of on the full serialized
JSON of the JWK. This corrects multiple problems:
1. MySQL/Mariadb no longer complain about key's being larger than the
largest allowed key size in an index
2. We no longer have to worry about large keys not being seen as unique
3. We no longer have to worry about the JWK's JSON being serialized with its inner keys in different orders and causing incorrectly empty queries or non-unique writes.
This change also hides the details of how Registrations are stored in
the database from the other services outside of SA. This will give us
greater flexibility if we need to move them to another database, or
change their schema, etc.
Also, adds some tests for NoSuchRegistration in the SA.
Only if the cache returns nothing for the CNAME query do we need to
look up CNAME/DNAME explicitly, in order to check CAAs on the parent
of the CNAME target rather than the parent of the original name.
This is the result of `godep save -r ./...` and
`git rm -r -f Godeps/_workspace/src/github.com/square`
Our fork is currently at the head of go-jose when Richard made the local nonce
changes, with the nonce changes added on top. In other words, the newly created
files are exactly equal to the deleted files.
In a separate commit I will bring our own go-jose fork up to the remote head,
then update our deps.
Also note: Square's go-jose repo contains a `cipher` package. Since we don't
make any changes to that package, we leave it imported as-is.
This allows us to log the remote address and registration object along with the
CSR.
Also, restore part of a comment on CertificateRequest that was deleted.
- Created some helper methods to run DNSSEC and reduce code reuse
- Support multiple DNS servers, but not in the Config file (yet)
- Fix typo; r=@rolandshoemaker
This approach performs a best-effort generation of the first OCSP response during
certificate issuance. In the event that OCSP generation fails, it logs a warning at
the Boulder-CA console, but returns successfully since the Certificate was itself
issued.
- Moved the transaction handling up to the `certificate-authority.go` file
- Simplified `certificate-authority-data.go`
- Created a mocks file in `test/` and reworked RA and CA to use it
- More audit logging to CA
- NewPendingAuthorization now uses a core.Authorization object, so
that foreign key constraints are followed
- core.Authorization now serializes RegistrationID to JSON, so it has to get
blanked out in WFE before transmission to client.
- Remove ParsedCertificate from core.Certificate, as type x509.Certificate cannot
be marshaled.
- Added AssertDeepEquals and AssertMarhsaledEquals to test-tools.go
- Caught several overloaded and misleadingly named errors in WFE
- Fix a bunch of typos in rpc-wrappers.go
- Unblank `id` in core.Registration JSON:
- It's not spec, but it's not hurting anything, and we reveal it to clients anyway.
- We need knowledge of the ID in RPC, so if we don't want to include this in the object, we need to make a transfer object to wrap it.
- Make the RPC logs much clearer as to who's talking to who
- Typo in WFE where we called a registration an authz
- Move dbMap construction and type converter into individual files in the sa package.
- Add DB configuration for the OCSP tool to the boulder config:
- left to the user if they want to use different boulder-config.json files
for different purposes.
- Added updater to Makefile
- Fix trailing ',' in the Boulder config, add more panic logging
- Ignore .pem files produced by the integration test
- Change RPC to use per-instance named reply-to queues.
- Finish OCSP Updater logic
- Rework RPC for OCSP to use a transfer object (due to serialization problems of x509.Certificate)
- Add SQL configuration options
- Increase the width of the authz and pending_authz tables' challenges field
- Make it configurable whether CREATE TABLE commands should run
- Added SQL debug logging (SA option: "SQLDebug")
- Added timestamps to the log prints to stdout
- Ignore *.pem in test/js
- Modified start.sh to support environment overrides for BOULDER_CONFIG, like the AMQP mode
- Changed boulder-test-config to open the server on the loopback device, so as to not cause firewall prompts on each integration test run for those of us being restrictive
- Renamed "key" column to "jwk" in DB, to avoid keyword conflict
- Set MaxLength on "jwk" column to 512
- Move setting the core.Registration.Key field from RA.NewRegistration to
WFE.NewRegistration to avoid a chicken-and-egg problem.
- Note: I kept the RPC wrapper object even though it now only has one field.
Seems like it's a good practice to use wrapper objects, even though we don't
everywhere.
- Don't ignore entropy underruns in challenges.go
- Correct identity crisis in Policy Authority; hopefully it will remember.
- Add a method `AuditObject` in audit-logger and convert RA/VA to use it
- Fix json typo in registration-authority that caused empty audit logs
- Fix vet issue in WFE where RegID was being printed as a 32-bit int instead of 64-bit
- Unfix the issue in WFE where RegID isn't right, per PR #215
- Auditing for general errors in executables
- Auditing for improper messages received by WFE
- Automatic audit wlogging of software errors
- Audit logging for mis-routed messages
- Audit logging for certificate requests
- Auditing for improper messages received by WFE
- Add audit events table
- Expect more details in TestRegistration in web-front-end_test.go
- Remove "extra" debug details from web-front-end.go per Issue #174
- Improve test coverage of web-front-end.go
- WFE audit updates for revocation support rebase
- Add audit messages to RPC for Improper Messages and Error Conditions
- Also note misrouted messages
Richard Barnes
Roland Shoemaker
Hugo Landau
Jeff Hodges
Alex Gaynor
Jacob Hoffman-Andrews
J.C. Jones
bifurcation
Jeremy Gillula
Tom Clegg
Romain Fliedel
Brad Warren
James 'J.C.' Jones