This PR updates the `publicsuffix-go` dependency to `fb1fc94`, the
latest autopull and the HEAD of master at the time of writing.
Per CONTRIBUTING.md the tests were verified to pass:
```
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.007s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.027s
```
Updates the various gRPC/protobuf libs (google.golang.org/grpc/... and github.com/golang/protobuf/proto) and the boulder-tools image so that we can update to the newest github.com/grpc-ecosystem/go-grpc-prometheus. Also regenerates all of the protobuf definition files.
Tests run on updated packages all pass.
Unblocks #2633fixes#2636.
Adds a daemon mode to `expiration-mailer` that is triggered by using the flag `--daemon` in order to follow deployability guidelines. If the `--daemon` flag is used the `mailer.runPeriod` config field is checked for a tick duration, if the value is `0` it exits.
Super lightweight implementation, OCSP-Updater has some custom ticker code which we use to do fancy things when the method being invoked in the loop takes longer expected, but that isn't necessary here.
Fixes#2617.
This fixes an issue caused by #2583. Prior to that PR, we would serve the "invalidEmail" problem type when a DNS lookup for an email base domain failed. After that PR, we would map "berrors.InvalidEmail" to the "InternalServerError" problem type, which caused 500 errors to be returned to the user.
This PR restores the behavior of returning "type": "...invalidEmail" to the user.
Fixes#2148.
Instead of just doing a blanket `DELETE FROM ...` this changes the `expired-authz-purger` to select all of the expired IDs (for both pending and finalized authorizations) then loop over them deleting each and its associated challenges from their respective tables.
Local testing indicates the performance of this is not awful but we should do a test run on staging to verify. If it ends up taking way too long to run there the easiest optimization would be to turn the slice of IDs into a channel and run multiple workers looping over the channel deleting stuff instead of just a single one.
Makes a few small integration test changes in order to facilitate deleting both pending and finalized authorizations.
This patch removes all usages of the `core.XXXError` and almost all usages of `probs` outside of the WFE and VA and replaces them with a unified internal error type. Since the VA uses `probs.ProblemDetails` quite extensively in challenges, and currently stores them in the DB I've saved this change for another change (it'll also require a migration). Since `ProblemDetails` should only ever be exposed to end-users all of its related logic should be moved into the `WFE` but since it still needs to be exposed to the VA and SA I've left it in place for now.
The new internal `errors` package offers the same convenience functions as `probs` does as well as a new simpler type testing method. A few small changes have also been made to error messages, mainly adding the library and function name to internal server errors for easier debugging (i.e. where a number of functions return the exact same errors and there is no other way to distinguish which method threw the error).
Also adds proper encoding of internal errors transferred over gRPC (the current encoding scheme is kept for `core` and `probs` errors since it'll be ideally be removed after we deploy this and follow-up changes) using `grpc/metadata` instead of the gRPC status codes.
Fixes#2507. Updates #2254 and #2505.
I think these are all the necessary changes to implement TLS-SNI-02 validations, according to the section 7.3 of draft 05:
https://tools.ietf.org/html/draft-ietf-acme-acme-05#section-7.3
I don't have much experience with this code, I'll really appreciate your feedback.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This commit resolves#2599 by adding support to the expiration-mailer to
treat the subject for email messages as a template. This allows for the
dynamic subject lines from #2435 to be used with a prefix for staging
emails.
Presently the CA and the ocsp-updater can race on the initial
submission of a certificate to the configured logs. This results
in double submitting certificates. In integration tests with the fake CT
server this manifests as an occasional failure of the
`test_ct_submission` test (Issue #2579).
The race we currently experience is expected to be fixed in
the future by a planned redesign so for now this commit works around the
failure by allowing either the expected number of submissions, or
exactly double the expected. This fixes#2579. The need to fix the
underlying race was captured in #2610.
The workaround was verified by submitting 10 builds to travis, all
succeeded.
The VA's `getAddr` function prior to this commit had an outdated comment
& a pointer to a TODO for Boulder Issue #593. That issue has been closed
and bdns' `LookupHost` supports AAAA records now. This commit updates
the comment to match the current behaviour and removes the TODO.
Switch from `gorp.v1` to `gorp.v2`. Removes `vendor/gopkg.in/gorp.v1` and vendors `vendor/gopkg/go-gorp/gorp.v2`, all tests pass.
Changes between `v1.7.1` and `v2.0.0`: c87af80f3c...4deece6103Fixes#2490.
This commit updates the `publicsuffix-go` dependency to version 0.3.2,
the latest autopull.
Per CONTRIBUTING.md the tests are verified to pass:
```
HEAD position was 5ebfcac... Fix outdated version number
HEAD is now at c12e7e9... autopull: 2017-03-04T06:00:47Z (#62)
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
=== RUN TestPublicSuffix
--- PASS: TestPublicSuffix (0.00s)
=== RUN TestEffectiveTLDPlusOne
--- PASS: TestEffectiveTLDPlusOne (0.00s)
PASS
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.007s
=== RUN TestValid
--- PASS: TestValid (0.00s)
=== RUN TestIncludePrivate
--- PASS: TestIncludePrivate (0.00s)
=== RUN TestIDNA
--- PASS: TestIDNA (0.00s)
=== RUN TestPsl
--- PASS: TestPsl (0.01s)
=== RUN TestNewListFromString
--- PASS: TestNewListFromString (0.00s)
=== RUN TestNewListFromString_IDNAInputIsUnicode
--- PASS: TestNewListFromString_IDNAInputIsUnicode (0.00s)
=== RUN TestNewListFromString_IDNAInputIsAscii
--- PASS: TestNewListFromString_IDNAInputIsAscii (0.00s)
=== RUN TestNewListFromFile
--- PASS: TestNewListFromFile (0.00s)
=== RUN TestListAddRule
--- PASS: TestListAddRule (0.00s)
=== RUN TestListFind
--- PASS: TestListFind (0.00s)
=== RUN TestNewRule_Normal
--- PASS: TestNewRule_Normal (0.00s)
=== RUN TestNewRule_Wildcard
--- PASS: TestNewRule_Wildcard (0.00s)
=== RUN TestNewRule_Exception
--- PASS: TestNewRule_Exception (0.00s)
=== RUN TestNewRule_FromASCII
--- PASS: TestNewRule_FromASCII (0.00s)
=== RUN TestNewRule_FromUnicode
--- PASS: TestNewRule_FromUnicode (0.00s)
=== RUN TestNewRuleUnicode_FromASCII
--- PASS: TestNewRuleUnicode_FromASCII (0.00s)
=== RUN TestNewRuleUnicode_FromUnicode
--- PASS: TestNewRuleUnicode_FromUnicode (0.00s)
=== RUN TestRuleMatch
--- PASS: TestRuleMatch (0.00s)
=== RUN TestRuleDecompose
--- PASS: TestRuleDecompose (0.00s)
=== RUN TestLabels
--- PASS: TestLabels (0.00s)
=== RUN TestCookieJarList
--- PASS: TestCookieJarList (0.00s)
PASS
ok github.com/weppos/publicsuffix-go/publicsuffix 0.027s
```
Initial work on this was ripped out of the 'coyote' branch since it's better suited
to being worked on separately.
This adds cmd/load-generator, a tool for generating load on various publicly facing
boulder services. It currently implements a generator for the WFE and one for the
OCSP-Responder.
This also adds a latency-charter.py that consumes output from load-generator to generate plots of latency during load tests.
Fixes#1510 and #2399.
The gRPC client reconnect code needs to be able to check if a error is temporary so that it can decide if it should attempt to reconnect or just fail and kill the client[1]. By wrapping the error we were receiving in our TLS handshake code we were removing the existing `Temporary` interface on the error. This meant that if a client attempted to reconnect to a server that was in the process of being shutdown, the client would consider that server permanently dead and never retry.
Fix is simple: don't wrap errors that we pass back into the gRPC internals so that they can be properly inspected.
[1]: aefc96d792/clientconn.go (L783)
Both the `20160818140745_AddRegStatus.sql` and
`20160914105917_RemoveChallengesAcctKeyAndTLS.sql` migrations have been
applied in production and can be moved out of `sa/_db-next/` to reflect
this fact.
This commit adds a small script `tag_and_upload.sh` that:
1) Builds the boulder-tools image with the correct tag
2) Prompts you to log in to dockerhub
3) Pushes the boulder-tools image
This means I won't have to remember how to do this next time we need to
bump our Go version :-)
Remove explicit `boulder-tools` docker pull cmd.
Per @jsha's comment in #2567 it should be possible to remove the explicit docker pull letsencrypt/boulder-tools since the docker-compose pull that precedes it will take care of it.
The NotAfter and IsExpired fields on the certificateStatus table
have been migrated in staging & production. Similarly the
CertStatusOptimizationsMigrated feature flag has been turned on after
a successful backfill operation. We have confirmed the optimization is
working as expected and can now clean out the duplicated v1 and v2
models, and the feature flag branching. The notafter-backfill command
is no longer useful and so this commit also cleans it out of the repo.
Note: Some unit tests were sidestepping the SA and inserting
certificateStatus rows explicitly. These tests had to be updated to
set the NotAfter field in order for the queries used by the
ocsp-updater and the expiration-mailer to perform the way the tests
originally expected.
Resolves#2530
Previously we had a 'latest' docker image tag for the `boulder-tools`
image that was *not* the most recent. This was confusing so we deleted
it this morning to close#2030.
Unfortunately Travis was defaulting to pulling the "latest" tag since
one wasn't specified (e.g. the way we do in `Dockerfile` and
`docker-compose.yml`). Resulting in build breakage:
```
docker pull letsencrypt/boulder-tools
Using default tag: latest
Pulling repository docker.io/letsencrypt/boulder-tools
Tag latest not found in repository docker.io/letsencrypt/boulder-tools
The command "docker pull letsencrypt/boulder-tools" failed and exited
with 1
```
This commit specifies the same tag as in `Dockerfile` and
`docker-compose.yml` for travis. We will need to update this tag when we
update the other places for a new boolder-tools image.
Instead of using `unwrapError/wrapError` in each of the wrapper functions do it in the server/client interceptors instead. This means we now consistently do error unwrapping/wrapping.
Fixes#2509.
This allows us to iterate more easily against the current acme module.
Also, remove nodejs from boulder-tools, clean up a few packages that weren't
previously cleaned up, and install a specific version of protoc-gen-go to match
our vendored grpc.
In the last weeks we made some large changes to the list of .RU and .SU domains in the PSL, due to some very old policy changes at the registry (2009) and more recent follow up.
Given the amount of pressure about these changes from certain users, most certainly because LE limits, I figured out you'll soon have people asking you to merge the changes. I've packaged a new release of publicsuffix-go, and updated the dependency in this PR.
$ git show master
commit c5490f26d8f43b84857ac54e23387b8ed9b100dd
Author: Simone Carletti <weppos@weppos.net>
Date: Tue Feb 7 23:26:14 2017 +0100
Release 0.3.2
➜ publicsuffix-go git:(master) go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.023s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.039s
Please note this release also includes the .ONION as per publicsuffix/list#374
Daniel McCarney
Roland Bracewell Shoemaker
Jacob Hoffman-Andrews
David Calavera
Roland Shoemaker
Sophie Herold
Simone Carletti