This code was never enabled in production. Our original intent was to
ship this as part of the ACMEv2 API. Before that could happen flaws were
identified in TLS-SNI-01|02 that resulted in TLS-SNI-02 being removed
from the ACME protocol. We won't ever be enabling this code and so we
might as well remove it.
There's a minor data race in UpdateAuthorization where the authorization can
be changed in the goroutine concurrently with returning it. This fixes it by
copying the authorization as an argument to the goroutine.
Fixes#3506.
Previously we introduced the concept of a "pending orders per account
ID" rate limit. After struggling with making an implementation of this
rate limit perform well we reevaluated the problem and decided a "new
orders per account per time window" rate limit would be a better fit for
ACMEv2 overall.
This commit introduces the new newOrdersPerAccount rate limit. The RA
now checks this before creating new pending orders in ra.NewOrder. It
does so after order reuse takes place ensuring the rate limit is only
applied in cases when a distinct new pending order row would be created.
To accomplish this a migration for a new orders field (created) and an
index over created and registrationID is added. It would be possible to
use the existing expires field for this like we've done in the past, but that
was primarily to avoid running a migration on a large table in prod. Since
we don't have that problem yet for V2 tables we can Do The Right Thing
and add a column.
For deployability the deprecated pendingOrdersPerAccount code & SA
gRPC bits are left around. A follow-up PR will be needed to remove
those (#3502).
Resolves#3410
Prior to this commit the comment on `pa.WillingToIssue` incorrectly
asserted that the input domain must not contain IDN labels. That's not
true anymore! 🎉
This commit updates `sa.getAllOrderAuthorizations` to not exclude
expired authorizations. The expired authorizations are used in
`sa.statusForOrder` to set the overall order status to invalid when one
or more authorizations are expired.
Fixes#3499
When we originally deployed the inline CT submission code, we wanted to
be conservative in case it increased our issuance error rate. However,
we've established that the success rate is quite good, so we'll remove
some complexity and make things more realistic by removing the code that
avoids returning errors when CT submission fails.
This change updates boulder-tools to use Go 1.10, and references a
newly-pushed image built using that new config.
Since boulder-tools pulls in the latest Certbot master at the time of
build, this also pulls in the latest changes to Certbot's acme module,
which now supports ACME v2. This means we no longer have to check out
the special acme-v2-integration branch in our integration tests.
This also updates chisel2.py to reflect some of the API changes that
landed in the acme module as it was merged to master.
Since we don't need additional checkouts to get the ACMEv2-compatible
version of the acme module, we can include it in the default RUN set for
local tests.
This puts puts the path for order finalization at the beginning of the URL, rather than the end, making it easier to generate stats about. It also allows us to split out GetOrder and FinalizeOrder from the previous "Order" function that combined some unrelated code.
Fixes#3440Fixes#3439 (order finalization now has its own stats like the other endpoints do)
We have a Dockerfile that builds off of boulder-tools, but does
basically nothing. Removing it saves a build step and avoids some
hard-to-diagnose edge cases.
This change triggered an issue with the "prom" container and
"docker-compose up", saying that prom can't be linked to a not-running
container. Since we rarely use prom in dev/test, I'm removing it for
now. The nice, futuristic way to get containers to talk to each other is
with docker networking, so we can bring it back with a switch to docker
networking in the future if we want.
This commit adds new ACMEv2 actions for the load-generator and an
example configuration for load testing ACME v2 against a local boulder
instance.
The load-generator's existing V1 code remains but the two ACME versions
can not be intermixed in the same load generator plan. The load
generator should be making 100% ACME v1 action calls or 100% ACME v2
action calls.
Follow-up work:
* Adding a short load generator run for V1 and V2 to the integration tests/CI
* Randomizing the # of names in pending orders - right now they are always 1 identifier orders.
* Making it easier to save state on an initial run without needing to create an empty JSON file
* DNS-01 support & Wildcard names
I'm going to start chipping at the "Follow-up" items but wanted to get the meat of this PR up for review now.
Resolves#3137
This removes some fields from "Extra" that are logged on every poll event and
aren't necessarily. For instance, authorizationID and challengeID can easily be
derived from the endpoint, and AuthorizationRequesterID is a duplicate of
Requester.
Use `t.Helper` and `t.Fatalf` instead of our own versions.
Remove some unused or single-user helpers.
Make the output of `AssetUnmarshaledEquals` clearer by showing one line per field.
Our various main.go functions gated some key code on whether the TLS
and/or GRPC config fields were present. Now that those fields are fully
deployed in production, we can simplify the code and require them.
Also, rename tls to tlsConfig everywhere to avoid confusion with the tls
package.
Avoid assigning to the same err from two different goroutines in
boulder-ca (fix a race).
Add a set of logs which will be submitted to but not relied on for their SCTs,
this allows us to test submissions to a particular log or submit to a log which
is not yet approved by a browser/root program.
Also add a feature which stops cancellations of remaining submissions when racing
to get a SCT from a group of logs.
Additionally add an informational log that always times out in config-next.
Fixes#3464 and fixes#3465.
* Re-vendor certificate-transparency-go to latest.
$ go test github.com/google/certificate-transparency-go{,/asn1,/client,/client/configpb,/jsonclient,/tls,/x509/pkix} golang.org/x/crypto/cryptobyte{,/asn1}
ok github.com/google/certificate-transparency-go 0.722s
ok github.com/google/certificate-transparency-go/asn1 0.011s
ok github.com/google/certificate-transparency-go/client 22.995s
? github.com/google/certificate-transparency-go/client/configpb [no test files]
ok github.com/google/certificate-transparency-go/jsonclient 0.020s
ok github.com/google/certificate-transparency-go/tls 0.096s
? github.com/google/certificate-transparency-go/x509/pkix [no test files]
ok golang.org/x/crypto/cryptobyte 0.013s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
* Bring in latest ct-go master.
The test for the certificates_per_name rate limit uses an exact domain
name that has an override in the rate limit config file to have a limit
of 0. This works correctly most of the time. However, if that mechanism
fails once (due to some bug), future runs of the integation test will
continue to fail, because there will now be an issued certificate for
"lim.it" in the DB, and subsequent attempts will be considered renewals.
This change adds a random subdomain to the test, so that it's not
eligible for the renewal exemption.
This commit adds support for the Akamai CCU v3 API. See
https://developer.akamai.com/api/purge/ccu/resources.html for more information.
The V2 and V3 API are close enough to one another that we can support
both with minimal changes. A new OCSP updated configuration parameter
"AkamaiV3Network" is used to determine if the cache client should use
the V2 API or the V3 API. When empty, V2 is used. When set to either
"production" or "staging", the V3 API is used.
Fixes#3437.
This commit adds support for the Akamai CCU v3 API. See
https://developer.akamai.com/api/purge/ccu/resources.html for more information.
The V2 and V3 API are close enough to one another that we can support
both with minimal changes. A new OCSP updated configuration parameter
"AkamaiV3Network" is used to determine if the cache client should use
the V2 API or the V3 API. When empty, V2 is used. When set to either
"production" or "staging", the V3 API is used.
We occasionally get deadline exceeded errors inside RA's NewCertificate
method, however it's not clear which SA queries cause these: we just see
"deadline exceeded" errors. This prefixes errors from the most relevant
queries with details about the request being made.
In #3454, I tried to update certificate-transparency-go, but that pulled in a bunch of extra package updates, making for a complicated PR. This PR breaks out one of the packages that needed update, to allow us to bring things up to date in a simpler, more piecemeal fashion.
$ go test golang.org/x/crypto/...
ok golang.org/x/crypto/acme 2.564s
ok golang.org/x/crypto/acme/autocert 0.634s
ok golang.org/x/crypto/argon2 0.118s
ok golang.org/x/crypto/bcrypt 2.282s
ok golang.org/x/crypto/blake2b 0.103s
ok golang.org/x/crypto/blake2s 0.072s
ok golang.org/x/crypto/blowfish 0.006s
ok golang.org/x/crypto/bn256 0.462s
2ok golang.org/x/crypto/cast5 4.288s
ok golang.org/x/crypto/chacha20poly1305 0.037s
ok golang.org/x/crypto/cryptobyte 0.012s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
ok golang.org/x/crypto/curve25519 0.029s
ok golang.org/x/crypto/ed25519 0.082s
? golang.org/x/crypto/ed25519/internal/edwards25519 [no test files]
ok golang.org/x/crypto/hkdf 0.003s
ok golang.org/x/crypto/internal/chacha20 0.002s
ok golang.org/x/crypto/md4 0.002s
ok golang.org/x/crypto/nacl/auth 1.473s
ok golang.org/x/crypto/nacl/box 0.007s
ok golang.org/x/crypto/nacl/secretbox 0.004s
ok golang.org/x/crypto/ocsp 0.034s
ok golang.org/x/crypto/openpgp 7.275s
ok golang.org/x/crypto/openpgp/armor 0.015s
ok golang.org/x/crypto/openpgp/clearsign 0.028s
ok golang.org/x/crypto/openpgp/elgamal 0.015s
? golang.org/x/crypto/openpgp/errors [no test files]
ok golang.org/x/crypto/openpgp/packet 0.170s
ok golang.org/x/crypto/openpgp/s2k 9.401s
ok golang.org/x/crypto/otr 0.321s
ok golang.org/x/crypto/pbkdf2 0.046s
ok golang.org/x/crypto/pkcs12 0.065s
ok golang.org/x/crypto/pkcs12/internal/rc2 0.014s
ok golang.org/x/crypto/poly1305 0.023s
ok golang.org/x/crypto/ripemd160 0.061s
ok golang.org/x/crypto/salsa20 0.029s
ok golang.org/x/crypto/salsa20/salsa 0.043s
ok golang.org/x/crypto/scrypt 0.815s
ok golang.org/x/crypto/sha3 0.263s
ok golang.org/x/crypto/ssh 1.175s
ok golang.org/x/crypto/ssh/agent 0.827s
ok golang.org/x/crypto/ssh/knownhosts 0.038s
ok golang.org/x/crypto/ssh/terminal 0.029s
ok golang.org/x/crypto/ssh/test 0.148s
ok golang.org/x/crypto/tea 0.012s
ok golang.org/x/crypto/twofish 0.013s
ok golang.org/x/crypto/xtea 0.002s
ok golang.org/x/crypto/xts 0.016s
In #3454, I tried to update certificate-transparency-go, but that pulled in a bunch of extra package updates, making for a complicated PR. This PR breaks out one of the packages that needed update, to allow us to bring things up to date in a simpler, more piecemeal fashion.
$ go test github.com/golang/protobuf/...
ok github.com/golang/protobuf/descriptor 0.004s
ok github.com/golang/protobuf/jsonpb 0.012s
? github.com/golang/protobuf/jsonpb/jsonpb_test_proto [no test
files]
ok github.com/golang/protobuf/proto 0.062s
? github.com/golang/protobuf/proto/proto3_proto [no test files]
? github.com/golang/protobuf/protoc-gen-go [no test files]
? github.com/golang/protobuf/protoc-gen-go/descriptor [no test
files]
ok github.com/golang/protobuf/protoc-gen-go/generator 0.002s
? github.com/golang/protobuf/protoc-gen-go/grpc [no test files]
? github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok github.com/golang/protobuf/ptypes 0.014s
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
This was added in the theory that it would stop some errors running
`docker compose go generate ./...` but in tests today, that command
succeeds right now even after removing this from the build steps and
rebuilding. This is not surprising, because as Roland pointed out, the
/alt-gopath/ path was never actually referenced anywhere.
With the CT policy changes, we cancel any outstanding requests in a
group once we've gotten a successful response from any log in that
group. That means various function calls will return early with an error
code indicating cancellation. We want to avoid logging such error codes,
because they were not really errors, they were intentional.
This change introduces a small utility package called "canceled", which
checks for both context.Canceled and the gRPC return code indicating
cancelled.
We recently reordered the programs in startservers.py to reflect
dependencies, so we wouldn't generate gRPC errors when a process comes
up before its backend. However, we missed the remote VAs. This change
moves them to the beginning.
Also, don't print "all processes started" after each process starts.
chisel had verify_ssl=False. Remove that, and set a sensible default
for REQUESTS_CA_BUNDLE to make it easier to run chisel on the command
line. Port the REQUESTS_CA_BUNDLE change into chisel2 as well.
Removes usage of the `EnforceChallengeDisable` feature, the feature itself is not removed as it is still configured in staging/production, once that is fixed I'll submit another PR removing the actual flag.
This keeps the behavior that when authorizations are retrieved from the SA they have their challenges populated, because that seems to make the most sense to me? It also retains TLS re-validation.
Fixes#3441.
This commit updates the github.com/miekg/pkcs11 dependency to
88ac7c418f89b164432a00c46ec7b7612d686b57, the tip of master at the time
of writing.
This incorporates a fix for Golang 1.9.4.
Confirmed upstream unit tests pass:
$> git rev-parse HEAD
88ac7c418f89b164432a00c46ec7b7612d686b57
$> go test ./...
ok github.com/miekg/pkcs11 0.676s
Resolves#3442
Boulder is fairly noisy about gRPC connection errors. This is a mixed
blessing: Our gRPC configuration will try to reconnect until it hits
an RPC deadline, and most likely eventually succeed. In that case,
we don't consider those to really be errors. However, in cases where
a connection is repeatedly failing, we'd like to see errors in the
logs about connection failure, rather than "deadline exceeded." So
we want to keep logging of gRPC errors.
However, right now we get a lot of these errors logged during
integration tests. They make the output hard to read, and may disguise
more serious errors. So we'd like to avoid causing such errors in
normal integration test operation.
This change reorders the startup of Boulder components by their gRPC
dependencies, so everything's backend is likely to be up and running
before it starts. It also reverses that order for clean shutdowns,
and waits for each process to exit before signalling the next one.
With these changes, I still got connection errors. Taking listenbuddy
out of the gRPC path fixed them. I believe the issue is that
listenbuddy is not a truly transparent proxy. In particular, it
accepts an inbound TCP connection before opening an outbound TCP
connection. If opening that outbound connection results in "connection
refused," it closes the inbound connection. That means gRPC sees a
"connection closed" (or "connection reset"?) rather than "connection
refused". I'm guessing it handles those cases differently, explaining
the different error results.
We've been using listenbuddy to trigger disconnects while Boulder is
running, to ensure that gRPC's reconnect code works. I think we can
probably rely on gRPC's reconnect to work. The initial problem that
led us to start testing this was a configuration problem; now that
we have the configuration we want, we should be fine and don't need
to keep testing reconnects on every integration test run.
For the upcoming SCT embedding changes, it will be useful to have a CT test server that blocks for nontrivial amounts of time before responding. This change introduces a config file for `ct-test-srv` that can be used to set up multiple "personalities" on various ports. Each personality can have a "latencySchedule" that determines how long it will sleep before servicing responding to a submission.
This change also introduces two new "personalities" on :4510 and :4511, plus configures CTLogGroups in the RA. Having four CT log personalities allows us to simulate two nontrivial log groups.
Note: This triggers Publisher to emit audit errors on timed-out submissions. We may want to make Publisher not treat those as errors, and instead only log an error if a whole log group fails.
Prior to this commit a logical error in the RA's `NewOrder` caused
a safety check that prevents authorization reuse with a non-wildcard
authz for a wildcard name to not work. This commit adds a test for the
condition that the safety check is designed for and fixes the logical
error. Prior to fixing the logical error the test fails. With the
corrected safety check the test passes.
Prior to this commit when building up the authorizations for a new-order
request we looked for any unexpired pending/valid authorizations owned
by the account and used them for the order. This allows a client to use
the V1 new-authz endpoint in combination with the V2 new-order endpoint
and we do not want to support this behaviour. All V2 authorizations
should be sourced from other V2 orders. This commit implements a new
parameter for the SA's getAuthorizations function that allows filtering
out legacy V1 authorizations by doing a JOIN on the order to
authorizations join table.
Resolves#3328
This commit resolves the case where an error during finalization occurs.
Prior to this commit if an error (expected or otherwise) occurred after
setting an order to status processing at the start of order
finalization the order would be stuck processing forever.
The SA now has a `SetOrderError` RPC that can be used by the RA to
persist an error onto an order. The order status calculation can use
this error to decide if the order is invalid. The WFE is updated to
write the error to the order JSON when displaying the order information.
Prior to this commit the order protobuf had the error field as
a `[]byte`. It doesn't seem like this is the right decision, we have
a specific protobuf type for ProblemDetails and so this commit switches
the error field to use it. The conversion to/from `[]byte` is done with
the model by the SA.
An integration test is included that prior to this commit left an order
in a stuck processing state. With this commit the integration test
passes as expected.
Resolves https://github.com/letsencrypt/boulder/issues/3403
The SA RPC previously called `GetOrderAuthorizations` only returns
**valid, unexpired** authorizations. This commit updates the name to
emphasize that it only returns valid order authzs.
Daniel McCarney