Commit Graph

102 Commits

Author SHA1 Message Date
Aaron Gable 89000bd61c
Add close-primes detection via Fermat's factorization (#5853)
Add a new check to GoodKey which attempts to factor the public modulus
of the presented key using Fermat's factorization method. This method
will succeed if and only if the prime factors are very close to each
other -- i.e. almost certainly were not selected independently from a
random uniform distribution, but were instead calculated via some other
less secure method.

To support this new feature, add a new config flag to the RA, CA, and
WFE, which all use the GoodKey checks. As part of adding this new config
value, refactor the GoodKey config items into their own config struct
which can be re-used across all services.

If the new `FermatRounds` config value has not been set, it will default
to zero, causing no factorization to be attempted.

Fixes #5850
Part of #5851
2021-12-14 09:19:33 -08:00
Aaron Gable 1a1cd24237
Add tests for the experimental `renewalInfo` endpoint (#5750)
Add a unit test and an integration test that both exercise the new
experimental ACME Renewal Info endpoint. These tests do not
yet validate the contents of the response, just that the appropriate
HTTP response code is returned, but they will be developed as the
code under test evolves.

Fixes #5674
2021-10-27 15:00:56 -07:00
Jacob Hoffman-Andrews ba0ea090b2
integration: save hierarchy across runs (#5729)
This allows repeated runs using the same hiearchy, and avoids spurious
errors from ocsp-updater saying "This CA doesn't have an issuer cert
with ID XXX"

Fixes #5721
2021-10-20 17:06:33 -07:00
Aaron Gable 9abb39d4d6
Honeycomb integration proof-of-concept (#5408)
Add Honeycomb tracing to all Boulder components which act as
HTTP servers, gRPC servers, or gRPC clients. Add many values
which we currently emit to logs to the trace spans. Add a way to
configure the Honeycomb integration to our config files, and by
default configure all of our tests to "mute" (send nothing).

Followup changes will refine the configuration, attempt to reduce
the new dependency load, and introduce better sampling.

Part of https://github.com/letsencrypt/dev-misc-tickets/issues/218
2021-05-24 16:13:08 -07:00
Andrew Gabbitas 0fdfbe1211
Deprecate StripDefaultSchemePort flag (#5265)
This flag is now enabled in Let's Encrypt staging/prod.

This change deprecates the flag and prepares it for deletion in a future
change. It can then be removed once no staging/prod configs reference the
flag.

Fixes #5236
2021-02-08 11:30:52 -08:00
Aaron Gable 379826d4b5
WFE2: Improve support for multiple issuers & chains (#5247)
This change simplifies and hardens the wfe2's support for having
multiple issuers, and multiple chains for each issuer, configured
and loaded in memory.

The only config-visible change is replacing the old two separate config
values (`certificateChains` and `alternateCertificateChains`) with a
single value (`chains`). This new value does not require the user to
know and hand-code the AIA URLs at which the certificates are available;
instead the chains are simply presented as lists of files. If this new
config value is present, the old config values will be ignored; if it
is not, the old config values will be respected.

Behind the scenes, the chain loading code has been completely changed.
Instead of loading PEM bytes directly from the file, and then asserting
various things (line endings, no trailing bits, etc) about those bytes,
we now parse a certificate from the file, and in-memory recreate the
PEM from that certificate. This approach allows the file loading to be
much more forgiving, while also being stricter: we now check that each
certificate in the chain is correctly signed by the next cert, and that
the last cert in the chain is a self-signed root.

Within the WFE itself, most of the internal structure has been retained.
However, both the internal `issuerCertificates` (used for checking
that certs we are asked to revoke were in fact issued by us) and the
`certificateChains` (used to append chains to end-entity certs when
served to clients) have been updated to be maps keyed by IssuerNameID.
This allows revocation checking to not have to iterate through the
whole list of issuers, and also makes it easy to double-check that
the signatures on end-entity certs are valid before serving them. Actual
checking of the validity will come in a follow-up change, due to the
invasive nature of the necessary test changes.

Fixes #5164
2021-01-27 15:07:58 -08:00
Aaron Gable 91d4e235ad
Deprecate the BlockedKeyTable feature flag (#4881)
This commit consists of three classes of changes:
1) Changing various command main.go files to always behave as they
   would have when features.BlockedKeyTable was true. Also changing
   one test in the same manner.
2) Removing the BlockedKeyTable flag from configuration in config-next,
   because the flag is already live.
3) Moving the BlockedKeyTable flag to the "deprecated" section of
   features.go, and regenerating featureflag_strings.go.

A future change will remove the BlockedKeyTable flag (and other
similarly deprecated flags) from features.go entirely.

Fixes #4873
2020-06-22 16:35:37 -07:00
Roland Bracewell Shoemaker 7673f02803
Use cmd/ceremony in integration tests (#4832)
This ended up taking a lot more work than I expected. In order to make the implementation more robust a bunch of stuff we previously relied on has been ripped out in order to reduce unnecessary complexity (I think I insisted on a bunch of this in the first place, so glad I can kill it now).

In particular this change:

* Removes bhsm and pkcs11-proxy: softhsm and pkcs11-proxy don't play well together, and any softhsm manipulation would need to happen on bhsm, then require a restart of pkcs11-proxy to pull in the on-disk changes. This makes manipulating softhsm from the boulder container extremely difficult, and because of the need to initialize new on each run (described below) we need direct access to the softhsm2 tools since pkcs11-tool cannot do slot initialization operations over the wire. I originally argued for bhsm as a way to mimic a network attached HSM, mainly so that we could do network level fault testing. In reality we've never actually done this, and the extra complexity is not really realistic for a handful of reasons. It seems better to just rip it out and operate directly on a local softhsm instance (the other option would be to use pkcs11-proxy locally, but this still would require manually restarting the proxy whenever softhsm2-util was used, and wouldn't really offer any realistic benefit).
* Initializes the softhsm slots on each integration test run, rather than when creating the docker image (this is necessary to prevent churn in test/cert-ceremonies/generate.go, which would need to be updated to reflect the new slot IDs each time a new boulder-tools image was created since slot IDs are randomly generated)
* Installs softhsm from source so that we can use a more up to date version (2.5.0 vs. 2.2.0 which is in the debian repo)
* Generates the root and intermediate private keys in softhsm and writes out the root and intermediate public keys to /tmp for use in integration tests (the existing test-{ca,root} certs are kept in test/ because they are used in a whole bunch of unit tests. At some point these should probably be renamed/moved to be more representative of what they are used for, but that is left for a follow-up in order to keep the churn in this PR as related to the ceremony work as possible)
Another follow-up item here is that we should really be zeroing out the database at the start of each integration test run, since certain things like certificates and ocsp responses will be signed by a key/issuer that is no longer is use/doesn't match the current key/issuer.

Fixes #4832.
2020-06-03 15:20:23 -07:00
Jacob Hoffman-Andrews 87fb6028c1
Add log validator to integration tests (#4782)
For now this mainly provides an example config and confirms that
log-validator can start up and shut down cleanly, as well as provide a
stat indicating how many log lines it has handled.

This introduces a syslog config to the boulder-tools image that will write
logs to /var/log/program.log. It also tweaks the various .json config
files so they have non-default syslogLevel, to ensure they actually
write something for log-validator to verify.
2020-04-20 13:33:42 -07:00
Roland Bracewell Shoemaker 9df97cbf06
Add a blocked keys table, and use it (#4773)
Fixes #4712 and fixes #4711.
2020-04-15 13:42:51 -07:00
Jacob Hoffman-Andrews 36c1f1ab2d
Deprecate some feature flags (#4771)
Deprecate some feature flags.

These are all enabled in production.
2020-04-13 15:49:55 -07:00
alexzorin 93cb918ce4
wfe: implement alternate certificate chains (#4714)
Closes #4567.

Enabled in `config-next`.

This PR cross-signs the existing issuers (`test-ca-cross.pem`, `test-ca2-cross.pem`) with a new root (`test-root2.key`, `test-root2.pem` = *c2ckling cryptogr2pher f2ke ROOT*).

The cross-signed issuers are referenced in wfe2's configuration, beside the existing `certificateChains` key:

```json
    "certificateChains": {
      "http://boulder:4430/acme/issuer-cert": [ "test/test-ca2.pem" ],
      "http://127.0.0.1:4000/acme/issuer-cert": [ "test/test-ca2.pem" ]
    },
    "alternateCertificateChains": {
      "http://boulder:4430/acme/issuer-cert": [ "test/test-ca2-cross.pem" ],
      "http://127.0.0.1:4000/acme/issuer-cert": [ "test/test-ca2-cross.pem" ]
    },
```

When this key is populated, the WFE will send links for all alternate certificate chains available for the current end-entity certificate (except for the chain sent in the current response):

    Link: <http://localhost:4001/acme/cert/ff5d3d84e777fc91ae3afb7cbc1d2c7735e0/1>;rel="alternate"

For backwards-compatibility, not specifying a chain is the same as specifying `0`: `/acme/cert/{serial} == /acme/cert/{serial}/0` and `0` always refers to the default certificate chain for that issuer (i.e. the value of `certificateChains[aiaIssuerURL]`).
2020-03-24 12:43:26 -07:00
Daniel McCarney bff9eb0534
CI/Dev: restore allowOrigins wfe/wfe2 config. (#4650)
In 67ec373a96 we removed "unused" WFE and WFE2
config elements. Unfortunately I missed that one of these elements,
`allowOrigins`, **is** used and without this config in
place CORS is broken.

We have unit tests for the CORS headers but we did not have any end-to-end
integration tests that would catch a problem with the WFE/WFE2 missing the
`allowOrigins` config element.

This commit restores the `allowOrigins` config value across the WFE/WFE2
configs and also adds a very small integration test. That test only checks one
CORS header and only for the HTTP ACMEv2 endpoint but I think it's sufficient
for the moment (and definitely better than nothing).

Prior to fixing the config elements the integration test fails as expected:
```
--- FAIL: TestWFECORS (0.00s)
    wfe_test.go:28: "" != "*"
FAIL
FAIL	github.com/letsencrypt/boulder/test/integration	0.014s
FAIL
```
2020-01-17 14:41:20 -05:00
Daniel McCarney 925540d7be
Boulder specific API for GETing "stale" ACME resources. (#4645)
This builds on the work @sh7dm started in #4600. I primarily did some
refactoring, added enforcement of the stale check for authorizations and
challenges, and completed the unit test coverage.

A new Boulder-specific (e.g. not specified by ACME / RFC 8555) API is added for
fetching order, authorization, challenge, and certificate resources by URL
without using POST-as-GET. Since we intend this API to only be used by humans
for debugging and we want to ensure ACME client devs use the standards compliant
method we restrict the GET API to only allowing access to "stale" resources
where the required staleness is defined by the WFE2 "staleTimeout"
configuration value (set to 5m in dev/CI).

Since authorizations don't have a creation date tracked we add
a `authorizationLifetimeDays` and `pendingAuthorizationLifetimeDays`
configuration parameter to the WFE2 that matches the RA's configuration. These
values are subtracted from the authorization expiry to find the creation date to
enforce the staleness check for authz/challenge GETs.

One other note: Resources accessed via the GET API will have Link relation URLs
pointing to the standard ACME API URL. E.g. a GET to a stale challenge will have
a response header with a link "up" relation URL pointing at the POST-as-GET URL
for the associated authorization. I wanted to avoid complicating
`prepAuthorizationForDisplay` and `prepChallengeForDisplay` to be aware of the
GET API and update or exclude the Link relations. This seems like a fine
trade-off since we don't expect machine consumption of the GET API results
(these are for human debugging).

Replaces #4600
Resolves #4577
2020-01-15 09:56:48 -05:00
Daniel McCarney 67ec373a96
CI/Dev: Delete old/unused WFE config elements. (#4641)
The config elements deleted from the four WFE config files are not used
anywhere.
2020-01-10 12:37:22 -05:00
Roland Bracewell Shoemaker 46e0468220 Make authz2 the default storage format (#4476)
This change set makes the authz2 storage format the default format. It removes
most of the functionality related to the previous storage format, except for
the SA fallbacks and old gRPC methods which have been left for a follow-up
change in order to make these changes deployable without introducing
incompatibilities.

Fixes #4454.
2019-10-21 15:29:15 -04:00
Roland Bracewell Shoemaker 31ed590edd
Strip default scheme ports from Host headers (#4448)
Fixes #4447.
2019-09-27 16:14:40 -07:00
Daniel McCarney 1cd9733c24
WFE2: allow revocation of precertificates. (#4433)
When the `features.PrecertificateRevocation` feature flag is enabled the WFE2
will allow revoking certificates for a submitted precertificate. The legacy WFE1
behaviour remains unchanged (as before (pre)certificates issued through the V1
API will be revocable with the V2 API).

Previously the WFE2 vetted the certificate from the revocation request by
looking up a final certificate by the serial number in the requested
certificate, and then doing a byte for byte comparison between the stored and
requested certificate.

Rather than adjust this logic to handle looking up and comparing stored
precertificates against requested precertificates (requiring new RPCs and an
additional round-trip) we choose to instead check the signature on the requested
certificate or precertificate and consider it valid for revocation if the
signature validates with one of the WFE2's known issuers. We trust the integrity
of our own signatures.

An integration test that performs a revocation of a precertificate (in this case
one that never had a final certificate issued due to SCT embedded errors) with
all of the available authentication mechanisms is included.

Resolves https://github.com/letsencrypt/boulder/issues/4414
2019-09-16 16:40:07 -04:00
Daniel McCarney f02e9da38f
Support admin. blocking public keys. (#4419)
We occasionally have reason to block public keys from being used in CSRs
or for JWKs. This work adds support for loading a YAML blocked keys list
to the WFE, the RA and the CA (all the components already using the
`goodekey` package).

The list is loaded in-memory and is intended to be used sparingly and
not for more complicated mass blocking scenarios. This augments the
existing debian weak key checking which is specific to RSA keys and
operates on a truncated hash of the key modulus. In comparison the
admin. blocked keys are identified by the Base64 encoding of a SHA256
hash over the DER encoding of the public key expressed as a PKIX subject
public key. For ECDSA keys in particular we believe a more thorough
solution would have to consider inverted curve points but to start we're
calling this approach "Good Enough".

A utility program (`block-a-key`) is provided that can read a PEM
formatted x509 certificate or a JSON formatted JWK and emit lines to be
added to the blocked keys YAML to block the related public key.

A test blocked keys YAML file is included
(`test/example-blocked-keys.yml`), initially populated with a few of the
keys from the `test/` directory. We may want to do a more through pass
through Boulder's source code and add a block entry for every test
private key.

Resolves https://github.com/letsencrypt/boulder/issues/4404
2019-09-06 16:54:26 -04:00
Roland Bracewell Shoemaker af41bea99a Switch to more efficient multi nonce-service design (#4308)
Basically a complete re-write/re-design of the forwarding concept introduced in
#4297 (sorry for the rapid churn here). Instead of nonce-services blindly
forwarding nonces around to each other in an attempt to find out who issued the
nonce we add an identifying prefix to each nonce generated by a service. The
WFEs then use this prefix to decide which nonce-service to ask to validate the
nonce.

This requires a slightly more complicated configuration at the WFE/2 end, but
overall I think ends up being a way cleaner, more understandable, easy to
reason about implementation. When configuring the WFE you need to provide two
forms of gRPC config:

* one gRPC config for retrieving nonces, this should be a DNS name that
resolves to all available nonce-services (or at least the ones you want to
retrieve nonces from locally, in a two DC setup you might only configure the
nonce-services that are in the same DC as the WFE instance). This allows
getting a nonce from any of the configured services and is load-balanced
transparently at the gRPC layer. 
* a map of nonce prefixes to gRPC configs, this maps each individual
nonce-service to it's prefix and allows the WFE instances to figure out which
nonce-service to ask to validate a nonce it has received (in a two DC setup
you'd want to configure this with all the nonce-services across both DCs so
that you can validate a nonce that was generated by a nonce-service in another
DC).

This balancing is implemented in the integration tests.

Given the current remote nonce code hasn't been deployed anywhere yet this
makes a number of hard breaking changes to both the existing nonce-service
code, and the forwarding code.

Fixes #4303.
2019-06-28 12:58:46 -04:00
Daniel McCarney fea7106927
WFE2: Add feature flag for Mandatory POST-As-GET. (#4251)
In November 2019 we will be removing support for legacy pre RFC-8555
unauthenticated GET requests for accessing ACME resources. A new
`MandatoryPOSTAsGET` feature flag is added to the WFE2 to allow
enforcing this change. Once this feature flag has been activated in Nov
we can remove both it and the WFE2 code supporting GET requests.
2019-06-07 08:36:13 -04:00
Roland Bracewell Shoemaker 4ca01b5de3
Implement standalone nonce service (#4228)
Fixes #3976.
2019-06-05 10:41:19 -07:00
Roland Bracewell Shoemaker 7ff30cf857 Remove account ID in WFE2 if feature enabled (#4160)
This helps encourage ACME client developers to use the RFC 8555
specified `Location` header instead.

Fixes #4136.
2019-04-17 13:38:25 -04:00
Roland Bracewell Shoemaker 51f29b9953
Implement WFE retrieval logic for v2 authorizations (#4085)
This changeset implements the logic required for the WFE to retrieve v2 authorizations and their associated challenges while still maintaining the logic to retrieve old authorizations/challenges. Challenge IDs for v2 authorizations are obfuscated using a pretty simply scheme in order to prevent hard coding of indexes. A `V2` field is added to the `core.Authorization` object and populated using the existing field of the same name from the protobuf for convenience. v2 authorizations and challenges use a `v2` prefix in all their URLs in order to easily differentiate between v1 and v2 URLs (e.g. `/acme/authz/v2/asd` and `/acme/challenge/v2/asd/123`), once v1 authorizations cease to exist this prefix can be safely removed. As v2 authorizations use int IDs this change switches from string IDs to int IDs, this mainly only effects tests.

Integration tests are put off for #4079 as they really need #4077 and #4078 to be properly effective.

Fixes #4041.
2019-02-26 13:14:05 -08:00
Daniel McCarney 413028aa1b
WFE2: Restore HeadNonceStatusOK feat. in conf-next (#4011)
This was accidentally removed in 842739bc but hasn't been deployed to
staging/prod yet.
2019-01-16 14:56:47 -05:00
Roland Bracewell Shoemaker 842739bccd Remove deprecated features that have been purged from prod and staging configs (#4001) 2019-01-15 16:16:35 -08:00
Daniel McCarney 52395a061a WFE2: Remove ACME13KeyRollover feature and legacy code. (#3999)
The draft 13 ACME v2 key change behaviour is now mandatory and we can remove the legacy WFE2 code.
2019-01-15 15:40:01 -08:00
Daniel McCarney b0f407dcf0 RA: Remove deprecated UpdateAuthorization RPC. (#3993)
Staging and prod both deployed the PerformValidationRPC feature flag. All running WFE/WFE2 instances are using the more accurately named PerformValidation RPC and we can strip out the old UpdateAuthorization bits. The feature flag for PerformValidationRPC remains until we clean up the staging/prod configs.

Resolves #3947 and completes the last of #3930
2019-01-07 16:35:27 -08:00
Daniel McCarney e5b8d530b7
wfe2: Return Status 200 for HEAD to new-nonce endpoint. (#3992)
Previously we mistakenly returned status 204 (no content) for all
requests to new-nonce, including HEAD. This status should only be used
for GET requests.

When the `HeadNonceStatusOK` feature flag is enabled we will now return
the correct status for HEAD requests. When the flag is disabled we return
status 204 to preserve backwards compatibility.
2019-01-07 12:58:30 -05:00
Daniel McCarney 8f5de538c1
RA: Add PerformValidation RPC to replace UpdateAuthorization. (#3942)
The existing RA `UpdateAuthorization` RPC needs replacing for
two reasons:

1. The name isn't accurate - `PerformValidation` better captures
the purpose of the RPC.
2. The `core.Challenge` argument is superfluous since Key 
Authorizations are not sent in the initiation POST from the client 
anymore. The corresponding unmarshal and verification is now 
removed. Notably this means broken clients that were POSTing
the wrong thing and failing pre-validation will now likely fail 
post-validation.

To remove `UpdateAuthorization` the new `PerformValidation` 
RPC is added alongside the old one. WFE and WFE2 are 
updated to use the new RPC when the perform validation
feature flag is enabled. We can remove 
`UpdateAuthorization` and its associated wrappers once all 
WFE instances have been updated.

Resolves https://github.com/letsencrypt/boulder/issues/3930
2018-11-28 10:12:47 -05:00
Jacob Hoffman-Andrews 4670be1210 Reduce log level for WFE in tests. (#3918)
Our Travis output is quite verbose with the WFE output, and it's very
rare that we have to reference it. I'd like to remove the INFO-level
logs (i.e. the logs of every request) so that it's easier to see real
errors, and faster to scroll to the bottom of logs of failed runs.
2018-11-01 09:50:41 -04:00
Roland Bracewell Shoemaker 876c727b6f Update gRPC (#3817)
Fixes #3474.
2018-08-20 10:55:42 -04:00
Daniel McCarney 0cb28c9e02
WFE2: Implement draft-13 keyrollover with feature flag. (#3813)
ACME draft-13 changed the inner JWS' body to contain the old key
that signed the outer JWS. Because this is a backwards incompatible
change with the draft-12 ACME v2 key rollover we introduce a new feature
flag `features.ACME13KeyRollover` and conditionally use the old or new
key rollover semantics based on its value.
2018-08-07 15:27:25 -04:00
Jacob Hoffman-Andrews dbcb16543e Start using multiple-IP hostnames for load balancing (#3687)
We'd like to start using the DNS load balancer in the latest version of gRPC. That means putting all IPs for a service under a single hostname (or using a SRV record, but we're not taking that path). This change adds an sd-test-srv to act as our service discovery DNS service. It returns both Boulder IP addresses for any A lookup ending in ".boulder". This change also sets up the Docker DNS for our boulder container to defer to sd-test-srv when it doesn't know an answer.

sd-test-srv doesn't know how to resolve public Internet names like `github.com`. Resolving public names is required for the `godep-restore` test phase, so this change breaks out a copy of the boulder container that is used only for `godep-restore`.

This change implements a shim of a DNS resolver for gRPC, so that we can switch to DNS-based load balancing with the currently vendored gRPC, then when we upgrade to the latest gRPC we won't need a simultaneous config update.

Also, this change introduces a check at the end of the integration test that each backend received at least one RPC, ensuring that we are not sending all load to a single backend.
2018-05-23 09:47:14 -04:00
Daniel McCarney 5597a77ba2
WFE2: Allow legacy Key ID prefix for ACME v2 JWS. (#3705)
While we intended to allow legacy ACME v1 accounts created through the WFE to work with the ACME v2 implementation and the WFE2 we neglected to consider that a legacy account would have a Key ID URL that doesn't match the expected for a V2 account. This caused `wfe2/verify.go`'s `lookupJWK` to reject all POST requests authenticated by a legacy account unless the ACME client took the extra manual step of "fixing" the URL.

This PR adds a configuration parameter to the WFE2 for an allowed legacy key ID prefix. The WFE2 verification logic is updated to allow both the expected key ID prefix and the configured legacy key ID prefix. This will allow us to specify the correct legacy URL in configuration for both staging/prod to allow unmodified V1 ACME accounts to be used with ACME v2.

Resolves https://github.com/letsencrypt/boulder/issues/3674
2018-05-11 15:57:56 -04:00
Jacob Hoffman-Andrews a4421ae75b Run gRPC backends on multiple IPs instead of multiple ports (#3679)
We're currently stuck on gRPC v1.1 because of a breaking change to certificate validation in gRPC 1.8. Our gRPC balancer uses a static list of multiple hostnames, and expects to validate against those hostnames. However gRPC expects that a service is one hostname, with multiple IP addresses, and validates all those IP addresses against the same hostname. See grpc/grpc-go#2012.

If we follow gRPC's assumptions, we can rip out our custom Balancer and custom TransportCredentials, and will probably have a lower-friction time in general.

This PR is the first step in doing so. In order to satisfy the "multiple IPs, one port" property of gRPC backends in our Docker container infrastructure, we switch to Docker's user-defined networking. This allows us to give the Boulder container multiple IP addresses on different local networks, and gives it different DNS aliases in each network.

In startservers.py, each shard of a service listens on a different DNS alias for that service, and therefore a different IP address. The listening port for each shard of a service is now identical.

This change also updates the gRPC service certificates. Now, each certificate that is used in a gRPC service (as opposed to something that is "only" a client) has three names. For instance, sa1.boulder, sa2.boulder, and sa.boulder (the generic service name). For now, we are validating against the specific hostnames. When we update our gRPC dependency, we will begin validating against the generic service name.

Incidentally, the DNS aliases feature of Docker allows us to get rid of some hackery in entrypoint.sh that inserted entries into /etc/hosts.

Note: Boulder now has a dependency on the DNS aliases feature in Docker. By default, docker-compose run creates a temporary container and doesn't assign any aliases to it. We now need to specify docker-compose run --use-aliases to get the correct behavior. Without --use-aliases, Boulder won't be able to resolve the hostnames it wants to bind to.
2018-05-07 10:38:31 -07:00
Daniel McCarney 054f181458 load-generator: send correct ACMEv2 Content-Type on POST (#3667)
load generator: send correct ACMEv2 Content-Type on POST.

This PR updates the Boulder load-generator to send the correct ACMEv2 Content-Type header when POSTing the ACME server. This is required for ACMEv2 and without it all POST requests to the WFE2 running a test/config-next configuration result in malformed 400 errors. While only required by ACMEv2 this commit sends it for ACMEv1 requests as well. No harm no foul.

integration tests: allow running just the load generator.
Prior to this PR an omission in an if statement in integration-test.py meant that you couldn't invoke test/integration-test.py with just the --load argument to only run the load generator. This commit updates the if to allow this use case.
2018-05-01 12:22:43 -07:00
Roland Bracewell Shoemaker 0a86573a73 Update integration tests 2018-04-20 13:18:40 -07:00
Daniel ac6672bc71
Revert "Revert "V2: implement "ready" status for Order objects (#3614)" (#3643)"
This reverts commit 3ecf841a3a.
2018-04-12 13:20:47 -04:00
Jacob Hoffman-Andrews 3ecf841a3a Revert "V2: implement "ready" status for Order objects (#3614)" (#3643)
This reverts commit 1d22f47fa2.

According to
https://github.com/letsencrypt/boulder/pull/3614#issuecomment-380615172,
this broke Certbot's tests. We'll investigate, and then roll forward
once we understand what broke.
2018-04-12 10:46:57 -04:00
Daniel McCarney 1d22f47fa2 V2: implement "ready" status for Order objects (#3614)
* SA: Add Order "Ready" status, feature flag.

This commit adds the new "Ready" status to `core/objects.go` and updates
`sa.statusForOrder` to use it conditionally for orders with all valid
authorizations that haven't been finalized yet. This state is used
conditionally based on the `features.OrderReadyStatus` feature flag
since it will likely break some existing clients that expect status
"Processing" for this state. The SA unit test for `statusForOrder` is
updated with a "ready" status test case.

* RA: Enforce order ready status conditionally.

This commit updates the RA to conditionally expect orders that are being
finalized to be in the "ready" status instead of "pending". This is
conditionally enforced based on the `OrderReadyStatus` feature flag.
Along the way the SA was changed to calculate the order status for the
order returned in `sa.NewOrder` dynamically now that it could be
something other than "pending".

* WFE2: Conditionally enforce order ready status for finalization.

Similar to the RA the WFE2 should conditionally enforce that an order's
status is either "ready" or "pending" based on the "OrderReadyStatus"
feature flag.

* Integration: Fix `test_order_finalize_early`.

This commit updates the V2 `test_order_finalize_early` test for the
"ready" status. A nice side-effect of the ready state change is that we
no longer invalidate an order when it is finalized too soon because we
can reject the finalization in the WFE. Subsequently the
`test_order_finalize_early` testcase is also smaller.

* Integration: Test classic behaviour w/o feature flag.

In the previous commit I fixed the integration test for the
`config/test-next` run that has the `OrderReadyStatus` feature flag set
but broke it for the `config/test` run without the feature flag.

This commit updates the `test_order_finalize_early` test to work
correctly based on the feature flag status in both cases.
2018-04-11 10:31:25 -07:00
Jacob Hoffman-Andrews a4f9de9e35 Improve nesting of RPC deadlines (#3619)
gRPC passes deadline information through the RPC boundary, but client and server have the same deadline. Ideally we'd like the server to have a slightly tighter deadline than the client, so if one of the server's onward RPCs or other network calls times out, the server can pass back more detailed information to the client, rather than the client timing out the server and losing the opportunity to log more detailed information about which component caused the timeout.

In this change, I subtract 100ms from the deadline on the server side of our interceptors, using our existing serverInterceptor. I also check that there is at least 100ms remaining in which to do useful work, so the server doesn't begin a potentially expensive task only to abort it.

Fixes #3608.
2018-04-06 15:40:18 +01:00
Daniel McCarney 17922a6d2d
Add CAAIdentities and Website to /directory "meta". (#3588)
This commit updates the WFE and WFE2 to have configuration support for
setting a value for the `/directory` object's "meta" field's
optional "caaIdentities" and "website" fields. The config-next wfe/wfe2
configuration are updated with values for these fields. Unit tests are
updated to check that they are sent when expected and not otherwise.

Bonus content: The `test.AssertUnmarshaledEquals` function had a bug
where it would consider two inputs equal when the # of keys differed.
This commit also fixes that bug.
2018-03-22 16:12:43 -04:00
Daniel McCarney 5172f03865
Enable WFE2 "EnforceV2ContentType" flag in config-next. (#3562)
We shipped this feature flag disabled because at the time Certbot's acme
module had a bug with revocation that sent the wrong content-type. That
has been fixed upstream and the current boulder-tools image has the fix.
We should be able to turn this flag on now for config-next without
breaking tests.
2018-03-14 16:09:58 -04:00
Jacob Hoffman-Andrews c556a1a20d
Reduce spurious errors in integration test (#3436)
Boulder is fairly noisy about gRPC connection errors. This is a mixed
blessing: Our gRPC configuration will try to reconnect until it hits
an RPC deadline, and most likely eventually succeed. In that case,
we don't consider those to really be errors. However, in cases where
a connection is repeatedly failing, we'd like to see errors in the
logs about connection failure, rather than "deadline exceeded." So
we want to keep logging of gRPC errors.

However, right now we get a lot of these errors logged during
integration tests. They make the output hard to read, and may disguise
more serious errors. So we'd like to avoid causing such errors in
normal integration test operation.

This change reorders the startup of Boulder components by their gRPC
dependencies, so everything's backend is likely to be up and running
before it starts. It also reverses that order for clean shutdowns,
and waits for each process to exit before signalling the next one.

With these changes, I still got connection errors. Taking listenbuddy
out of the gRPC path fixed them. I believe the issue is that
listenbuddy is not a truly transparent proxy. In particular, it
accepts an inbound TCP connection before opening an outbound TCP
connection. If opening that outbound connection results in "connection
refused," it closes the inbound connection. That means gRPC sees a
"connection closed" (or "connection reset"?) rather than "connection
refused". I'm guessing it handles those cases differently, explaining
the different error results.

We've been using listenbuddy to trigger disconnects while Boulder is
running, to ensure that gRPC's reconnect code works. I think we can
probably rely on gRPC's reconnect to work. The initial problem that
led us to start testing this was a configuration problem; now that
we have the configuration we want, we should be fine and don't need
to keep testing reconnects on every integration test run.
2018-02-12 18:17:50 -08:00
Daniel McCarney d6a33d1108 Return full cert chain for V2 cert GET. (#3366)
This commit implements a mapping from certificate AIA Issuer URL to PEM
encoded certificate chain. GET's to the V2 Certificate endpoint will
return a full PEM encoded certificate chain in addition to the leaf cert
using the AIA issuer URL of the leaf cert and the configured mapping.

The boulder-wfe2 command builds the chain mapping by reading the
"wfe" config section's 'certificateChains" field, specifying a list
of file paths to PEM certificates for each AIA issuer URL. At startup
the PEM file contents are ready, verified and separated by a newline.
The resulting populated AIA issuer URL -> PEM cert chain mapping is
given to the WFE for use with the Certificate endpoint.

Resolves #3291
2018-01-19 11:23:44 -08:00
Daniel McCarney ba264a5091 Remove unused WFE2 feature flags. (#3375)
The WFE2 doesn't check any of the feature flags that are configured in
the `test/config/wfe2.json` and `test/config-next/wfe2.json` config
files - we default to acting as if all new features are enabled for the
V2 work. This commit removes the flags from the config to avoid
confusion or expectations that changing the config will disable the
features.
2018-01-17 12:28:19 -08:00
Jacob Hoffman-Andrews 827f7859f2 Fix issuerCert in test configs. (#3310)
Previously, there was a disagreement between WFE and CA as to what the correct
issuer certificate was. Consolidate on test-ca2.pem (h2ppy h2cker fake CA).
    
Also, the CA configs contained an outdated entry for "IssuerCert", which was not
being used: The CA configs now use an "Issuers" array to allow signing by
multiple issuer certificates at once (for instance when rolling intermediates).
Removed this outdated entry, and the config code for CA to load it. I've
confirmed these changes match what is currently in production.

Added an integration test to check for this problem in the future.

Fixes #3309, thanks to @icing for bringing the issue to our attention!

This also includes changes from #3321 to clarify certificates for WFE.
2018-01-09 07:56:39 -05:00
Jacob Hoffman-Andrews fdd854a7e5 Fix various WFE2 bugs. (#3292)
- Encode certificate as PEM.
- Use lowercase for field names.
- Use termsOfServiceAgreed instead of Agreement
- Use a different ToS URL for v2 that points at the v2 HTTPS port.

Resolves #3280
2017-12-19 13:13:29 -08:00
Jacob Hoffman-Andrews 071fc0120f Remove facebookgo/httpdown. (#3168)
Its purpose is now served by net/http's Shutdown().
2017-10-17 08:55:43 -04:00
Jacob Hoffman-Andrews 8afec60433 Remove unneeded dns config value for WFE. (#3057) 2017-09-08 14:32:36 -04:00
Daniel McCarney bd3e2747ba Duplicate WFE to WFE2. (#2839)
This PR is the initial duplication of the WFE to create a WFE2
package. The rationale is briefly explained in `wfe2/README.md`.

Per #2822 this PR only lays the groundwork for further customization
and deduplication. Presently both the WFE and WFE2 are identical except
for the following configuration differences:

* The WFE offers HTTP and HTTPS on 4000 and 4430 respectively, the WFE2
  offers HTTP on 4001 and 4431.
* The WFE has a debug port on 8000, the WFE2 uses the next free "8000
  range port" and puts its debug service on 8013

Resolves https://github.com/letsencrypt/boulder/issues/2822
2017-07-05 13:32:45 -07:00