Remove the redis-tls, wfe-tls, and mail-test-srv keys which were
generated by minica and then checked in to the repo. All three are
replaced by the dynamically-generated ipki directory.
Part of https://github.com/letsencrypt/boulder/issues/7476
Delete the ocsp-updater service, and the //ocsp/updater library that
supports it. Remove test configs for the service, and remove references
to the service from other test files.
This service has been fully shut down for an extended period now, and is
safe to remove.
Fixes#6499
## Remove cfssl recommendation
While it is a valuable PKI toolkit, it really isn't an alternative to
boulder -- there are other private ACME CA projects, and I don't think
we should be in the business of recommending other software when there's
many tradeoffs to be made.
## Remove references to "two API versions"
This removes the reference to running two WFEs, and simplifies some of
the description around "objects" being passed around, which I don't
think is helpful for understanding how Boulder works as the RPCs aren't
generally broadcasting updated objects in the way the removed paragraph
suggests.
## Update information about solving ACME challenges
In #6619 we removed the VA PortConfig, so information about ports 5001
and 5002 are obsolete.
As well, the docker host IP is almost always (barring a user changing
it) the same, so while there's a longer explanation in the README, a
comment in docker-compose.yml is a useful quick reference.
Bring in the release docs from the boulder-release-process repo,
so that they're adjacent to all of our other docs. This allows us to
delete that repo. Also update references to that repo to instead point
to the new doc here.
Also make minor organization updates to other docs to keep the root
of this repository clean.
- Remove GOPATH-style path structure, which isn't needed with Go
modules.
- Remove check for existing of docker buildx builder instance, since it
was unreliable.
These tests are testing functionality that is no longer in use in
production deployments of Boulder. As we go about removing wfe1
functionality, these tests will break, so let's just remove them
wholesale right now. I have verified that all of the tests removed in
this PR are duplicated against wfe2.
One of the changes in this PR is to cease starting up the wfe1 process
in the integration tests at all. However, that component was serving
requests for the AIA Issuer URL, which gets queried by various OCSP and
revocation tests. In order to keep those tests working, this change also
adds an integration-test-only handler to wfe2, and updates the CA
configuration to point at the new handler.
Part of #5681
- Remove `.travis.yml`
- Remove references to Travis in `test.sh`
- Update documentation in `test/boulder-tools/README.md`, `README.MD`,
and `CONTRIBUTING.MD`
- Update comments in `.github/workflows/boulder-ci.yml`
Fixes#5329
Modifies test.sh to provide a long and short option flagged menu
with helpful documentation. Improves UX by printing the settings
used before each run. Improves visual parsability by printing a
bold blue section heading with a title for each test. Using set flags
-eu and an EXIT trap we now exit at any failure or undefined var
and print FAILURE in red on the last line. If the script doe not exit
prematurely, then SUCCESS is printed in green on the last line.
Adds the following options: Parse and print a full list of available
integration tests. Enable the -race flag for unit tests. Enable next
config. Flag to add integration test and unit test filter arguments.
Fixes, -race flag not being enabled by the previous tests. The script
comments seem to indicate that this is desirable, but either through
error or intention TRAVIS was never being set to true, I confirmed
this behavior by running CI tests and observing which options were
set before making this change. If this is not desirable, we can make
a change in .travis.yml to omit the -e flag on unit test and unit test
with next-config.
Fixed dir var not being set. Since the previous script did not use the
set -u flag, undefined vars would not cause an exit. Once this flag was
set it was apprent that var dir was never being defined. Instead, the
script was just using .coverprofile for the coverage test option.
Removes FAILURE and SUCCESS on exit from the Python wrapper
in test/integration-test.py.
Modifies .travis.yml to to use the new short option flags instead of
environment variables. Included comments here to describe all of
the available flags.
Modifies README.md to add examples of the new script in use as
well as examples of running tests without the new script. This should
give folks a good idea of what the script is wrapping.
Part of #5139
This change brings the three top-level markdown documents (README,
CONTRIBUTING, and CODE_OF_CONDUCT) in line with standard markdown
formatting and styling, as informed by
markdownlint/markdownlint:docs/RULES.md@master
Primarily, it:
Hard-wraps long lines to be 80 characters or less, except for
long URLs and code blocks;
Uniformly wraps all code blocks in triple-backticks, and supplies
language information for syntax highlighting; and
Makes a few other small tweaks.
goveralls was not able to detect the current branch since #4735. Instead
of figuring out which variables are needed for it to work correctly,
this commit just propagates all environment variables with the TRAVIS_
prefix. In addition, to match the migration from travis-ci.org to
travis-ci.com, the service name was changed to `travis-pro`, as per the
Coveralls docs: https://docs.coveralls.io/supported-ci-services
Also updates the badges in the readme file. The badge wasn't updated when
we migrated to travis-ci.com. The travis-ci.org address now just has an
annoying redirect page which requires you to click a link to go to the
new page on travis-ci.com. The new badge just takes you there directly.
The `tests/integration/_common.sh` script that provided the
`certbot_test` alias we describe in the README's "Working with Certbot"
section is being replaced by a first class command installed via `pip`
in the virtualenv. This commit updates the README instructions
accordingly.
- docker-rebuild isn't needed now that boulder and bhsm containers run directly off
the boulder-tools image.
- Remove DNS options from RA config.
- Remove GSB options from VA config.
* Remove the challenge whitelist
* Reduce the signature for ChallengesFor and ChallengeTypeEnabled
* Some unit tests in the VA were changed from testing TLS-SNI to testing the same behavior
in TLS-ALPN, when that behavior wasn't already tested. For instance timeouts during connect
are now tested.
Fixes#4109
This is primarily based on @rolandshoemaker's `readme-updates` branch and the closed PR: #3195
The README is restructured to be user-driven. Information that is strictly contributor focused (e.g. dep upgrades) is moved to CONTRIBUTING.md. The review feedback from #3195 from @jsha was all about text that we removed entirely in 29cdd78155
The README now links to the production deployment guide in the wiki and describes our general position on using Boulder.
The CONTRIBUTING.md guide links to the various docs/ pages and was updated.
The docs/acme-divergences page is updated for draft-15.
The DESIGN.md document is moved from the root of the repo into docs/. It has been updated to cover ACMEv2 and precertificates/SCT embedding.
Resolves https://github.com/letsencrypt/boulder/issues/3850
None of the development team use this approach to running a dev env. It
no longer works without modifying `test/startservers.py` and the `test/`
configurations. Given that it has been broken for a month+ and has only
provoked one user issue I think we can be fairly confident that few others
are using this method of setting up a Boulder development environment
and should prioritize our time/docs accordingly.
We're currently stuck on gRPC v1.1 because of a breaking change to certificate validation in gRPC 1.8. Our gRPC balancer uses a static list of multiple hostnames, and expects to validate against those hostnames. However gRPC expects that a service is one hostname, with multiple IP addresses, and validates all those IP addresses against the same hostname. See grpc/grpc-go#2012.
If we follow gRPC's assumptions, we can rip out our custom Balancer and custom TransportCredentials, and will probably have a lower-friction time in general.
This PR is the first step in doing so. In order to satisfy the "multiple IPs, one port" property of gRPC backends in our Docker container infrastructure, we switch to Docker's user-defined networking. This allows us to give the Boulder container multiple IP addresses on different local networks, and gives it different DNS aliases in each network.
In startservers.py, each shard of a service listens on a different DNS alias for that service, and therefore a different IP address. The listening port for each shard of a service is now identical.
This change also updates the gRPC service certificates. Now, each certificate that is used in a gRPC service (as opposed to something that is "only" a client) has three names. For instance, sa1.boulder, sa2.boulder, and sa.boulder (the generic service name). For now, we are validating against the specific hostnames. When we update our gRPC dependency, we will begin validating against the generic service name.
Incidentally, the DNS aliases feature of Docker allows us to get rid of some hackery in entrypoint.sh that inserted entries into /etc/hosts.
Note: Boulder now has a dependency on the DNS aliases feature in Docker. By default, docker-compose run creates a temporary container and doesn't assign any aliases to it. We now need to specify docker-compose run --use-aliases to get the correct behavior. Without --use-aliases, Boulder won't be able to resolve the hostnames it wants to bind to.
This PR updates the `test/boulder-tools/tag_and_upload.sh` script to template a `Dockerfile` for building multiple copies of `boulder-tools`: one per supported Go version. Unfortunately this is required because only Docker 17+ supports an env var in a Dockerfile `FROM`. It's best if we can stay on package manger installed versions of Docker which precludes 17+ 😞.
The `docker-compose.yml` is updated to version "3" to allow specifying a `GO_VERSION` env var in the respective Boulder `image` directives. This requires `docker-compose` version 1.10.0+ which in turn requires Docker engine version 1.13.0+. The README is updated to reflect these new requirements. This Docker engine version is commonly available in package managers (e.g. Ubuntu 16.04). A sufficient `docker-compose` version is not, but this is a simple one binary Go application that is easy to update outside of package managers.
The `.travis.yml` config file is updated to set the `GO_VERSION` in the build matrix, allowing build tasks for different Go versions. Since the `docker-compose.yml` now requires `docker-compose` 1.10.0+ the
`.travis.yml` also gains a new `before_install` for setting up a modern `docker-compose` version.
Lastly tools and images are updated to support both Go 1.10 (our current Go version) and Go 1.10.1 (the new point release). By default Go 1.10 is used, we can switch this once staging/prod are updated.
_*TODO*: One thing I haven't implemented yet is a `sed` expression in `tag_and_upload.sh` that updates both `image` lines in `docker-compose.yml` with an up-to-date tag. Putting this up for review while I work on that last creature comfort._
Resolves https://github.com/letsencrypt/boulder/issues/3551
Replaces https://github.com/letsencrypt/boulder/pull/3620 (GH got stuck from a yaml error)
From... ahem... some frustrating debugging I determined that the Boulder
docker environment fails in strange & mysterious ways if you do not have
sufficient RAM. This commit adds this fact to the README to save future
souls my torment.
This commit also adds a cd to the intial git clone instructions to
ensure the user is in the correct directory to run docker-compose up
from.
After talking to @jsha, this updates Boulder's docker-compose.yml to version 2. I'm currently working on moving some Certbot tests from EC2 to Docker and this allows me to take advantage of networking features like embedded DNS which is used by default in newer versions of Docker Compose.
This shouldn't change any behavior of the file. One notable thing is I had to add network_mode: bridge to the bhsm service. I don't believe this is a change in behavior though since bhsm was included in the links section for boulder
Fixes#976.
This implements a new rate limit, InvalidAuthorizationsPerAccount. If a given account fails authorization for a given hostname too many times within the window, subsequent new-authz attempts for that account and hostname will fail early with a rateLimited error. This mitigates the misconfigured clients that constantly retry authorization even though they always fail (e.g., because the hostname no longer resolves).
For the new rate limit, I added a new SA RPC, CountInvalidAuthorizations. I chose to implement this only in gRPC, not in AMQP-RPC, so checking the rate limit is gated on gRPC. See #2406 for some description of the how and why. I also chose to directly use the gRPC interfaces rather than wrapping them in core.StorageAuthority, as a step towards what we will want to do once we've moved fully to gRPC.
Because authorizations don't have a created time, we need to look at the expires time instead. Invalid authorizations retain the expiration they were given when they were created as pending authorizations, so we use now + pendingAuthorizationLifetime as one side of the window for rate limiting, and look backwards from there. Note that this means you could maliciously bypass this rate limit by stacking up pending authorizations over time, then failing them all at once.
Similarly, since this limit is by (account, hostname) rather than just (hostname), you can bypass it by creating multiple accounts. It would be more natural and robust to limit by hostname, like our certificate limits. However, we currently only have two indexes on the authz table: the primary key, and
(`registrationID`,`identifier`,`status`,`expires`)
Since this limit is intended mainly to combat misconfigured clients, I think this is sufficient for now.
Corresponding PR for website: letsencrypt/website#125
Pulls in logging improvements in OCSP Responder and the CT client, plus a handful of API changes. Also, the CT client verifies responses by default now.
This change includes some Boulder diffs to accommodate the API changes.
By default the Boulder dev env's VA config test/config/va.json has
a portConfig that connects to port 5002 for HTTP-01 challenges and
port 5001 for TLS-SNI-01 challenges.
This can be confusing to someone that follows the FAKE_DNS guidelines
for using a client from the host machine but expects the VA to reach out
on port 80 or 443 like production/staging's VA.
This commit updates the README to include a note on the non-standard
ports and how to change them.
Previously the instructions assumed you had Go setup on your host, which
somewhat defeates the point of running Boulder inside Docker, since it requires
more initial setup. These instructions make first-time users less likely to hit
the oci runtime error described later in the README.
We have seen a couple issues from folks that run into trouble using `docker-compose up` with invalid `$GOPATH`'s configured (e.g. see issues #2150, #2141, #2112).
This PR adds a sentence to the README indicate that if you see a docker "oci runtime error" or a failure to create the container it may be caused by your `$GOPATH` and to check that first.
As a follow-up to #2098 this PR changes the sentence describing "an alias for letsencrypt" to reference an "alias to certbot" instead. This fixes what was an outdated reference to the client formerly known as Let's Encrypt, now Certbot.
The slow start guide's "working with a client" section of the README still referenced a `letsencrypt` path instead of the correct `certbot` path when describing sourcing the integration test alias script. This PR updates the path to use certbot.
Docker's now our main dev env, and running outside of Docker requires a change
to not use SoftHSM. Update the README to reflect that.
Also include instructions in the README on how to run SoftHSM on the host, update make-softhsm to match the current setup, and describe the mounting of GOPATH for instant updates to code.
Aaron Gable