Python 2 is over in 1 month 4 days: https://pythonclock.org/
This rolls forward most of the changes in #4313.
The original change was rolled back in #4323 because it
broke `docker-compose up`. This change fixes those original issues by
(a) making sure `requests` is installed and (b) sourcing a virtualenv
containing the `requests` module before running start.py.
Other notable changes in this:
- Certbot has changed the developer instructions to install specific packages
rather than rely on `letsencrypt-auto --os-packages-only`, so we follow suit.
- Python3 now has a `bytes` type that is used in some places that used to
provide `str`, and all `str` are now Unicode. That means going from `bytes` to
`str` and back requires explicit `.decode()` and `.encode()`.
- Moved from urllib2 to requests in many places.
A unit test is included to verify that a TLS-ALPN-01 challenge to
a TLS 1.3 only server doesn't succeed when the `GODEBUG` value to
disable TLS 1.3 in `docker-compose.yml` is set. Without this env var
the test fails on the Go 1.13 build because of the new default:
```
=== RUN TestTLSALPN01TLS13
--- FAIL: TestTLSALPN01TLS13 (0.04s)
tlsalpn_test.go:531: expected problem validating TLS-ALPN-01 challenge against a TLS 1.3 only server, got nil
FAIL
FAIL github.com/letsencrypt/boulder/va 0.065s
```
With the env var set the test passes, getting the expected connection
problem reporting a tls error:
```
=== RUN TestTLSALPN01TLS13
2019/09/13 18:59:00 http: TLS handshake error from 127.0.0.1:51240: tls: client offered only unsupported versions: [303 302 301]
--- PASS: TestTLSALPN01TLS13 (0.03s)
PASS
ok github.com/letsencrypt/boulder/va 1.054s
```
Since we plan to eventually enable TLS 1.3 support and the `GODEBUG`
mechanism tested in the above test is platform-wide vs package
specific I decided it wasn't worth the time investment to write a
similar HTTP-01 unit test that verifies the TLS 1.3 behaviour on a
HTTP-01 HTTP->HTTPS redirect.
Resolves https://github.com/letsencrypt/boulder/issues/4415
* Use `check_call` instead of `check_output`, we don't care about
capturing the output and instead want it to go to stdout so test
failures can be debugged.
* Don't use `shell=True`, it isn't needed here.
* Pipe through the test case filter so that it can be used with
`--test.run` to limit the Go integration tests run.
This reverts commit 796a7aa2f4.
People's tests have been breaking on `docker-compose up` with the following output:
```
ImportError: No module named requests
```
Fixes#4322
* integration: move to Python3
- Add parentheses to all print and raise calls.
- Python3 distinguishes bytes from strings. Add encode() and
decode() calls as needed to provide the correct type.
- Use requests library consistently (urllib3 is not in Python3).
- Remove shebang from Python files without a main, and update
shebang for integration-test.py.
Because the package versions in go.mod match what we use in Godeps.json,
there are no substantive code diffs. However, there are some tiny
differences resulting from how go mod vendors things differently than
godep:
go mod does not preserve executable permissions on shell scripts
Some packages have import lines like:
package ocsp // import "golang.org/x/crypto/ocsp"
godep used to remove the comment from these lines, but go mod vendor does not.
This introduces several indirect dependencies that we didn't have
before. This is because godep used to operate at a package level, but
go mod operates at a module (~= repository) level. So if we used a
given repository, but didn't use all of its packages, we wouldn't
previously care about the transitive dependencies of the packages we
weren't using. However, in the go mod world, once we care about the
repository, we care about all of that repository's transitive
dependencies. AFAICT this doesn't affect vendoring.
Fixes#4116
Adds plumbing in `test.sh` to modify whether the `--load` argument is
provided to `test/integration-test.py`. Use `-e INT_SKIP_LOAD=true` with
`docker-compose run ... ./test.sh` to skip load generation.
The `set` builtin needs to have a `-o` flag before `pipefail` to
actually set that option. Without `-o`, `pipefail` was ignored.
Also, we had a number of test phases that ran various types of linters,
all of them fast. This moves them all together in a single test phase
to make things easier to read, and to ensure they all run before
the integration and unit tests.
Instead of tracking failure status and using the `run` function, just
use the traditional `set -ex pipefail`. This makes the whole thing exit
immediately if any command fails.
Also remove the "contexts" notion and the unused $GOTESTFLAGS variable.
Simplify the gofmt check significantly.
Remove obsolete `sed` for cleaning up after gomock.
Use inefassign's recursion to discover files, and don't bother with run_and_expect_silence.
It will exit 1 on error.
Install grpc codes package from vendor. Without that change it was failing to install.
Precursor to #4116. Since some of our dependencies impose a minimum
version on these two packages higher than what we have in Godeps, we'll
have to bump them anyhow. Bumping them independently of the modules
update should keep things a little simpler.
In order to get protobuf tests to pass, I had to update protoc-gen-go in
boulder-tools. Now we download a prebuilt binary instead of using the
Ubuntu package, which is stuck on 3.0.0. This also meant I needed to
re-generate our pb.go files, since the new version generates somewhat
different output.
This happens to change the tag for pbutil, but it's not a substantive change - they just added a tagged version where there was none.
$ go test github.com/miekg/dns/...
ok github.com/miekg/dns 4.675s
ok github.com/miekg/dns/dnsutil 0.003s
ok github.com/golang/protobuf/descriptor (cached)
ok github.com/golang/protobuf/jsonpb (cached)
? github.com/golang/protobuf/jsonpb/jsonpb_test_proto [no test files]
ok github.com/golang/protobuf/proto (cached)
? github.com/golang/protobuf/proto/proto3_proto [no test files]
? github.com/golang/protobuf/proto/test_proto [no test files]
ok github.com/golang/protobuf/protoc-gen-go (cached)
? github.com/golang/protobuf/protoc-gen-go/descriptor [no test files]
ok github.com/golang/protobuf/protoc-gen-go/generator (cached)
ok github.com/golang/protobuf/protoc-gen-go/generator/internal/remap (cached)
? github.com/golang/protobuf/protoc-gen-go/grpc [no test files]
? github.com/golang/protobuf/protoc-gen-go/plugin [no test files]
ok github.com/golang/protobuf/ptypes (cached)
? github.com/golang/protobuf/ptypes/any [no test files]
? github.com/golang/protobuf/ptypes/duration [no test files]
? github.com/golang/protobuf/ptypes/empty [no test files]
? github.com/golang/protobuf/ptypes/struct [no test files]
? github.com/golang/protobuf/ptypes/timestamp [no test files]
? github.com/golang/protobuf/ptypes/wrappers [no test files]
## CI: restore load-generator run.
This restores running the `load-generator` during CI to make sure it doesn't bitrot. It was previously removed while we debugged the VA getting jammed up and not cleanly shutting down.
Since the global `pebble-challtestsrv` and the `load-generator`'s internal chall test srv will conflict this requires moving the `load-generator` run to the end of integration tests and updating `startservers.py` to allow the load gen integration test code to stop the `pebble-challtestsrv` before starting the `load-generator`.
The `load-generator` and associated config are updated to allow specifying bind addresses for the DNS interface of the internal challtestsrv. Multiple addresses are supported so that the `load-generator`'s chall test srv can listen on port DNS ports Boulder is configured to use. The `load-generator` config now accepts a `fakeDNS` parameter that can be used to specify the default IPv4 address returned by the `load-generator`'s DNS server for A queries.
## load-generator: support different challenges/strategies.
Updates the load-generator to support HTTP-01, DNS-01, and TLS-ALPN-01 challenge response servers. A new challenge selection configuration parameter (`ChallengeStrategy`) can be set to `"http-01"`, `"dns-01"`, or `"tls-alpn-01"` to solve only challenges of that type. Using `"random"` will let the load-generator choose a challenge type randomly.
Resolves https://github.com/letsencrypt/boulder/issues/3900
Resolves https://github.com/letsencrypt/boulder/issues/3872
**Note to reviewers**: There's an outstanding bug that I've tracked down to the `--load` stage of the integration tests that results in one of the remote VA instances in the `test/config-next` configuration under Go 1.11.1 to fail to cleanly shut down. I'm working on finding the root cause but in the meantime I've disabled `--load` during CI so we can unblock moving forward with getting Go 1.11.1 in dev/CI. Tracking this in https://github.com/letsencrypt/boulder/issues/3889
In preparation of `vgo` it seems many upstream projects are adding new
tags. This is causing variations in the `godep-restore` phase of CI when
the CI Godeps picks up a new `Comment` field that isn't present in the
committed JSON. This has broken master ~3 times now and so it seems
prudent to ignore the Comment field entirely for the short term. We
continue to wait for the glorious day in which we can switch to `vgo`
and never muck with `Godeps.json` again.
Prior to Go 1.9 (https://golang.org/doc/go1.9),
various go commands would expand "./..." to include vendor directories.
We worked around this by listing "./..." then grepping out vendor. Now
that we are on Go 1.10 this is no longer necessary. Remove the TESTPATHS
hack.
We still need to exclude certain test directories when running errcheck,
so some of the "go list" logic gets moved into the errcheck stanza.
Also, as of Go 1.10, running coverage on multiple packages in one run is
supported, so replace the "for" loop in the coverage stanza with a
single command.
Also, remove GITHUB_SECRET_FILE and "die," both of which were unused.
- Remove acme-v2 test phase.
- Rename integration-test-v2.py to v2_integration, so it can be imported.
- Import all symbols from v2_integration before running test_*.
- In chisel2:
- Rename DIRECTORY so it doesn't collide.
- Incidental logging and error fixes.
- Merge v1 and v2 load testing into a single function.
- Run cert-checker just once, after all other test cases.
- In v2_integration:
- Remove unnecessary imports.
- Import chisel2 methods in the chisel2 namespace so they don't
collide with chisel methods.
- Remove main and shutdown code.
This commit adds short 15s runs of the load generator against the V1 and
V2 APIs during the three integration test runs (v1 config, v1
config-next, and v2). 15s was selected because 30s caused too much
output and the build log to be truncated.
Presently the latency output is *not* being checked for errors. This was
too flaky in practice.
A fix for a race condition in the load-generator code itself related to
HTTP status code tracking is included in this commit.
The pending authz rate limit also needed to be adjusted to keep the
load-generator from failing requests after hitting 429s.
This change updates boulder-tools to use Go 1.10, and references a
newly-pushed image built using that new config.
Since boulder-tools pulls in the latest Certbot master at the time of
build, this also pulls in the latest changes to Certbot's acme module,
which now supports ACME v2. This means we no longer have to check out
the special acme-v2-integration branch in our integration tests.
This also updates chisel2.py to reflect some of the API changes that
landed in the acme module as it was merged to master.
Since we don't need additional checkouts to get the ACMEv2-compatible
version of the acme module, we can include it in the default RUN set for
local tests.
chisel had verify_ssl=False. Remove that, and set a sensible default
for REQUESTS_CA_BUNDLE to make it easier to run chisel on the command
line. Port the REQUESTS_CA_BUNDLE change into chisel2 as well.
Previously, there was a disagreement between WFE and CA as to what the correct
issuer certificate was. Consolidate on test-ca2.pem (h2ppy h2cker fake CA).
Also, the CA configs contained an outdated entry for "IssuerCert", which was not
being used: The CA configs now use an "Issuers" array to allow signing by
multiple issuer certificates at once (for instance when rolling intermediates).
Removed this outdated entry, and the config code for CA to load it. I've
confirmed these changes match what is currently in production.
Added an integration test to check for this problem in the future.
Fixes#3309, thanks to @icing for bringing the issue to our attention!
This also includes changes from #3321 to clarify certificates for WFE.
Fixes https://github.com/letsencrypt/boulder/issues/3205.
Previously, we would only move aside Godeps.json before running `godep save ./...`. However, in order to get a true picture of what is needed, we must also remove the existing `vendor/` directory.
This change also removes some unnecessary dependencies that have piled up over the years, generally test dependencies. Godep used to vendor such dependencies but no longer does.
Godep apparently breaks when trying to parse code that specifies build tags for versions of golang above that with which it was built (which it shouldn't be parsing in the first place). This breaks the travis tests since `golang.org/x/net/context` now contains golang 1.9 specific code. In order to get around this we temporarily disable the error check for `godep save ./...` in test.sh. Opened #2965 to revert this once Godep is fixed or we move to golang 1.9.
Requires an update to `golang.org/x/net` and adding `golang.org/x/text`.
```
[roland@niya:~/gopath/src/golang.org/x/net]$ go test ./...
ok golang.org/x/net/bpf 0.472s
ok golang.org/x/net/context 0.090s
ok golang.org/x/net/context/ctxhttp 0.161s
? golang.org/x/net/dict [no test files]
ok golang.org/x/net/dns/dnsmessage 0.044s
ok golang.org/x/net/html 0.094s
ok golang.org/x/net/html/atom 0.003s
ok golang.org/x/net/html/charset 0.027s
ok golang.org/x/net/http2 80.253s
? golang.org/x/net/http2/h2i [no test files]
ok golang.org/x/net/http2/hpack 0.064s
ok golang.org/x/net/icmp 0.026s
ok golang.org/x/net/idna 0.035s
? golang.org/x/net/internal/iana [no test files]
? golang.org/x/net/internal/nettest [no test files]
ok golang.org/x/net/internal/socket 0.005s
ok golang.org/x/net/internal/timeseries 0.024s
ok golang.org/x/net/ipv4 0.013s
ok golang.org/x/net/ipv6 0.036s
ok golang.org/x/net/lex/httplex 0.004s
ok golang.org/x/net/nettest 1.164s
ok golang.org/x/net/netutil 0.898s
ok golang.org/x/net/proxy 0.004s
ok golang.org/x/net/publicsuffix 0.202s
ok golang.org/x/net/trace 0.018s
ok golang.org/x/net/webdav 0.061s
ok golang.org/x/net/webdav/internal/xml 0.014s
ok golang.org/x/net/websocket 0.022s
ok golang.org/x/net/xsrftoken 0.025s
[roland@niya:~/gopath/src/golang.org/x/text]$ go test ./...
? golang.org/x/text [no test files]
ok golang.org/x/text/cases 0.439s
? golang.org/x/text/cmd/gotext [no test files]
ok golang.org/x/text/collate 0.038s
ok golang.org/x/text/collate/build 0.024s
? golang.org/x/text/collate/tools/colcmp [no test files]
ok golang.org/x/text/currency 2.961s
ok golang.org/x/text/encoding 0.005s
ok golang.org/x/text/encoding/charmap 0.060s
ok golang.org/x/text/encoding/htmlindex 0.005s
ok golang.org/x/text/encoding/ianaindex 0.030s
? golang.org/x/text/encoding/internal [no test files]
? golang.org/x/text/encoding/internal/enctest [no test files]
? golang.org/x/text/encoding/internal/identifier [no test files]
ok golang.org/x/text/encoding/japanese 0.098s
ok golang.org/x/text/encoding/korean 0.032s
ok golang.org/x/text/encoding/simplifiedchinese 0.100s
ok golang.org/x/text/encoding/traditionalchinese 0.012s
ok golang.org/x/text/encoding/unicode 0.013s
ok golang.org/x/text/encoding/unicode/utf32 0.071s
ok golang.org/x/text/feature/plural 0.352s
ok golang.org/x/text/internal 0.009s
ok golang.org/x/text/internal/catmsg 0.034s
ok golang.org/x/text/internal/colltab 1.817s
ok golang.org/x/text/internal/export/idna 0.040s
? golang.org/x/text/internal/format [no test files]
? golang.org/x/text/internal/gen [no test files]
ok golang.org/x/text/internal/number 0.028s
ok golang.org/x/text/internal/stringset 0.021s
ok golang.org/x/text/internal/tag 0.044s
? golang.org/x/text/internal/testtext [no test files]
ok golang.org/x/text/internal/triegen 0.357s
ok golang.org/x/text/internal/ucd 0.023s
? golang.org/x/text/internal/utf8internal [no test files]
ok golang.org/x/text/language 0.033s
ok golang.org/x/text/language/display 3.917s
ok golang.org/x/text/message 0.033s
ok golang.org/x/text/message/catalog 0.069s
ok golang.org/x/text/runes 0.039s
ok golang.org/x/text/search 0.019s
? golang.org/x/text/secure [no test files]
ok golang.org/x/text/secure/bidirule 0.032s
ok golang.org/x/text/secure/precis 0.066s
ok golang.org/x/text/transform 0.106s
? golang.org/x/text/unicode [no test files]
ok golang.org/x/text/unicode/bidi 0.026s
ok golang.org/x/text/unicode/cldr 0.114s
ok golang.org/x/text/unicode/norm 4.009s
ok golang.org/x/text/unicode/rangetable 1.516s
ok golang.org/x/text/unicode/runenames 0.011s
ok golang.org/x/text/width 0.310s
```
Fixes#2963.
- Add OCSP graphs
- Graph overall request rate
- Separate out WFE vs OCSP graphs
- Fix challenge graph (add a / to endpoint)
- Some incidental changes to "step"
- Add a lint script to check for common dashboard mistakes
This removes the config and code to output to statsd.
- Change `cmd.StatsAndLogging` to output a `Scope`, not a `Statter`.
- Remove the prefixing of component name (e.g. "VA") in front of stats; this was stripped by `autoProm` but now no longer needs to be.
- Delete vendored statsd client.
- Delete `MockStatter` (generated by gomock) and `mocks.Statter` (hand generated) in favor of mocking `metrics.Scope`, which is the interface we now use everywhere.
- Remove a few unused methods on `metrics.Scope`, and update its generated mock.
- Refactor `autoProm` and add `autoRegisterer`, which can be included in a `metrics.Scope`, avoiding global state. `autoProm` now registers everything with the `prometheus.Registerer` it is given.
- Change va_test.go's `setup()` to not return a stats object; instead the individual tests that care about stats override `va.stats` directly.
Fixes#2639, #2733.
Having "coverage" in the default RUN is leaving ".coverprofile" files
all over the src tree. This commit removes this task from the default
RUN list for local work. It is included in travis.yml and will still run
for CI or when specified explicitly with a command line env var
override.
The unit test runs in CI have been taking ~20 minutes. The root cause is
using `-race` on every individual `go test` invocation. We can't switch
to one big `go test` with `-race` instead of individuals if we want test
coverage to be reported. The workaround is to do one big `go test` with
`-race` first, and then many individual `go test`'s to collect coverage
*without* `-race`. This is still faster overall than the current state
of affairs.
Resolves https://github.com/letsencrypt/boulder/issues/2695
This allows us to iterate more easily against the current acme module.
Also, remove nodejs from boulder-tools, clean up a few packages that weren't
previously cleaned up, and install a specific version of protoc-gen-go to match
our vendored grpc.
Add a new tiny client called chisel, in place of test.js. This reduces the
number of language runtimes Boulder depends on for its tests. Also, since chisel
uses the acme Python library, we get more testing of that library, which
underlies Certbot. This also gives us more flexibility to hook different parts
of the issuance flows in our tests.
Reorganize integration-test.py itself. There was not clear separation of
specific test cases. Some test cases were added as part of run_node_test; some
were wrapped around it. There is now much closer to one function per test case.
Eventually we may be able to adopt Python's test infrastructure for these test
cases.
Remove some unused imports; consolidate on urllib2 instead of urllib.
For getting serial number and expiration date, replace shelling out to OpenSSL
with using pyOpenSSL, since we already have an in-memory parsed certificate.
Replace ISSUANCE_FAILED, REVOCATION_FAILED, MAILER_FAILED with simple die, since
we don't use these. Later, I'd like to remove the other specific exit codes. We
don't make very good use of them, and it would be more effective to just use
stack traces or, even better, reporting of which test cases failed.
Make single_ocsp_sign responsible for its own subprocess lifecycle.
Skip running startservers if WFE is already running, to make it easier to
iterate against a running Boulder (saves a few seconds of Boulder startup).
Some commands, like our errcheck command, are very long. When we echo these both
before and after running them, it can obscure what is often a single-line
failure message. Removing the echo after failure makes it easier to spot the
real failure message.
This PR introduces the ability for the ocsp-updater to only resubmit certificates to logs that we are missing SCTs from. Prior to this commit when a certificate was missing one or more SCTs we would submit it to every log, causing unnecessary overhead for us and the log operator.
To accomplish this a new RPC endpoint is added to the Publisher service "SubmitToSingleCT". Unlike the existing "SubmitToCT" this RPC endpoint accepts a log URI and public key in addition to the certificate DER bytes. The certificate is submitted directly to that log, and a cache of constructed resources is maintained so that subsequent submissions to the same log can reuse the stat name, verifier, and submission client.
Resolves#1679
Unlike the `$GOTESTFLAGS` var there was no way to pass in a default
value to `test.sh` via `docker-compose -e` to indicate you only want to
run specific unit tests.
This commit puts the default `$TESTPATHS` into `$DEFAULT_TESTPATHS` and
assigns it to `$TESTPATHS` only if there isn't already a `$TESTPATHS`
value provided.
Example usage, running just the SA unit tests, with the "next" config,
using `-race -v`:
```
docker-compose run -e BOULDER_CONFIG_DIR="test/config-next"
-e RUN="unit" -e GOTESTFLAGS="-v -race"
-e TESTPATHS="github.com/letsencrypt/boulder/sa" boulder ./test.sh
```
Remove the code that echoes exit status on integration failure. It was buggy and
would always print 0, which was confusing. Instead, use `run`.
Rename run_and_comment to run_and_expect_silence, since it doesn't comment
anymore.
This PR modifies the `test.sh` script to allow a `rpm` value in the `RUN` parameter passed to the script via the environment. When present, `make rpm` is invoked and a good status is required for the build to pass.
The `Makefile` was modified to add a `-f` to the `fpm` invocation used by the `rpm` build task to allow the output rpm to be overwritten if present. Otherwise multiple runs of identical bulld (e.g. on a local dev machine) would collide on the .rpm already being present.
Finally `.travis.yml` is updated to include `rpm` in the `RUN` used during CI such that an RPM is built by default for CI runs. I left the default `RUN` in `test.sh` unmodified, so an RPM will not be built for local runs (e.g. `docker-compose run boulder ./test.sh`).
This fixes#2085
This PR changes the `certbot` integration tests that are run from the `0.8.0` release to master.
Note: this is only updating the *tests* that are run. We continue to invoke the `certbot` binary that is installed from jessie-backports as part of the `boulder-tools` image.
This fixes#2021
Jacob Hoffman-Andrews
Roland Bracewell Shoemaker
Daniel McCarney
Daniel McCarney
Roland Bracewell Shoemaker
Blake Griffith