Having "coverage" in the default RUN is leaving ".coverprofile" files
all over the src tree. This commit removes this task from the default
RUN list for local work. It is included in travis.yml and will still run
for CI or when specified explicitly with a command line env var
override.
The unit test runs in CI have been taking ~20 minutes. The root cause is
using `-race` on every individual `go test` invocation. We can't switch
to one big `go test` with `-race` instead of individuals if we want test
coverage to be reported. The workaround is to do one big `go test` with
`-race` first, and then many individual `go test`'s to collect coverage
*without* `-race`. This is still faster overall than the current state
of affairs.
Resolves https://github.com/letsencrypt/boulder/issues/2695
This allows us to iterate more easily against the current acme module.
Also, remove nodejs from boulder-tools, clean up a few packages that weren't
previously cleaned up, and install a specific version of protoc-gen-go to match
our vendored grpc.
Add a new tiny client called chisel, in place of test.js. This reduces the
number of language runtimes Boulder depends on for its tests. Also, since chisel
uses the acme Python library, we get more testing of that library, which
underlies Certbot. This also gives us more flexibility to hook different parts
of the issuance flows in our tests.
Reorganize integration-test.py itself. There was not clear separation of
specific test cases. Some test cases were added as part of run_node_test; some
were wrapped around it. There is now much closer to one function per test case.
Eventually we may be able to adopt Python's test infrastructure for these test
cases.
Remove some unused imports; consolidate on urllib2 instead of urllib.
For getting serial number and expiration date, replace shelling out to OpenSSL
with using pyOpenSSL, since we already have an in-memory parsed certificate.
Replace ISSUANCE_FAILED, REVOCATION_FAILED, MAILER_FAILED with simple die, since
we don't use these. Later, I'd like to remove the other specific exit codes. We
don't make very good use of them, and it would be more effective to just use
stack traces or, even better, reporting of which test cases failed.
Make single_ocsp_sign responsible for its own subprocess lifecycle.
Skip running startservers if WFE is already running, to make it easier to
iterate against a running Boulder (saves a few seconds of Boulder startup).
Some commands, like our errcheck command, are very long. When we echo these both
before and after running them, it can obscure what is often a single-line
failure message. Removing the echo after failure makes it easier to spot the
real failure message.
This PR introduces the ability for the ocsp-updater to only resubmit certificates to logs that we are missing SCTs from. Prior to this commit when a certificate was missing one or more SCTs we would submit it to every log, causing unnecessary overhead for us and the log operator.
To accomplish this a new RPC endpoint is added to the Publisher service "SubmitToSingleCT". Unlike the existing "SubmitToCT" this RPC endpoint accepts a log URI and public key in addition to the certificate DER bytes. The certificate is submitted directly to that log, and a cache of constructed resources is maintained so that subsequent submissions to the same log can reuse the stat name, verifier, and submission client.
Resolves#1679
Unlike the `$GOTESTFLAGS` var there was no way to pass in a default
value to `test.sh` via `docker-compose -e` to indicate you only want to
run specific unit tests.
This commit puts the default `$TESTPATHS` into `$DEFAULT_TESTPATHS` and
assigns it to `$TESTPATHS` only if there isn't already a `$TESTPATHS`
value provided.
Example usage, running just the SA unit tests, with the "next" config,
using `-race -v`:
```
docker-compose run -e BOULDER_CONFIG_DIR="test/config-next"
-e RUN="unit" -e GOTESTFLAGS="-v -race"
-e TESTPATHS="github.com/letsencrypt/boulder/sa" boulder ./test.sh
```
Remove the code that echoes exit status on integration failure. It was buggy and
would always print 0, which was confusing. Instead, use `run`.
Rename run_and_comment to run_and_expect_silence, since it doesn't comment
anymore.
This PR modifies the `test.sh` script to allow a `rpm` value in the `RUN` parameter passed to the script via the environment. When present, `make rpm` is invoked and a good status is required for the build to pass.
The `Makefile` was modified to add a `-f` to the `fpm` invocation used by the `rpm` build task to allow the output rpm to be overwritten if present. Otherwise multiple runs of identical bulld (e.g. on a local dev machine) would collide on the .rpm already being present.
Finally `.travis.yml` is updated to include `rpm` in the `RUN` used during CI such that an RPM is built by default for CI runs. I left the default `RUN` in `test.sh` unmodified, so an RPM will not be built for local runs (e.g. `docker-compose run boulder ./test.sh`).
This fixes#2085
This PR changes the `certbot` integration tests that are run from the `0.8.0` release to master.
Note: this is only updating the *tests* that are run. We continue to invoke the `certbot` binary that is installed from jessie-backports as part of the `boulder-tools` image.
This fixes#2021
The `letsencrypt/boulder-tools` image was recently updated, pulling in version
0.8.0 of certbot. That version stores the output of `certonly` requests in a
different path. In test.sh, we check out a specific tagged release of certbot in
order to get its integration tests. Prior to this commit, we were using
certbot 0.8.0 with the integration tests from version 0.6.0 of certbot,
which looked for `certonly` output in the wrong place, and failed.
This commit changes test.sh to checkout the 0.8.0 branch, and also removes a
temporary shim we used to make the `certbot` command call out to the
`letsencrypt` command.
Also, since the latest version of `letsencrypt/boulder-tools` includes an updated
`protoc-gen-go`, this change also updates the support packages to match.
Since they are only run inside an "if Travis" block, and we know those tools are
installed in the Docker image we use on Travis. This restores coverage reporting
to our builds.
https://github.com/letsencrypt/boulder/pull/1850
That change broke the certbot tests because it switched to a MariaDB
10.1-specific syntax. certbot/certbot#3058 changes the certbot tests to use
Boulder's docker-compose.yml, so they will get MariaDB 10.1 automatically.
* MariaDB 10.1
* MariaDB 10.1 in Docker
* Run docker stuff.
* Improve test.js error.
* Lower log level
* Revert dockerfile to master
* Export debug ports, set FAKE_DNS, and remove container_name.
* Remove typo.
* Make integration-test.py wait for debug ports.
* Use 10.1 and export more Boulder ports.
* Test updates for Docker
Listen on 0.0.0.0 for utility servers.
Make integration-test.py just wait for ports rather than calling startservers.
Run docker-compose in test.sh.
Remove bypass when database exists.
Separate mailer test into its own function in integration test.
Print better errors in test.js.
* Always bring up mysql container.
* Wait for MySQL to come up.
* Put it in travis-before-install.
* Use 127
* Remove manual docker-up.
* Add ifconfig
* Switch to docker-compose run
* It works!
* Remove some spurious env vars.
* Add bash
* try running it
* Add all deps.
* Pass through env.
* Install everything in the Dockerfile.
* Fix install of ruby
* More improvements
* Revert integration test to run directly
Also remove .git from dockerignore and add some packages.
* Revert integration-test.py to master.
* Stop ignoring test/js
* Start from boulder-tools.
* Add boulder-tools.
* Tweak travis.yml
* Separate out docker-compose pull as install.
* Build in install phase; don't bother with go install in Dockerfile
* Add virtualenv
* Actually build rabbitmq-setup
* Remove FAKE_DNS
* Trivial change
* Pull boulder-tools as a separate step so it gets its own timing info.
* Install certbot and protobuf from repos.
* Use cerbot from debian backports.
* Fix clone
* Remove CERTBOT_PATH
* Updates
* Go back to letsencrypt for build.sh
* Remove certbot volume.
* go back to preinstalled letsencrypt
* Restore ENV
* Remove BASH_ENV
* Adapt reloader test so it psses when run as root.
* Fixups for review.
* Revert test.js
* Revert startservers.py
* Revert Makefile.
* Fix go generate command in metrics.
The previous command only worked on OS X. This one works on Linux but not
OS X.
Also add generate phase of test.sh.
* Add mockgen to test setup.
* Fix github-pr-status output.
* Fix envvar style.
* Set xtrace.
* Fix test.sh
* Fix test.sh some more.
* Fix mockgen command.
* Add dependencies for running `go generate`.
* Add protoc-gen-go.
* Fix go get command.
* Fix generate.
* Wait for all.
* Fix generate.
* Update generated pb.
* Fix generate commands for vendored world.
* Update documentation for new vendor style.
* Update grpc package to latest.
* Update caaChecker proto with latest.
* Run go generate only over TESTPATHS
* See if Travis passes under 1.6
* Switch back to 1.5.
* Trim run command.
* Run stringer from correct directory.
* Move generate command.
* Restore and generate
* Fix path.
* list contents of GOPATH.
* Fix stringer by prebuilding.
* Try another import path.
* regenerate bcode_string.
* remove excess package
* pull jsha fork of protoc-gen-go that echoes
* Echo protoc version.
* install from source
* CD back.
* Go back to normal protoc-gen-go
* Fix path
* Move protobuf install into test/setup.sh
* Move before_install to install.
* Set PATH.
* Follow 301 with curl.
* Shuffle test order.
* Swap back test order.
* Restore all tests.
* Restore 1.5.3 to Travis.
* Remove unnecessary wait-or-exit
* Generate metrics mock with latest mockgen.
* Wrap TESTPATHS in curlies
* Remove spurious bracket
* Split out CAA checking service (minus logging etc)
* Add example.yml config + follow general Boulder style
* Update protobuf package to correct version
* Add grpc client to va
* Add TLS authentication in both directions for CAA client/server
* Remove go lint check
* Add bcodes package listing custom codes for Boulder
* Add very basic (pull-only) gRPC metrics to VA + caa-service
* Fix all errcheck errors
* Add errcheck to test.sh
* Add a new sa.Rollback method to make handling errors in rollbacks easier.
This also causes a behavior change in the VA. If a HTTP connection is
abruptly closed after serving the headers for a non-200 response, the
reported error will be the read failure instead of the non-200.
Remove -v for go install.
Don't print success after every successful command.
Remove `run` in front of `go test` to avoid printing each command.
This reduces the number of log lines taken by successful unittests in Travis by
66%, and makes failures much more visible and easy to read.
This accomplishes two things:
- setup.sh should now be usable by the client integration test.
- setup.sh can be used by new project members to simplify first setup.
Update the README to indicate the new file, and to correct some out-of-date
information.
In https://github.com/letsencrypt/boulder/pull/1110 we put
the activate command in the wrong place so it didn't run if
LETSENCRYPT_PATH was set.
Also remove SIMPLE_HTTP_PORT which is no longer necessary. It was used to keep
the build passing as the client transitioned ports. The client now defaults to
5002.
This gets us closer to allowing the client repo to use
integration-test.py. They have a different path without "venv" in it for
their virtualenv set up.
Updates #1101
This generates spurious warnings in Travis:
test.sh: line 94: [: x: integer expression expected
And also prevents certain conditions from failing when they should.
Make `make` aware of output files so it doesn't always have to rebuild. Also
make it use `go install`, which is faster than building files individually.
Now that make is faster, use it in startservers.py to consolidate building
logic. This also has the handy side-effect that ./start.py exposes useful build
information through /build, whereas before only the .rpm packaged version did.
Additionally, this allows us to remove `make` from the Travis matrix, since we
are running `make` as part of the integration test. This means each PR only
triggers two Travis builds instead of one, which means we will get results from
Travis faster.
Also, change the Travis matrix logic to be a list of actions to run, rather than
a list of actions to skip. That fixes
https://github.com/letsencrypt/boulder/issues/817.
Enumerate specific sections of test.sh to run, rather than sections to skip.
Note: ./start.py now installs into ./bin/ instead of $GOPATH/bin.
Only set up GitHub secret file (for PR status reporting) when available, and
decrypt it into /tmp rather than $HOME, to avoid accidentally caching it once
Travis' caching features are available.
Clone letsencrypt repo into $HOME instead of $TMP, to make it possible to cache
eventually.
Remove unused `mysql` dependency in Travis.
Override default Travis install command to prevent it from adding
Godeps/_workspace to GOPATH. When that happens, it hides failures that should
arise from importing non-vendorized paths.
Previously, test.sh was responsible for running venv/bin/activate, meaning that
`python test/amqp-integration-test.py` would fail to run the letsencrypt client.
Now, so long as LETSENCRYPT_PATH is already set to a valid dir (e.g. in your
.bashrc), `python test/amqp-integration-test.py` should work.
This brings the TravisCI build time down to 4 minutes.
It does so by preventing the TravisCI job from having to build the
shared dependencies of the tests multiple times for each `go test` call.
Plus, add to test.sh all future possible packages by using `go list` and
all future possible cmd's to the Makefile with `find`.
Also, fix a bug in external-cert-importer from a bad merge somewhere.
This changes moves from using SQLite in the integration tests and in the
test/boulder-config.json.
It does not port the unit tests over, unfortunately. That's a much more
invasive change.
This also updates the Dockerfile to include the MariaDB and RabbitMQ
requirements of start.py as well as adjusts the CMD to expose the
boulder server to the host machine. The Dockerfile also needed to have
its Go version bumped and the test.sh had to grow some explict
"function"s.
Updates #132
The run function and && combine poorly and cause exported variables to
somehow fail to cross over. run also does status code checking so we can
just use run twice and delete that extra unneeded exit command.
Along the way, move the warning message to where it belongs (creating a
temp directory instead of the user providing a stable path), and use the
new url for letsencrypt/letsencrypt.
Previously, the gofmt test was not properly causing Travis to fail, because it
would call run_and_comment on a printf command, which always succeeded. Instead,
we create a function for the gofmt check that outputs on stdout and returns a
meaningful status code, then call run_and_comment on that.
Also fix up a few places that expected TRAVIS_PULL_REQUEST=false, which doesn't
happen when running locally.
The race detector has found at least one race in our current code. See
issue #465. Turn it on for the unit and integration tests running in
TravisCI.
Also, allow the local user to add new test flags with the `GOTESTFLAGS`
environment variable.
To ease speed of debugging issues, the ability to skip the unit or
integration tests is also provided.
amqp-integration-test.py gains a way to print out what processes
failed to start.
test.sh gains:
* the ability to continue the build correctly if
LETSENCRYPT_PATH was provided but does not exist on disk.
* an explanatory exit message if the LETSENCRYPT_PATH does
exist, but there is not finished build in it.
* a working detection for a python 2.7 binary on OS X
The github-secret.json file can't be decrypted if the PR is coming from
someone who is not a maintainer on boulder. So, just use the boring old
status updates from TravisCI and let the tests continue to run.
This uses a node.js module to post `status` updates to Github, and uses a Travis
secret to authenticate.
- Post comments from static analysis tools
- Change to posting from LetsEncryptBot
- For integration testing, only fail if the compile fails, or
the NodeJS-client fails. Log if the Python client fails.
Travis:
* Downloads the Let's Encrypt client
* Installs system requirements for client
* Sets up virtualenv
Dockerfile:
* Buildout for development
* Includes numerous pacakges needed for integration testing
(including all of the above in Travis)
test.sh:
* If no path is defined for the LE client
* Download the Let's Encrypt client
* Set up virtualenv
test/amqp-integration-test.py:
* Run client test with sensible defaults
* One test: auth for foo.com
This allows us to use the same PKCS#11 key for both cert signing and OCSP
signing, and simplifies config and startup.
This also starts building with -tags pkcs11 in all scripts, which is required
now that the CA can choose between pkcs11 and non-pkcs11.
In order to successfully issue using a pkcs11 key, you'll need to run a version
of Go built off the master branch. The released versions are missing this
commit:
fe40cdd756,
which is necessary for PKCS#11 signing.
Daniel McCarney