Adds a daemon which monitors the new blockedKeys table and checks for any unexpired, unrevoked certificates that are associated with the added SPKI hashes and revokes them, notifying the user that issued the certificates.
Fixes#4772.
In #3708, we added formatters for the the convenience methods in the
`probs` package.
However, in #4783, @alexzorin pointed out that we were incorrectly
passing an error message through fmt.Sprintf as the format parameter
rather than as a value parameter.
I proposed a fix in #4784, but during code review we concluded that the
underlying problem was the pattern of using format-style functions that
don't have some variant of printf in the name. That makes this wrong:
`probs.DNS(err.Error())`, and this right: `probs.DNS("%s", err)`. Since
that's an easy mistake to make and a hard one to spot during code review,
we're going to stop using this particular pattern and call `fmt.Sprintf`
directly.
This PR reverts #3708 and adds some `fmt.Sprintf` where needed.
For now this mainly provides an example config and confirms that
log-validator can start up and shut down cleanly, as well as provide a
stat indicating how many log lines it has handled.
This introduces a syslog config to the boulder-tools image that will write
logs to /var/log/program.log. It also tweaks the various .json config
files so they have non-default syslogLevel, to ensure they actually
write something for log-validator to verify.
The problem with this approach is that there is no way to guarantee the output
is copied to syslog / stdout before shutdown. This is particularly evident when
`log.Fatal` is used, because that calls `os.Exit` immediately after `l.Output`,
creating a race condition where the log line might or might not get printed
before the program exits.
Reverting this change means that in case some component does call `log.Fatal`
we'll still get the output from stdout.
This also changes one instance in cmd/shell.go where we call `log.Fatal` to use
`logger.Errf`.
Since Boulder's log system adds checksums to lines, but log-validator
processes entries on a per-line basis, including newlines in log
messages can cause a validation failure.
This can happen when a misconfiguration redirects a certain path to
itself, doubled. After 10 redirects the error message can get quite
long. Instead we halt things at 2000 bytes, which should be more than
enough.
Previously, the test called `.Round(time.Minute)` on the expected
and actual expiration times, intending to perform an "approximately
equal" function.
However, when the expected and actual times differed by a second, but
they happened to fall on opposite sides of a rounding interval (i.e. 30
seconds into a minute), they would be rounded in opposite directions,
resulting in a conclusion that they were not equal.
This change instead defines an acceptable range of plus or minus a
minute for the expiration time, and checks that the actual expiration
time is in that interval.
This makes it easier to configure additional linters, and provides us an
easy command to run locally.
The initial set of linters reflects those we are already running:
govet gofmt ineffassign errcheck misspell staticcheck
Note that misspell is in addition to the Python codespell package.
Since the invocation of these linters from golangci-lint is slightly
different from how we currently invoke them, there are some new
findings. This PR won't pass tests until #4763, #4764, and #4765 are
merged.
Incidentally, rename strat -> strategy to appeal misspell.
In addition to base64(sha256(spki)).
As part of that, change KeyDigest to return [32]byte, and add KeyDigestB64 which provides the base64-encoded output that KeyDigest used to provide. Also update all call sites.
Right now we show output like:
Traceback (most recent call last):
File "test/integration-test.py", line 60, in run_go_tests
subprocess.check_call(cmdLine, shell=False, stderr=subprocess.STDOUT)
File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['go', 'test', '-tags', 'integration', '-count=1', '-race', './test/integration']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test/integration-test.py", line 414, in
main()
File "test/integration-test.py", line 293, in main
run_go_tests(args.test_case_filter)
File "test/integration-test.py", line 62, in run_go_tests
raise(Exception("%s. Output:\n%s" % (e, e.output)))
Exception: Command '['go', 'test', '-tags', 'integration', '-count=1', '-race', './test/integration']' returned non-zero exit status 1. Output:
None
This change removes the try / raise clauses that were causing this
double exception logging. The original purpose of these clauses was to
make sure we logged output on failure. To continue to fulfill that
purpose, I switched the run function to use check_call instead of
check_output. check_output captures the stdout; check_call emits it to
the caller's stdout as normal, so we still see the output.
I also changed the two cases that actually wanted to process output so
they use check_output directly.
This adds staticcheck to our "lints" CI, with a list of excluded checks. Some of these are checks that we don't care about much (like error string capitalization). Others are nice to fix (possible nil pointer dereferences in _test.go files), but we'd like to land the automated checking first to catch any new issues, then later winnow down the list.
This builds on #4726, #4725, and #4722, which addressed some of the categories of findings from staticcheck.
Instead of installing Certbot from the repo, install the python-acme
library (the only piece we need) from the apt repository. This also
allows us to skip installing build dependencies for Certbot.
Uninstall cmake after building.
Clean the various Go caches.
Move codespell and acme into requirements.txt. Don't use virtualenv anymore.
This reduces image size from 1.4 GB to 1.0 GB.
Incidentally, move the Go install to its own phase in the Dockerfile.
This will give it its own image layer, making rebuilds faster.
We have a comment in .travis.yml indicating that Travis' version of docker-compose is too old for us, but that's no longer true, so we can simplify things and speed up by using the stock installed one.
The list of custom hostnames in .travis.yml that all point to localhost are no longer used. Instead we use a custom DNS server within our docker-compose setup. So this PR removes those hostnames.
Remove staging and release as special branches to have tests run. We no longer use branches as part of our release process.
In our builds, under "View config", Travis was showing two warnings and
4 info-level reports:
root: deprecated key sudo (The key `sudo` has no effect anymore.)
root: deprecated key sudo (The key `sudo` has no effect anymore.)
root: missing dist, using the default xenial
root: missing os, using the default linux
root: key matrix is an alias for jobs, using jobs
root: key matrix is an alias for jobs, using jobs
I believe this should fix all of these. I've selected xenial as the dist
for now because that's what we've been defaulting to, but we can switch
to bionic, which is newer (though it doesn't make much difference since
we're running most things in Docker).
As part of the process, pin specific versions of protoc-gen-go, mockgen,
and goveralls. Protoc-gen-go recently released a version that was incompatible
with our current version of gRPC. Mockgen has a version that was generating
spurious diffs in our generate test phase, and goveralls recently added
some code that calls git branch --format=..., which breaks on the version of
git in our Docker image.
Pinning versions required forcing go get into module-aware mode, since the
old-style go get doesn't understand versions.
This is currently unused, and was a copy of the Go upstream package
anyhow. We will either re-copy when needed or wait for Go 1.15, which
will include the updates we need.
Merges gen-ca and gen-key into a single tool that can be used to complete a key/certificate generation ceremony. The driving idea here is that instead of having to write out multiple long commands in a specific order in order to complete a ceremony a configuration file is fed to a single binary. This config file contains all of the information needed to complete the ceremony, and can be easily tested outside of the secure environment before hand without fear of later typing a command/flag incorrectly etc.
The tooling works against the test hardware I have (there are minimal changes to the actual PKCS#11 code behind the scenes). Specific attention should be given to the documentation, and the general UX of the tool.
Fixes#4639 and fixes#4667.
Roland Bracewell Shoemaker
Jacob Hoffman-Andrews
Daniel McCarney