Previously, the CA would log the full PEM encoded certificate signing request and the signed certificate. This PR, removes that, and logs a base64 encoded version of the request and the certificate.
Due to the fact that cfssl returns the certificate in PEM format regardless, we decode the certificate into DER and log the base64 version. If there is any failure in the decoding from PEM to DER, the CA will log the PEM encoded version of the certificate (this is the only case).
Furthermore, the Signing Success message was moved after the certificate is checked for 0 length and decoded into DER, which seems to make logical sense to me.
Fixes#1968
Part of #2080.
This change vendors `crypto/x509`, `crypto/x509/pkix`, and `encoding/asn1` from 1d5f6a765d. That commit is a direct child of the Go 1.5.4 release tag, so it contains the same code as the current Go version we are using. In that commit I rewrote imports in those packages so they depend on each other internally rather than calling out to the standard library, which would cause type disagreements.
I changed the imports in each place where we're parsing CSRs, and imported under a different name `oldx509`, both to avoid collisions and make it clear what's going on. Places that only use `x509` to parse certificates are not changed, and will use the current standard library.
This will unblock us from moving to Go 1.6, and subsequently Go 1.7.
This PR modifies the `test.sh` script to allow a `rpm` value in the `RUN` parameter passed to the script via the environment. When present, `make rpm` is invoked and a good status is required for the build to pass.
The `Makefile` was modified to add a `-f` to the `fpm` invocation used by the `rpm` build task to allow the output rpm to be overwritten if present. Otherwise multiple runs of identical bulld (e.g. on a local dev machine) would collide on the .rpm already being present.
Finally `.travis.yml` is updated to include `rpm` in the `RUN` used during CI such that an RPM is built by default for CI runs. I left the default `RUN` in `test.sh` unmodified, so an RPM will not be built for local runs (e.g. `docker-compose run boulder ./test.sh`).
This fixes#2085
After #2069 landed we no longer have a `boulder-config.json` file. This breaks the `make rpm` flow because the `rpm` make target has an explicit reference to this file as an argument to `fpm`.
This commit replaces the `test/boulder-config.json` reference with its replacement, the `test/config/` directory.
This PR adds a `printStatus` function that is called every iteration of the mailer's `run()` loop. The status output is logged at the `info` level and includes the destination email, the current message being sent, the total number of messages to send, and the elapsed time since `run()` started.
The status output can be disabled by lowering the default syslog level in the `notify-mailer` config.
Additionally, this PR adds stats support for the mailer package. Three new stats are
published during the `MailerImpl`'s `SendMail` function (called in a loop by the mailer utilities):
`Mailer.SendMail.Attempts`
`Mailer.SendMail.Successes`
`Mailer.SendMail.Errors`
This PR removes two stats from the `expiration-mailer` that are redundant copies of the above:
`Mailer.Expiration.Errors.SendingNag.SendFailure`
`Mailer.Expiration.Sent`
This resolves#2026.
Presently the `contact-exporter` command only exports registration IDs that have existing certificates that [have not yet expired](217b79b8c2/cmd/contact-exporter/main.go (L39)). The threshold is [the current time](217b79b8c2/cmd/contact-exporter/main.go (L42)) when run.
This PR adds a user configurable grace period that we can use to offset this expiration check. E.g. export all registration IDs that have unexpired certificates, or certificates that expired in the last N days. This resolves#1959
This PR also catches up the documentation strings for both the `expiration-mailer` and the `contact-exporter` to include the changes landed in #1958. Both have been updated to describe operating on registration IDs in place of bare emails.
This PR removes the use of the global configuration variable BOULDER_CONFIG. It also removes the global configuration struct cmd.Config. Furthermore, it removes the dependency codegangsta/cli and the last bit of code that was using it cmd/single-ocsp/main.go.
This is the final (hopefully) pull request in the work to remove the reliance on a global configuration structure. Included below is a history of all other pull requests relevant in accomplishing this:
WFE (#1973)
RA (#1974)
SA (#1975)
CA (#1978)
VA (#1979)
Publisher (#2008)
OCSP Updater (#2013)
OCSP Responder (#2017)
Admin Revoker (#2053)
Expiration Mailer (#2036)
Cert Checker (#2058)
Orphan Finder (#2059)
Single OCSP (this PR)
Closes#1962
This PR removes the Shutdown{Kill,Stop}Timeout fields from `cmd/boulder-wfe/main.go`. These fields are set in `main.go` but never used. The relevant config fields only need to be used when creating an `httpdown.HTTP` in `cmd/boulder-wfe/main.go`.
Existing duration parsing was replaced with `cmd.ConfigDuration` use.
For #2001 an optimization was added to the WFE to avoid invoking the RA's `UpdateRegistration` method when a trivial (e.g. `{"resource:"reg"}`) update is received. Instead the WFE returned the trivial update back to the client immediately.
This is contrary to the ACME spec which indicates:
> Servers SHOULD NOT respond to GET requests for registration resources as these requests are not authenticated. If a client wishes to query the server for information about its account (e.g., to examine the “contact” or “certificates” fields), then it SHOULD do so by sending a POST request with an empty update. That is, it should send a JWS whose payload is trivial ({“resource”:”reg”}).
The optimization regression was captured in issue #2066 when it broke at least one client implementation.
This removes the empty reg update optimization and passes all POST's to the RA. The RA will in turn fetch the existing registration to return to the client. The second half of the #2001 optimizations remains in place, no DB UPDATE's will be performed if the new registration content doesn't differ from the existing registration content (as determine by the return of `registration.MergeUpdate`).
Since the WFE optimization is no longer in place the `FailureRegistrationAuthority` mock isn't required and is removed. Similarly `TestEmptyRegistration` in `wfe_test.go` is changed from testing for the optimization to testing for the ACME described "get registration info" behaviour.
This fixes#2066
Another step in completing #1962, which will remove the global configuration file and codegangsta/cli from boulder. 3 more to go!
This PR, is a little bit different than others in that there was a lot more reliance on codegangsta/cli especially in the implementation of subcommands. I put some thought into creating our own SubCommand struct, but given the lack of complexity it seemed unnecessary as the same could be accomplished with slightly more advanced usage of os and flag.
This pulls in two changes that fix the case where pkcs11 failed to load the module. One where the return value was nil, and one where an empty string could be passed to libtool-ltdl.
Tests pass for both packages.
Right now we use the Source field for both DB and file URLs. However, we want to move to the DBConnect config field, so that we can take advantage of the code that reads DSNs from a file on disk. It turns out the existing code didn't work if you configure a dbConnect string, because it would error out with:
"source" parameter not found in JSON config
After rearranging, both methods should work.
Previously, if a certificate was skipped by the expiration mailer due to being
renewed already, we wouldn't update its lastnag time. However, this meant that
already-renewed certificates would clog up the results of the query
expiration-mailer does to find expired certs. Since this query has a limit (1000
in practice), we might find only renewed certificates on each query, even when
there are non-renewed certificates available to alert about. Then we'd never
make forward progress.
This change updates the stored lastExpirationNagSent field when a certificate is
skipped over due to renewal, so that it isn't included in the first-step query.
Fixes#2054
Docker's now our main dev env, and running outside of Docker requires a change
to not use SoftHSM. Update the README to reflect that.
Also include instructions in the README on how to run SoftHSM on the host, update make-softhsm to match the current setup, and describe the mounting of GOPATH for instant updates to code.
This PR adds a stat that is emitted when any of the nag groups are operating at capacity. The mailer is considered at capacity when the number of certs returned by the query in findExpiringCertificates is equal to the configured -cert_limit.
The at capacity stat names take the form: "Mailer.Expiration.Errors.Nag-XXXXX.AtCapacity" where XXXXX is the String() representation of the nagCheck offset nag time. Allowing the capacity alert to be specified per-nag group. As an example, a nag time of 48hrs with a nag check of 24hrs would produce a stat: "Mailer.Expiration.Errors.Nag-72h0m0s.AtCapacity" when it reached a capacity state.
This will allow creation of an alert for the conditions that caused issue #2002 to manifest.
In order to unit test with a mock statter it was also required to swap out the time.Since calls to equivalent dateB.sub(dateA) calls using the fake clock.
As described in issue #2025 if the contents of a DBConnectFile argument ends in a trailing newline an error is produced when later using the contents as a DB configuration URL.
This PR changes the URL function of the DBConfig type to strip leading and trailing whitespace (including newlines) when reading the DBConnectFile contents. This resolves#2025.
Presently the `contact-exporter` dumps a list of email addresses to an intermediate file. That file can be provided as input to `notify-mailer` to send messages to the listed email addresses. This has the downside that if a user updates their registration contact address between the time the export is run and the notification emails are sent the message will go to the wrong address.
This PR addresses this by adding a new default output mode to the `contact-exporter` that writes JSON serialized objects containing both a registration ID and an email address to an intermediate file. The `notify-mailer` is updated to support reading this file, deserializing the JSON, and resolving the reg IDs to fresh email addresses before sending messages.
The "classic" behaviour of using bare email addresses as the intermediate form is supported with both the `contact-exporter` and the `notify-mailer` by providing the `--emails` bool flag at the command line.
Resolves#1951
Updates `github.com/cloudflare/cfssl` and `github.com/google/certificate-transparency/go` to current master. CFSSL has re-structured some of the `signer/local` code which should be given a once-over. Otherwise everything seems mostly benign and/or doesn't affect our usage.
Vendored tests pass.
Introduces the `authorizationLifetimeDays` and `pendingAuthorizationLifetimeDays` configuration options for `RA`.
If the values are missing from configuration, the code defaults back to the current values (300/7 days).
fixes#2024
The PKCS11 proxy requires `test/test-ca.key.pem` in DER form. Rather
than generating it when it doesn't exist in `test/entrypoint.sh` and
adding it to the gitignore we've opted to check it in directly.
Instead of reading the CA key from a file on disk into memory and using that for signing in `boulder-ca` this patch adds a new Docker container that runs SoftHSM and pkcs11-proxy in order to hold the key and perform signing operations. The pkcs11-proxy module is used by `boulder-ca` to talk to the SoftHSM container.
This exercises (almost) the full pkcs11 path through boulder and will allow testing various HSM related failures in the future as well as simplifying tuning signing performance for benchmarking.
Fixes#703.
This PR changes the `certbot` integration tests that are run from the `0.8.0` release to master.
Note: this is only updating the *tests* that are run. We continue to invoke the `certbot` binary that is installed from jessie-backports as part of the `boulder-tools` image.
This fixes#2021
This PR adds two optimizations to fix the optimistic lock errors observed in #1986.
First, the WFE now returns early for registration POST's (before invoking the RA and SA) when the POST body is the trivial update (`{"resource":"reg"}`). This prevents any DB operations from being performed when there is no work to be done.
Second, the RA now tracks whether a update actually changes the base registration's `Contact` slice, or `Agreement` string. If the proposed update doesn't change either of these fields then the RA will return early before handing the update to the SA.
Both changes save database operations from being performed needlessly and will help avoid the optimistic lock errors we observed when a problematic client was POSTing the trivial update repeatedly in a short period.
The fix was verified as follows: I checked out master and artificially introduced lock contention into the SA by adding a 2s sleep into `UpdateRegistration` between fetching the `existingRegModel` to get the `LockCol` value and calling `ssa.dbMap.Update`. With the sleep in place & two certbot clients posting matching registration updates the lock contention error is produced as expected. After checking out the `empty-reg-updates` branch, re-adding the sleep to the SA, and performing the same two client reg updates no error is produced.
This PR removes the use of all anonymous struct fields that were introduced by myself as per my work on splitting up boulder-config (#1962).
The root of the bug was related to the loading of the json configuration file into the config struct. The config structs contained several embedded (anonymous) fields. An embedded (anonymous) field in a struct actually results in the flattening of the json structure. This caused json.Unmarshal to look not at the nested level, but at the root level of the json object and hence not find the nested field (i.e. AllowedSigningAlgos).
See https://play.golang.org/p/6uVCsEu3Df for a working example.
This fixes the reported bug: #2018
Switches from using `boulder-tools:latest` to using `boulder-tools:2016-07-05`. Using this tag format going forward we can preserve old versions of the image and not break master tests when we need to update images.
For the notify-mailer, this PR fixes a bug with the -end parameter where the default (99999999) would cause a slice index out of range error. This was fixed by setting the -end value to len(m.destinations) in run when it is too large.
For both the notify-mailer and the contact-exporter a bug was fixed that was comparing the required flags against nil when the defaults were set to a non-nil pointer to "". This resulted in confusing errors when the mandatory arguments were not provided.
This PR also adds a separated config example for both the notify-mailer and the contact-exporter into test/config and test/config-next respectively.
Finally a documentation string was added to describe the overall design & usage of both tools, including example invocations.
Boulder issue #2004 describes a panic observed in `getCerts` caused by an index out of range. It appears as though this is caused by a race condition between the initial `SelectOne` lookup for the count of certificates, and the subsequent individual `Select` queries to fetch the Certificates. If the number of eligible certificates changes between these points (e.g. due to certificates expiring) there is a potential that one of the Select calls will return an empty result set. If this happens, then the `lastSerial` update will access an index out of range.
This PR adds an explicit `len` check to the processing loop before the `lastSerial` update. If there are no results returned from the DB query then the loop is broken. This resolves#2004
A test case for the fix was written and included in this PR. The testcase initially caused the out of range panic observed in #2004. After adding the `len` fix in this commit the test began passing without error.
The RA test uses a `DummyValidationAuthority` which is called from `ra.UpdateAuithorization` in a goroutine. Both the tester and the spawned goroutine want to mutate/read fields on the `DummyValidationAuthority` which causes a race. This PR fixes the race by blocking on a `chan struct{}` until the `DummyValidationAuthority` has mutated the required state.
Fixes#1980.
https://github.com/letsencrypt/boulder/pull/1995
This PR replaces the `x/net/publicsuffix` package with `weppos/publicsuffix-go`.
The conversations that leaded to this decision are #1479 and #1374. To summarize the discussion, the main issue with `x/net/publicsuffix` is that the package compiles the list into the Go source code and doesn't provide a way to easily pull updates (e.g. by re-parsing the original PSL) unless the entire package is recompiled.
The PSL update frequency is almost daily, which makes very hard to recompile the official Golang package to stay up-to-date with all the changes. Moreover, Golang maintainers expressed some concerns about rebuilding and committing changes with a frequency that would keep the package in sync with the original PSL. See https://github.com/letsencrypt/boulder/issues/1374#issuecomment-182429297
`weppos/publicsuffix-go` contains a compiled version of the list that is updated weekly (or more frequently). Moreover, the package can read and parse a PSL from a String or a File which will effectively decouple the Boulder source code with the list itself. The main benefit is that it will be possible to update the definition by simply downloading the latest list and restarting the application (assuming the list is persisted in memory).
Jacob Hoffman-Andrews
Ben Irving
Daniel McCarney
Roland Bracewell Shoemaker
Patrick Figel
Roland Bracewell Shoemaker
Simone Carletti