Introduces the `authorizationLifetimeDays` and `pendingAuthorizationLifetimeDays` configuration options for `RA`.
If the values are missing from configuration, the code defaults back to the current values (300/7 days).
fixes#2024
This PR removes the use of all anonymous struct fields that were introduced by myself as per my work on splitting up boulder-config (#1962).
The root of the bug was related to the loading of the json configuration file into the config struct. The config structs contained several embedded (anonymous) fields. An embedded (anonymous) field in a struct actually results in the flattening of the json structure. This caused json.Unmarshal to look not at the nested level, but at the root level of the json object and hence not find the nested field (i.e. AllowedSigningAlgos).
See https://play.golang.org/p/6uVCsEu3Df for a working example.
This fixes the reported bug: #2018
For the notify-mailer, this PR fixes a bug with the -end parameter where the default (99999999) would cause a slice index out of range error. This was fixed by setting the -end value to len(m.destinations) in run when it is too large.
For both the notify-mailer and the contact-exporter a bug was fixed that was comparing the required flags against nil when the defaults were set to a non-nil pointer to "". This resulted in confusing errors when the mandatory arguments were not provided.
This PR also adds a separated config example for both the notify-mailer and the contact-exporter into test/config and test/config-next respectively.
Finally a documentation string was added to describe the overall design & usage of both tools, including example invocations.
Boulder issue #2004 describes a panic observed in `getCerts` caused by an index out of range. It appears as though this is caused by a race condition between the initial `SelectOne` lookup for the count of certificates, and the subsequent individual `Select` queries to fetch the Certificates. If the number of eligible certificates changes between these points (e.g. due to certificates expiring) there is a potential that one of the Select calls will return an empty result set. If this happens, then the `lastSerial` update will access an index out of range.
This PR adds an explicit `len` check to the processing loop before the `lastSerial` update. If there are no results returned from the DB query then the loop is broken. This resolves#2004
A test case for the fix was written and included in this PR. The testcase initially caused the out of range panic observed in #2004. After adding the `len` fix in this commit the test began passing without error.
In #1923 we changed reg.Contact to a pointer, which can be nil if the corresponding data from the DB is the literal string "null". This causes panics in expiration-mailer, which we need to fix.
This change fixes modelToRegistration to always return a pointer to a non-nil slice. It also adds an extra sanity check in expiration-mailer itself.
Fixes#1993https://github.com/letsencrypt/boulder/pull/1997
In #1971 we added the CAASERVFAILExceptions config field and argument to NewDNSResolverImpl. This argument only needs to be passed to the VA, where we do CAA validations. However, I accidentally added code to the RA as well to use this new config field. This changes backs that out.
The `letsencrypt/boulder-tools` image was recently updated, pulling in version
0.8.0 of certbot. That version stores the output of `certonly` requests in a
different path. In test.sh, we check out a specific tagged release of certbot in
order to get its integration tests. Prior to this commit, we were using
certbot 0.8.0 with the integration tests from version 0.6.0 of certbot,
which looked for `certonly` output in the wrong place, and failed.
This commit changes test.sh to checkout the 0.8.0 branch, and also removes a
temporary shim we used to make the `certbot` command call out to the
`letsencrypt` command.
Also, since the latest version of `letsencrypt/boulder-tools` includes an updated
`protoc-gen-go`, this change also updates the support packages to match.
Adds a test for CSRs generated using a pre-1.0.2 version of OpenSSL and a buggy client which will fail to parse with Golang 1.6+.
This test checks the values of the bytes in the 8th and 9th offsets, which in a properly formatted CSR should be the version integer declaration bytes, and if the malformed values are present will return a error to the user informing them that they are using an old version of OpenSSL and/or a client which doesn't explicitly set the CSR version.
Fixes#1902.
Moves the wfe to it's own config file.
Each config will now belong in `test/config` and `test/config-next` analogous to `boulder-config` and `boulder-config-next`.
While testing with real proxies I noticed the original CDR implementation was actually pretty broken, this refactors a bit and fixes a number of bugs. With this patch fallback to GPDNS over three distributed test proxies worked perfectly.
(Side note: `nginx` is not a viable forward proxy for this use as it doesn't support SSL, and a bunch of other _real_ forward proxy features, I ended up just using `squid3`.)
The main error in the previous implementation was the fallback was implemented in `getCAASet` which is only called in the old code path (the local CAA impl instead of the remote service) which mean't it wasn't actually being tested in the integration test. This also refactors a few repeated blocks into their own functions. Also there was a unicode encoding problem somewhere with the query string but for the life of me I can't figure out why it was broken now.
Moves the `subscriberAgreementURL` from a top-level config variable to the WFE section. The top level configuration is still respected until we can change the prod configs.
The `regID` parameter in the PA's `WillingToIssue` function was originally used for whitelisting purposes, but is not used any longer. This PR removes it.
Adds a server side unary RPC interceptor which includes basic stats. We could also use this to add a server request ID to the context.Context to identify the call through the system, but really I'd rather do that on the client side before the RPC is sent which requires the client interceptor implementation upstream. Also updates google.golang.org/grpc.
Updates #1880.
For #1928 a notify-mailer command was added to facilitate sending notification emails to a list of email addresses.
The other half of this use-case is generating the list of emails given to notify-mailer as input. This pull request adds a contact-exporter command that allows the operator to dump all of the contact email addresses from boulder into a text file or stdout.
Only contacts with at least one non-expired certificate will be exported. Similarly, only contacts that are a mailto: email address (e.g. not contacts with just a phone number) will be exported.
A new read-only database user (contact_exporter) is created with select permissions on the two required tables (certificates and registrations) for use with the exporter tool.
Future work:
Support exporting registration IDs instead of email addresses
https://github.com/letsencrypt/boulder/pull/1939
Under the new defaults, if the syslog section is missing, we'll use the default
config that we use in prod: no logs to stdout, INFO and below to syslog.
This allows us to remove the syslog section from prod configs, and potentially
move it to individual service configs in the future.
* Improve syslog defaults.
* Add stdout logging for purger test.
* Use plain int for sysloglevel.
* Fix JSON syntax
* Fix syslog config for expired-authz-purger.
https://github.com/letsencrypt/boulder/pull/1932
This commit adds a new notify-mailer command. Outside of the new command, this PR also:
Adds a new SMTPConfig to cmd/config.go that is shared between the expiration mailer and the notify mailer.
Modifies mail/mailer.go to add an smtpClient interface.
Adds a dryRunClient to mail/mailer.go that implements the smtpClient interface.
Modifies the mail/mailer.go MailerImpl and constructor to use the SMTPConfig and a dialer. The missing functions from the smtpClient interface are added.
The notify-mailer command supports checkpointing through --start and --end parameters. It supports dry runs by using the new dryRunClient from the mail package when given the --dryRun flag. The speed at which emails are sent can be tweaked using the --sleep flag.
Unit tests for notify-mailer's checkpointing behaviour, the checkpoint interval/sleep parameter sanity, the sleep behaviour, and the message content construction are included in main_test.go.
Future work:
A separate command to generate the list of destination emails provided to notify-mailer
Support for using registration IDs as input and resolving the email address at runtime.
Resolves#1928. Credit to @jsha for the initial work - I'm just completing the branch he started.
* Adds `notify-mailer` command.
* Adds a new SMTPConfig to `cmd/config.go` that is shared between the
expiration mailer and the notify mailer.
* Modifies `mail/mailer.go` to add an `smtpClient` interface.
* Adds a `dryRunClient` to `mail/mailer.go` that implements the
`smtpClient` interface.
* Modifies the `mail/mailer.go` `MailerImpl` and constructor to use the
SMTPConfig and a dialer. The missing functions from the `smtpClient`
interface are added.
* Fix errcheck warnings
* Review feedback
* Review feedback pt2
* Fixes#1446 - invalid message-id generation.
* Change -configFile to -config
* Test message ID with friendly email
https://github.com/letsencrypt/boulder/pull/1936
The RA UpdateRegistration function merges a base registration object with an update by calling Registration.MergeUpdate. Prior to this commit MergeUpdate only allowed the updated registration object to overwrite the Contact field of the existing registration if the updated reg. defined at least one AcmeURL. This prevented clients from being able to outright remove the contact associated with an existing registration.
This commit removes the len() check on the input.Contact in MergeUpdate to allow the r.Contact field to be overwritten by a []*core.AcmeURL(nil) Contact field. Subsequently clients can now send an empty contacts list in the update registration POST in order to remove their reg contact.
Fixes#1846
* Allow removing registration contact.
* Adds a test for `MergeUpdate` contact removal.
* Change `Registration.Contact` type to `*[]*core.AcmeURL`.
* End validateContacts early for empty contacts
* Test removing reg. contact more thoroughly.
Presently clients may request a new AuthZ be created for a domain that they have already proved authorization over. This results in unnecessary bloat in the authorizations table and duplicated effort.
This commit alters the `NewAuthorization` function of the RA such that before going through the work of creating a new AuthZ it checks whether there already exists a valid AuthZ for the domain/regID that expires in more than 24 hours from the current date. If there is, then we short circuit creation and return the existing AuthZ. When this case occurs the `RA.ReusedValidAuthz` counter is incremented to provide visibility.
Since clients requesting a new AuthZ and getting an AuthZ back expect to turn around and post updates to the corresponding challenges we also return early in `UpdateAuthorization` when asked to update an AuthZ that is already valid. When this case occurs the `RA.ReusedValidAuthzChallenge` counter is incremented.
All of the above behaviour is gated by a new RA config flag `reuseValidAuthz`. In the default case (false) the RA does **not** reuse any AuthZ's and instead maintains the historic behaviour; always creating a new AuthZ when requested, irregardless of whether there are already valid AuthZ's that could be reused. In the true case (enabled only in `boulder-config-next.json`) the AuthZ reuse described above is enabled.
Resolves#1854
Resolves#1810 by automatically updating the RA ratelimit.RateLimitConfig whenever the backing config file is changed. Much like the Policy Authority uses a reloader instance to support updating the Hostname policy on the fly, this PR changes the Registration Authority to use a reloader for the rate limit policy file.
Access to the ra.rlPolicies member is protected with a RWMutex now that there is a potential for the values to be reloaded while a reader is active.
A test is introduced to ensure that writing a new policy YAML to the policy config file results in new values being set in the RA's rlPolicies instance.
https://github.com/letsencrypt/boulder/pull/1894
Both `test-cert.pem` and `test-cert-b.pem` have expiry dates of June 5th, 2016. Since I'm writing this commit on June 6th, 2016 you might be able to guess that this caused a problem :-)
Both `TestGenerateOCSPResponse` and `TestFindStaleOCSPResponses` were failing with a differing actual vs expected value in testing. The SQL query for `findStaleOCSPResponses` specifically excludes certificates that have a `cert.expires` that is `< now()`. Since both test certs were newly expired, the unit tests were getting different results than expected.
This commit replaces both test certs with regenerated copies that expire in 2036. Note: since these are regenerated the public keys and a few other fields not-relevant to testing may have changed.
As documented in Issue #1872 the `getSerialsIssuedSince` `for { }` loop will never terminate under sufficient issuance load. The behaviour prior to this commit would only terminate the loop when `sql.ErrNoRows` or exactly 0 rows were returned by the query. If we sustain sufficient issuance requests that there are always a few certificates issued since the last query the loop will never terminate.
This commit changes the loop behaviour such that rather than terminating when exactly *0* rows are returned, we terminate when a number of rows less than the configured `batchSize` is returned.
* Add cmd/expired-authz-purger with integration test
* Return count
* gofmt >.>
* add to boulder-config-next.json
* Commit missing file
* Exec on the dbMap
* fprintf the error message
* Review fixes + test
* Review fixes pt. 1
* Review fixes pt. 2 (actually add test file this time :|)
* Fix prompt
* Switch to using flag lib
* Use COUNT(1)
* Revert config -> flag stuff
* Review fixes
* Revert-revert COUNT(1) change
* Review fixes pt. 1
* Nest config struct
* Test review fixes
* Factor out getting future output with FAKECLOCK
* Review fixes pt. 2
* Review fixes pt. 3
Fixes#799, ensuring that these files at least count towards our coverage numbers and show up on coveralls' "least covered" list. Improving their coverage is part of the overall long-term project of coverage improvement.
In this PR, logger is passed to the following callers:
NewWebFrontEndImpl
NewCertificateAuthorityImpl
NewValidationAuthorityImpl
NewAmqpRPCServer
newUpdater
NewRegistrationAuthorityServer
This reduces the usage of a global singleton logger and allows tests to consistently use a mock logger.
Fixes#1642
* remove blog.Get() in wfe
* remove blog.Get() from va
* remove Blog.Get() from ca
* remove blog.Get() from oscp updater, ampq rpc server, registration authority server
* removed some pointless logging code
* remove one added newline
* fix format issue
* fix setup function to return *blog.Mock instead of being passed in
* remove useless blog.NewMock() call
- Run both gRPC and AMQP servers simultaneously
- Take explicit constructor parameters and unexport fields that were previously set by users
- Remove transitional DomainCheck code in RA now that GSB is enabled.
- Remove some leftover UpdateValidation dummy methods.
The rate limiting code previously lived in the `cmd` package without a clear justification for why. This commit moves the rate limiting code to its own `ratelimit` package and updates import paths as required. Notably all references from the `cmd` package's exported `LoadRateLimitPolicies`, `RateLimitPolicy`, and `RateLimitConfig` were moved to use `ratelimit`. This removed the `cmd` import from a couple of callers (nice!).
* Split CSR testing and name hoisting into own functions, verify CSR in RA & CA
* Move tests around and various other fixes
* 1.5.3 doesn't have the needed stringer
* Move functions to their own lib
* Remove unused imports
* Move MaxCNLength and BadSignatureAlgorithms to csr package
* Always normalizeCSR in VerifyCSR and de-export it
* Update comments
Patrick Figel