Commit Graph

216 Commits

Author SHA1 Message Date
Aaron Gable 993953bf81
Remove ca gRPC wrapper (#5330)
Delete the CertificateAuthorityClientWrapper, OCSPGeneratorClientWrapper,
and CertificateAuthorityServerWrapper structs, which provided no error
checking above and beyond their wrapped types. Replace them with the
corresponding auto-generated gRPC types in calling code. Update some
mocks to have the necessary variadic grpc.CallOption parameter. Finally,
delete the now-unused core.CertificateAuthority interface.

Fixes #5324
2021-03-11 10:45:46 -08:00
Aaron Gable 16c7a21a57
RA: Multi-issuer support for OCSP purging (#5160)
The RA is responsible for contacting Akamai to purge cached OCSP
responses when a certificate is revoked and fresh OCSP responses need to
be served ASAP. In order to do so, it needs to construct the same OCSP
URLs that clients would construct, and that Akamai would cache. In order
to do that, it needs access to the issuing certificate to compute a hash
across its Subject Info and Public Key.

Currently, the RA holds a single issuer certificate in memory, and uses
that cert to compute all OCSP URLs, on the assumption that all certs
we're being asked to revoke were issued by the same issuer.

In order to support issuance from multiple intermediates at the same
time (e.g. RSA and ECDSA), and to support rollover between different
issuers of the same type (we may need to revoke certs issued by two
different issuers for the 90 days in which their end-entity certs
overlap), this commit changes the configuration to provide a list of
issuer certificates instead.

In order to support efficient lookup of issuer certs, this change also
introduces a new concept, the Chain ID. The Chain ID is a truncated hash
across the raw bytes of either the Issuer Info or the Subject Info of a
given cert. As such, it can be used to confirm issuer/subject
relationships between certificates. In the future, this may be a
replacement for our current IssuerID (a truncated hash over the whole
issuer certificate), but for now it is used to map revoked certs to
their issuers inside the RA.

Part of #5120
2020-11-06 13:58:32 -08:00
Jacob Hoffman-Andrews cfe943fea5
Factor out idForIssuer. (#5102)
This was already part done: There is an ID() method in issuance. This
change extends that by:
 - Defining a type alias indicating something is an IssuerID.
 - Defining issuance.Certificate, which also has an ID() method,
   so that components that aren't the CA can load certificates and
   use the type system to mark them as issuers (and get their IDs).
 - Converting akamai-purger and ca to use the new types.
 - Removing idForIssuer from ca.go.
2020-10-06 15:04:43 -07:00
Aaron Gable 2d14cfb8d1
Add gRPC Health service to all Boulder services (#5093)
This health service implements the gRPC Health Checking
Protocol, as defined in 
https://github.com/grpc/grpc/blob/master/doc/health-checking.md
and as implemented by the gRPC authors in
https://pkg.go.dev/google.golang.org/grpc/health@v1.29.0

It simply instantiates a health service, and attaches it to the same
gRPC server that is handling requests to the primary (e.g. CA) service.
When the main service would be shut down (e.g. because it caught a
signal), it also sets the status of the service to NOT_SERVING.

This change also imports the health client into our grpc client,
ensuring that all of our grpc clients use the health service to inform
their load-balancing behavior.

This will be used to replace our current usage of polling the debug
port to determine whether a given service is up and running. It may
also be useful for more comprehensive checks and blackbox probing
in the future.

Part of #5074
2020-10-06 12:14:02 -07:00
Roland Bracewell Shoemaker 7853b12cb3
Remove support for issuing certificates with no CN (#5008)
We'd like to issue certs with no CN eventually, but it's not
going to happen any time soon. In the mean time, the existing
code never gets exercised and is rather complex, so this
removes it.
2020-08-05 09:15:30 -07:00
Aaron Gable 3a03e86e89
Standardize all proto import names (#4970)
We previously used mixed case names for proto imports
(e.g. both `caPB` and `rapb`), sometimes in the same file.
This change standardizes on the all-lowercase spelling,
which was predominant throughout the codebase.
2020-07-20 16:29:17 -07:00
Aaron Gable 91d4e235ad
Deprecate the BlockedKeyTable feature flag (#4881)
This commit consists of three classes of changes:
1) Changing various command main.go files to always behave as they
   would have when features.BlockedKeyTable was true. Also changing
   one test in the same manner.
2) Removing the BlockedKeyTable flag from configuration in config-next,
   because the flag is already live.
3) Moving the BlockedKeyTable flag to the "deprecated" section of
   features.go, and regenerating featureflag_strings.go.

A future change will remove the BlockedKeyTable flag (and other
similarly deprecated flags) from features.go entirely.

Fixes #4873
2020-06-22 16:35:37 -07:00
Roland Bracewell Shoemaker 9df97cbf06
Add a blocked keys table, and use it (#4773)
Fixes #4712 and fixes #4711.
2020-04-15 13:42:51 -07:00
Jacob Hoffman-Andrews 0a0a776882
Fix some misspellings. (#4764)
Found by golangci-lint's `misspell` linter.

`strat` was a false positive vs `start` but I figured it's easier to
just change the name.
2020-04-08 17:26:10 -07:00
Jacob Hoffman-Andrews bef02e782a
Fix nits found by staticcheck (#4726)
Part of #4700
2020-03-30 10:20:20 -07:00
Roland Bracewell Shoemaker b8ee84da7b
Switch GenerateOCSP to directly use protos instead of wrapper (#4549) 2019-11-14 11:10:33 -08:00
Jacob Hoffman-Andrews 49043a4156 Clarify public key blocklist documentation. (#4523)
Previously, we referred to "DER encoded PKIX public keys", but PKIX (RFC 5280)
doesn't define a standalone "public key" type. Instead, it defines
SubjectPublicKeyInfo, containing an algorithm and a BIT STRING. As a
result, SPKI and SPKI hash are more commonly used terms, and we're more
likely to get reports based on those. We should mirror that terminology
in our documentation.
2019-11-04 09:10:36 -05:00
Daniel McCarney f02e9da38f
Support admin. blocking public keys. (#4419)
We occasionally have reason to block public keys from being used in CSRs
or for JWKs. This work adds support for loading a YAML blocked keys list
to the WFE, the RA and the CA (all the components already using the
`goodekey` package).

The list is loaded in-memory and is intended to be used sparingly and
not for more complicated mass blocking scenarios. This augments the
existing debian weak key checking which is specific to RSA keys and
operates on a truncated hash of the key modulus. In comparison the
admin. blocked keys are identified by the Base64 encoding of a SHA256
hash over the DER encoding of the public key expressed as a PKIX subject
public key. For ECDSA keys in particular we believe a more thorough
solution would have to consider inverted curve points but to start we're
calling this approach "Good Enough".

A utility program (`block-a-key`) is provided that can read a PEM
formatted x509 certificate or a JSON formatted JWK and emit lines to be
added to the blocked keys YAML to block the related public key.

A test blocked keys YAML file is included
(`test/example-blocked-keys.yml`), initially populated with a few of the
keys from the `test/` directory. We may want to do a more through pass
through Boulder's source code and add a block entry for every test
private key.

Resolves https://github.com/letsencrypt/boulder/issues/4404
2019-09-06 16:54:26 -04:00
Roland Bracewell Shoemaker acc44498d1 RA: Make RevokeAtRA feature standard behavior (#4268)
Now that it is live in production and is working as intended we can remove
the old ocsp-updater functionality entirely.

Fixes #4048.
2019-06-20 14:32:53 -04:00
Jacob Hoffman-Andrews 0759d2d440 cmd: Split out config structs (#4200)
This follows up on some refactoring we had done previously but not
completed. This removes various binary-specific config structs from the
common cmd package, and moves them into their appropriate packages. In
the case of CT configs, they had to be moved into their own package to
avoid a dependency loop between RA and ctpolicy.
2019-05-06 11:11:08 -04:00
Jacob Hoffman-Andrews 677b9b88ad Remove GSB support. (#4115)
This is no longer enabled in prod; cleaning up the code.

https://community.letsencrypt.org/t/let-s-encrypt-no-longer-checking-google-safe-browsing/82168
2019-03-15 10:24:44 -07:00
Jacob Hoffman-Andrews d1e6d0f190 Remove TLS-SNI-01 (#4114)
* Remove the challenge whitelist
* Reduce the signature for ChallengesFor and ChallengeTypeEnabled
* Some unit tests in the VA were changed from testing TLS-SNI to testing the same behavior
  in TLS-ALPN, when that behavior wasn't already tested. For instance timeouts during connect 
  are now tested.

Fixes #4109
2019-03-15 09:05:24 -04:00
Roland Bracewell Shoemaker 3e54cea295 Implement direct revocation at RA (#4043)
Implements a feature that enables immediate revocation instead of marking a certificate revoked and waiting for the OCSP-Updater to generate the OCSP response. This means that as soon as the request returns from the WFE the revoked OCSP response should be available to the user. This feature requires that the RA be configured to use the standalone Akamai purger service.

Fixes #4031.
2019-02-14 14:47:42 -05:00
Roland Bracewell Shoemaker 232a5f828f Fix ineffectual assignments (#4052)
* in boulder-ra we connected to the publisher and created a publisher gRPC client twice for no apparent reason
* in the SA we ignored errors from `getChallenges` in `GetAuthorizations` which could result in a nil challenge being returned in an authorization
2019-02-13 15:39:58 -05:00
Daniel McCarney 1cf44c0546 RA: Exit early if CTLogGroups2 config is invalid. (#4025)
The `boulder-ra` component should fail to start if the `CTLogGroups2` configuration is empty, or if any of the configured log groups have no logs specified. This avoids more ambiguous errors down the road.

This PR also removes the deprecated `CTLogGroups` field from the RA struct. It isn't being used in any configurations.

Based on initial work in https://github.com/letsencrypt/boulder/pull/3990 by @michalmedvecky. Resolves #3941.
2019-01-22 13:14:26 -08:00
Roland Bracewell Shoemaker 196f019851 Add support for temporal CT logs (#3853)
Required a little bit of rework of the RA issuance flow (to add parsing of the precert to determine the expiration date, and moving final cert parsing before final cert submission) and RA tests, but I think it shouldn't create any issues...

Fixes #3197.
2018-09-14 16:14:42 -07:00
Daniel McCarney d39babdcf3
RA: Remove vestigial DNS config/setup. (#3854)
In db01b0b we removed email validation from the RA. This was the only
use of the `bdns` package by the RA and so we can go one step further
and delete the remaining setup, configuration and `bdns` fields.
2018-09-13 13:39:23 -04:00
Joel Sing f8a023e49c Remove various unnecessary uses of fmt.Sprintf (#3707)
Remove various unnecessary uses of fmt.Sprintf - in particular:

- Avoid calls like t.Error(fmt.Sprintf(...)), where t.Errorf can be used directly.

- Use strconv when converting an integer to a string, rather than using
  fmt.Sprintf("%d", ...). This is simpler and can also detect type errors at
  compile time.

- Instead of using x.Write([]byte(fmt.Sprintf(...))), use fmt.Fprintf(x, ...).
2018-05-11 11:55:25 -07:00
Daniel McCarney aa810a3142 gRPC: publish RPC latency stat in server interceptor. (#3665)
We may see RPCs that are dispatched by a client but do not arrive at the server for some time afterwards. To have insight into potential request latency at this layer we want to publish the time delta between when a client sent an RPC and when the server received it.

This PR updates the gRPC client interceptor to add the current time to the gRPC request metadata context when it dispatches an RPC. The server side interceptor is updated to pull the client request time out of the gRPC request metadata. Using this timestamp it can calculate the latency and publish it as an observation on a Prometheus histogram.

Accomplishing the above required wiring a clock through to each of the client interceptors. This caused a small diff across each of the gRPC aware boulder commands.

A small unit test is included in this PR that checks that a latency stat is published to the histogram after an RPC to a test ChillerServer is made. It's difficult to do more in-depth testing because using fake clocks makes the latency 0 and using real clocks requires finding a way to queue/delay requests inside of the gRPC mechanisms not exposed to Boulder.

Updates https://github.com/letsencrypt/boulder/issues/3635 - Still TODO: Explicitly logging latency in the VA, tracking outstanding RPCs as a gauge.
2018-04-25 15:37:22 -07:00
Jacob Hoffman-Andrews 4dcbf5c883
Run multiples of services in integration tests (#3662)
Fixes #3653.
2018-04-24 16:00:40 -07:00
Roland Bracewell Shoemaker 24cd01d033 Revert to setting full addresses instead of just ports 2018-04-23 12:39:28 -07:00
Roland Bracewell Shoemaker 5c4eaf841f Review fixes 2018-04-20 16:03:55 -07:00
Roland Bracewell Shoemaker ccb02419c5 Revert client changes + addr debug override 2018-04-20 12:46:33 -07:00
Roland Bracewell Shoemaker d424d0580b Allow cli override of gRPC listen and service addresses 2018-04-20 12:35:12 -07:00
Roland Bracewell Shoemaker 0e6713e573 Randomize order of CT logs when submitting precerts (#3660)
* Randomize order of CT logs when submitting precerts so we maximize the chances we actually exercise all of the logs in a group and not just the first in the list.

* Add metrics for winning logs
2018-04-20 15:00:10 -04:00
Daniel McCarney 74d5decc67 Remove `TotalCertificates` rate limit. (#3638)
The `TotalCertificates` rate limit serves to ensure we don't
accidentally exceed our OCSP signing capacity by issuing too many
certificates within a fixed period. In practice this rate limit has been
fragile and the associated queries have been linked to performance
problems.

Since we now have better means of monitoring our OCSP signing capacity
this commit removes the rate limit and associated code.
2018-04-12 13:25:47 -07:00
Daniel McCarney 299e53b237 RA,CA: Refuse to start with MaxNames == 0. (#3634)
This commit updates the `boulder-ra` and `boulder-ca` commands to refuse
to start if their configured `MaxNames` is 0 (the default value). This
should always be set to a positive number.

This commit also updates `csr/csr.go` to always apply the max names
check since it will never be 0 after the change above.

Also refactor `FailOnError` to pull out a separate `Fail` function.

Related to https://github.com/letsencrypt/boulder/issues/3632
2018-04-10 10:53:23 -07:00
Roland Bracewell Shoemaker cc5ec34539 Allow configuration of multiple DNS resolvers (#3612)
* Allow configuration of multiple DNS resolvers
* Use multiple DNS resolvers in integration tests

Fixes #3611.
2018-04-05 11:51:22 -04:00
Jacob Hoffman-Andrews 9da5a7e1fc Cleanup: TLS and GRPC configs are mandatory. (#3476)
Our various main.go functions gated some key code on whether the TLS
and/or GRPC config fields were present. Now that those fields are fully
deployed in production, we can simplify the code and require them.
    
Also, rename tls to tlsConfig everywhere to avoid confusion with the tls
package.
    
Avoid assigning to the same err from two different goroutines in
boulder-ca (fix a race).
2018-02-26 10:16:50 -05:00
Roland Bracewell Shoemaker 0b53063a72 ctpolicy: Add informational logs and don't cancel remaining submissions (#3472)
Add a set of logs which will be submitted to but not relied on for their SCTs,
this allows us to test submissions to a particular log or submit to a log which
is not yet approved by a browser/root program.

Also add a feature which stops cancellations of remaining submissions when racing
to get a SCT from a group of logs.

Additionally add an informational log that always times out in config-next.

Fixes #3464 and fixes #3465.
2018-02-23 21:51:50 -05:00
Roland Bracewell Shoemaker 9e23edf850 Use ctpolicy package in RA (#3422)
And collect the metrics on success/failure rates. Built on top of #3414.

Fixes #3413.
2018-02-08 13:33:42 -08:00
Maciej Dębski 44984cd84a Implement regID whitelist for allowed challenge types. (#3352)
This updates the PA component to allow authorization challenge types that are globally disabled if the account ID owning the authorization is on a configured whitelist for that challenge type.
2018-01-10 13:44:53 -05:00
Jacob Hoffman-Andrews 68d5cc3331
Restore gRPC metrics (#3265)
The go-grpc-prometheus package by default registers its metrics with Prometheus' global registry. In #3167, when we stopped using the global registry, we accidentally lost our gRPC metrics. This change adds them back.

Specifically, it adds two convenience functions, one for clients and one for servers, that makes the necessary metrics object and registers it. We run these in the main function of each server.

I considered adding these as part of StatsAndLogging, but the corresponding ClientMetrics and ServerMetrics objects (defined by go-grpc-prometheus) need to be subsequently made available during construction of the gRPC clients and servers. We could add them as fields on Scope, but this seemed like a little too much tight coupling.

Also, update go-grpc-prometheus to get the necessary methods.

```
$ go test github.com/grpc-ecosystem/go-grpc-prometheus/...
ok      github.com/grpc-ecosystem/go-grpc-prometheus    0.069s
?       github.com/grpc-ecosystem/go-grpc-prometheus/examples/testproto [no test files]
```
2017-12-07 15:44:55 -08:00
Roland Bracewell Shoemaker bdea281ae0 Remove CAA SERVFAIL exceptions code (#3262)
Fixes #3080.
2017-12-05 14:39:37 -08:00
Jacob Hoffman-Andrews f366e45756 Remove global state from metrics gathering (#3167)
Previously, we used prometheus.DefaultRegisterer to register our stats, which uses global state to export its HTTP stats. We also used net/http/pprof's behavior of registering to the default global HTTP ServeMux, via DebugServer, which starts an HTTP server that uses that global ServeMux.

In this change, I merge DebugServer's functions into StatsAndLogging. StatsAndLogging now takes an address parameter and fires off an HTTP server in a goroutine. That HTTP server is newly defined, and doesn't use DefaultServeMux. On it is registered the Prometheus stats handler, and handlers for the various pprof traces. In the process I split StatsAndLogging internally into two functions: makeStats and MakeLogger. I didn't port across the expvar variable exporting, which serves a similar function to Prometheus stats but which we never use.

One nice immediate effect of this change: Since StatsAndLogging now requires and address, I noticed a bunch of commands that called StatsAndLogging, and passed around the resulting Scope, but never made use of it because they didn't run a DebugServer. Under the old StatsD world, these command still could have exported their stats by pushing, but since we moved to Prometheus their stats stopped being collected. We haven't used any of these stats, so instead of adding debug ports to all short-lived commands, or setting up a push gateway, I simply removed them and switched those commands to initialize only a Logger, no stats.
2017-10-13 11:58:01 -07:00
Jacob Hoffman-Andrews 0a72f768a7 Remove ProfileCmd. (#3166)
These stats are now all collected by Prometheus.
2017-10-13 10:02:04 -04:00
Jacob Hoffman-Andrews 4128e0d95a Add time-dependent integration testing (#3060)
Fixes #3020.

In order to write integration tests for some features, especially related to rate limiting, rechecking of CAA, and expiration of authzs, orders, and certs, we need to be able to fake the passage of time in integration tests.

To do so, this change switches out all clock.Default() instances for cmd.Clock(), which can be set manually with the FAKECLOCK environment variable. integration-test.py now starts up all servers once before the main body of tests, with FAKECLOCK set to a date 70 days ago, and does some initial setup for a new integration test case. That test case tries to fetch a 70-day-old authz URL, and expects it to 404.

In order to make this work, I also had to change a number of our test binaries to shut down cleanly in response to SIGTERM. Without that change, stopping the servers between the setup phase and the main tests caused startservers.check() to fail, because some processes exited with nonzero status.

Note: This is an initial stab at things, to prove out the technique. Long-term, I think we will want to use an idiom where test cases are classes that have a number of optional setup phases that may be run at e.g. 70 days prior and 5 days prior. This could help us avoid a proliferation of global state as we add more time-dependent test cases.
2017-09-13 12:34:14 -07:00
Jacob Hoffman-Andrews 20ec1e3e4e Filter spurious shutdown errors. (#3052)
Previously, we would produce an error an a nonzero status code on shutdown,
because gRPC's GracefulStop would cause s.Serve() to return an error. Now we
filter that specific error and treat it as success. This also allows us to kill
process with SIGTERM instead of SIGKILL in integration tests.

Fixes #2410.
2017-09-07 13:45:32 -07:00
Jacob Hoffman-Andrews b0c7bc1bee Recheck CAA for authorizations older than 8 hours (#3014)
Fixes #2889.

VA now implements two gRPC services: VA and CAA. These both run on the same port, but this allows implementation of the IsCAAValid RPC to skip using the gRPC wrappers, and makes it easier to potentially separate the service into its own package in the future.

RA.NewCertificate now checks the expiration times of authorizations, and will call out to VA to recheck CAA for those authorizations that were not validated recently enough.
2017-08-28 16:40:57 -07:00
Roland Bracewell Shoemaker 90ba766af9 Add NewOrder RPCs + methods to SA and RA (#2907)
Fixes #2875, #2900 and #2901.
2017-08-11 14:24:25 -04:00
Roland Bracewell Shoemaker 05d869b005 Rename DNSResolver -> DNSClient (#2878)
Fixes #639.

This resolves something that has bugged me for two+ years, our DNSResolverImpl is not a DNS resolver, it is a DNS client. This change just makes that obvious.
2017-07-18 08:37:45 -04:00
Jacob Hoffman-Andrews 63a25bf913 Remove clientName everywhere. (#2862)
This used to be used for AMQP queue names. Now that AMQP is gone, these consts
were only used when printing a version string at startup. This changes
VersionString to just use the name of the current program, and removes
`const clientName = ` from many of our main.go's.
2017-07-12 10:28:54 -07:00
Roland Bracewell Shoemaker 8ce2f8b432 Basic RSA known weak key checking (#2765)
Adds a basic truncated modulus hash check for RSA keys that can be used to check keys against the Debian `{openssl,openssh,openvpn}-blacklist` lists of weak keys generated during the [Debian weak key incident](https://wiki.debian.org/SSLkeys).

Testing is gated on adding a new configuration key to the WFE, RA, and CA configs which contains the path to a directory which should contain the weak key lists.

Fixes #157.
2017-05-25 09:33:58 -07:00
Jacob Hoffman-Andrews b17b5c72a6 Remove statsd from Boulder (#2752)
This removes the config and code to output to statsd.

- Change `cmd.StatsAndLogging` to output a `Scope`, not a `Statter`.
- Remove the prefixing of component name (e.g. "VA") in front of stats; this was stripped by `autoProm` but now no longer needs to be.
- Delete vendored statsd client.
- Delete `MockStatter` (generated by gomock) and `mocks.Statter` (hand generated) in favor of mocking `metrics.Scope`, which is the interface we now use everywhere.
- Remove a few unused methods on `metrics.Scope`, and update its generated mock.
- Refactor `autoProm` and add `autoRegisterer`, which can be included in a `metrics.Scope`, avoiding global state. `autoProm` now registers everything with the `prometheus.Registerer` it is given.
- Change va_test.go's `setup()` to not return a stats object; instead the individual tests that care about stats override `va.stats` directly.

Fixes #2639, #2733.
2017-05-15 10:19:54 -04:00
Jacob Hoffman-Andrews 6719dc17a6 Remove AMQP config and code (#2634)
We now use gRPC everywhere.
2017-04-03 10:39:39 -04:00
Jacob Hoffman-Andrews 510e279208 Simplify gRPC TLS configs. (#2470)
Previously, a given binary would have three TLS config fields (CA cert, cert,
key) for its gRPC server, plus each of its configured gRPC clients. In typical
use, we expect all three of those to be the same across both servers and clients
within a given binary.

This change reuses the TLSConfig type already defined for use with AMQP, adds a
Load() convenience function that turns it into a *tls.Config, and configures it
for use with all of the binaries. This should make configuration easier and more
robust, since it more closely matches usage.

This change preserves temporary backwards-compatibility for the
ocsp-updater->publisher RPCs, since those are the only instances of gRPC
currently enabled in production.
2017-01-06 14:19:18 -08:00
Jacob Hoffman-Andrews 9b8dacab03 Split out separate RPC services for issuing and for signing OCSP (#2452)
This allows finer-grained control of which components can request issuance. The OCSP Updater should not be able to request issuance.

Also, update test/grpc-creds/generate.sh to reissue the certs properly.

Resolves #2417
2017-01-05 15:08:39 -08:00
Jacob Hoffman-Andrews 27a1446010 Move timeouts into client interceptor. (#2387)
Previously we had custom code in each gRPC wrapper to implement timeouts. Moving
the timeout code into the client interceptor allows us to simplify things and
reduce code duplication.
2016-12-05 10:42:26 -05:00
Roland Bracewell Shoemaker 03fdd65bfe Add gRPC server to SA (#2374)
Adds a gRPC server to the SA and SA gRPC Clients to the WFE, RA, CA, Publisher, OCSP updater, orphan finder, admin revoker, and expiration mailer.

Also adds a CA gRPC client to the OCSP Updater which was missed in #2193.

Fixes #2347.
2016-12-02 17:24:46 -08:00
Roland Bracewell Shoemaker a87379bc6e Add gRPC server to RA (#2350)
Fixes #2348.
2016-11-29 15:34:35 -08:00
Roland Bracewell Shoemaker 595204b23f Implement improved signal catching in services that already use it (#2333)
Implements a less RPC focused signal catch/shutdown method. Certain things that probably could also use this (i.e. `ocsp-updater`) haven't been given it as they would require rather substantial changes to allow for a graceful shutdown approach.

Fixes #2298.
2016-11-18 21:05:04 -05:00
Roland Bracewell Shoemaker c5f99453a9 Switch CT submission RPC from CA -> RA (#2304)
With the current gRPC design the CA talks directly to the Publisher when calling SubmitToCT which crosses security bounadries (secure internal segment -> internet facing segment) which is dangerous if (however unlikely) the Publisher is compromised and there is a gRPC exploit that allows memory corruption on the caller end of a RPC which could expose sensitive information or cause arbitrary issuance.

Instead we move the RPC call to the RA which is in a less sensitive network segment. Switching the call site from the CA -> RA is gated on adding the gRPC PublisherService object to the RA config.

Fixes #2202.
2016-11-08 11:39:02 -08:00
Jacob Hoffman-Andrews 32c03f942b Don't start DebugServer until server's ready. (#2271)
This makes availability of DebugServer a better proxy for readiness of the
component.
2016-10-21 16:57:14 -04:00
Jacob Hoffman-Andrews 1958dc9065 Update total issued count asynchronously. (#2246)
Previously the lock on total issued count would exacerbate problems when the
count query was slow, which it often is.

Fixes #1809.
2016-10-20 14:17:34 -07:00
Roland Bracewell Shoemaker 5fabc90a16 Add IDN support (#2215)
Add feature flagged support for issuing for IDNs, fixes #597.

This patch expects that clients have performed valid IDN2008 encoding on any label that includes unicode characters. Invalid encodings (including non-compatible IDN2003 encoding) will be rejected. No script-mixing or script exclusion checks are performed as we assume that if a name is resolvable that it conforms to the registrar's policies on these matters and if it uses non-standard scripts in sub-domains etc that browsers should be the ones choosing how to display those names.

Required a full update of the golang.org/x/net tree to pull in golang.org/x/net/idna, all test suites pass.
2016-10-06 13:05:37 -04:00
Roland Bracewell Shoemaker 7f0b7472e2 Add gRPC support to CA (#2193)
Fixes #2171.
2016-09-21 14:13:43 -07:00
Roland Bracewell Shoemaker 239bf9ae0a Very basic feature flag impl (#1705)
Updates #1699.

Adds a new package, `features`, which exposes methods to set and check if various internal features are enabled. The implementation uses global state to store the features so that services embedded in another service do not each require their own features map in order to check if something is enabled.

Requires a `boulder-tools` image update to include `golang.org/x/tools/cmd/stringer`.
2016-09-20 16:29:01 -07:00
Roland Bracewell Shoemaker e187c92715 Add gRPC client side metrics (#2151)
Fixes #1880.

Updates google.golang.org/grpc and github.com/jmhodges/clock, both test suites pass. A few of the gRPC interfaces changed so this also fixes those breakages.
2016-09-09 15:17:36 -04:00
Roland Bracewell Shoemaker c8f1fb3e2f Remove direct usages of go-statsd-client in favor of using metrics.Scope (#2136)
Fixes #2118, fixes #2082.
2016-09-07 19:35:13 -04:00
Ben Irving b587d4e663 Simplify KeyPolicy code (#2092)
This PR, removes the allowedSigningAlgos configuration struct and hard codes a key policy.

Fixes #1844
2016-07-30 16:15:19 -07:00
Patrick Figel 8cd74bf766 Make (pending)AuthorizationLifetime configurable (#2028)
Introduces the `authorizationLifetimeDays` and `pendingAuthorizationLifetimeDays` configuration options for `RA`.

If the values are missing from configuration, the code defaults back to the current values (300/7 days).

fixes #2024
2016-07-12 15:18:22 -04:00
Ben Irving 298774e1db Remove embedded (anonymous) fields from configs (#2019)
This PR removes the use of all anonymous struct fields that were introduced by myself as per my work on splitting up boulder-config (#1962).

The root of the bug was related to the loading of the json configuration file into the config struct. The config structs contained several embedded (anonymous) fields. An embedded (anonymous) field in a struct actually results in the flattening of the json structure. This caused json.Unmarshal to look not at the nested level, but at the root level of the json object and hence not find the nested field (i.e. AllowedSigningAlgos).

See https://play.golang.org/p/6uVCsEu3Df for a working example.

This fixes the reported bug: #2018
2016-07-07 10:16:41 -07:00
Ben Irving c4f7fb580d Split up boulder-config.json (RA) (#1974)
Part of #1962
2016-06-29 13:43:55 -07:00
Jacob Hoffman-Andrews 55657fad0d RA doesn't need CAASERVFAILExceptions. (#1992)
In #1971 we added the CAASERVFAILExceptions config field and argument to NewDNSResolverImpl. This argument only needs to be passed to the VA, where we do CAA validations. However, I accidentally added code to the RA as well to use this new config field. This changes backs that out.
2016-06-29 11:23:58 -07:00
Jacob Hoffman-Andrews 0c0e94dfaf Add enforcement for CAA SERVFAIL (#1971)
https://github.com/letsencrypt/boulder/pull/1971
2016-06-28 11:00:23 -07:00
Daniel McCarney 9abc212448 Reuse valid authz for subsequent new authz requests (#1921)
Presently clients may request a new AuthZ be created for a domain that they have already proved authorization over. This results in unnecessary bloat in the authorizations table and duplicated effort.

This commit alters the `NewAuthorization` function of the RA such that before going through the work of creating a new AuthZ it checks whether there already exists a valid AuthZ for the domain/regID that expires in more than 24 hours from the current date. If there is, then we short circuit creation and return the existing AuthZ. When this case occurs the `RA.ReusedValidAuthz` counter is incremented to provide visibility.

Since clients requesting a new AuthZ and getting an AuthZ back expect to turn around and post updates to the corresponding challenges we also return early in `UpdateAuthorization` when asked to update an AuthZ that is already valid. When this case occurs the `RA.ReusedValidAuthzChallenge` counter is incremented.

All of the above behaviour is gated by a new RA config flag `reuseValidAuthz`. In the default case (false) the RA does **not** reuse any AuthZ's and instead maintains the historic behaviour; always creating a new AuthZ when requested, irregardless of whether there are already valid AuthZ's that could be reused. In the true case (enabled only in `boulder-config-next.json`) the AuthZ reuse described above is enabled.

Resolves #1854
2016-06-10 16:44:16 -04:00
Ben Irving 438580f206 Remove last of UseNewVARPC (#1914)
`UseNewVARPC` is no longer necessary and is safe to be removed. We default to using the newer VA RPC code.
2016-06-09 10:12:46 -04:00
Daniel McCarney 4c289f2a8f Reload ratelimit policy automatically at runtime (#1894)
Resolves #1810 by automatically updating the RA ratelimit.RateLimitConfig whenever the backing config file is changed. Much like the Policy Authority uses a reloader instance to support updating the Hostname policy on the fly, this PR changes the Registration Authority to use a reloader for the rate limit policy file.

Access to the ra.rlPolicies member is protected with a RWMutex now that there is a potential for the values to be reloaded while a reader is active.

A test is introduced to ensure that writing a new policy YAML to the policy config file results in new values being set in the RA's rlPolicies instance.

https://github.com/letsencrypt/boulder/pull/1894
2016-06-08 12:11:46 -07:00
Ben Irving a713823cd5 Remove most instances of blog.Get() (#1870)
In this PR, logger is passed to the following callers:
  NewWebFrontEndImpl
  NewCertificateAuthorityImpl
  NewValidationAuthorityImpl
  NewAmqpRPCServer
  newUpdater
  NewRegistrationAuthorityServer

This reduces the usage of a global singleton logger and allows tests to consistently use a mock logger.

Fixes #1642

* remove blog.Get() in wfe
* remove blog.Get() from va
* remove Blog.Get() from ca
* remove blog.Get() from oscp updater, ampq rpc server, registration authority server
* removed some pointless logging code
* remove one added newline
* fix format issue
* fix setup function to return *blog.Mock instead of being passed in
* remove useless blog.NewMock() call
2016-06-02 14:26:39 -07:00
Jacob Hoffman-Andrews cf65d7d595 Improve VA and RA initialization (#1867)
- Run both gRPC and AMQP servers simultaneously
- Take explicit constructor parameters and unexport fields that were previously set by users
- Remove transitional DomainCheck code in RA now that GSB is enabled.
- Remove some leftover UpdateValidation dummy methods.
2016-06-02 11:57:37 -07:00
Kane York 0430c08388 Implement gRPC for VA's RPCs (#1738)
Adds appropriate protobufs, and sets up RA / VA to use the gRPC versions of the calls in boulder-config-next.json.

Fixes #1626.
2016-05-31 16:44:48 -07:00
Daniel McCarney 19635b2b6c The rate limiting code previously lived in the `cmd` package without a clear justification for why. This commit moves the rate limiting code to its own `ratelimit` package and updates import paths as required. Notably all references from the `cmd` package's exported `LoadRateLimitPolicies`, `RateLimitPolicy`, and `RateLimitConfig` were moved to use `ratelimit`.This removed the `cmd` import from a couple of callers (nice!).
The rate limiting code previously lived in the `cmd` package without a clear justification for why. This commit moves the rate limiting code to its own `ratelimit` package and updates import paths as required. Notably all references from the `cmd` package's exported `LoadRateLimitPolicies`, `RateLimitPolicy`, and `RateLimitConfig` were moved to use `ratelimit`. This removed the `cmd` import from a couple of callers (nice!).
2016-05-31 17:38:17 -04:00
Roland Bracewell Shoemaker 54573b36ba Remove all stray copyright headers and appends the initial line to LICENSE.txt (#1853) 2016-05-31 12:32:04 -07:00
Roland Bracewell Shoemaker 5abe7e3cdf Move CSR normalization/verification to their own methods (#1826)
* Split CSR testing and name hoisting into own functions, verify CSR in RA & CA

* Move tests around and various other fixes

* 1.5.3 doesn't have the needed stringer

* Move functions to their own lib

* Remove unused imports

* Move MaxCNLength and BadSignatureAlgorithms to csr package

* Always normalizeCSR in VerifyCSR and de-export it

* Update comments
2016-05-26 14:17:41 -07:00
Jacob Hoffman-Andrews 2c39c684cb Fix HostnamePolicyFile (#1785)
CA was referencing RA's config.
Also, the FailOnError function needs a non-nil error to fail.

* Bump Godep version.
2016-05-04 08:12:08 -07:00
Jacob Hoffman-Andrews e5e4fb744a * Delete Policy DB.This is no longer needed now that we have a JSON policy file.* Fix tests.* Revert Dockerfile.* Fix create_db* Simplify user addition.* Fix tests.* Fix tests* Review fixes.https://github.com/letsencrypt/boulder/pull/1773
* Delete Policy DB.

This is no longer needed now that we have a JSON policy file.

* Fix tests.
* Revert Dockerfile.
* Fix create_db
* Simplify user addition.
* Fix tests.
* Fix tests
* Review fixes.

https://github.com/letsencrypt/boulder/pull/1773
2016-04-29 12:12:24 -07:00
Jacob Hoffman-Andrews e6c17e1717 Switch to new vendor style (#1747)
* Switch to new vendor style.

* Fix metrics generate command.

* Fix miekg/dns types_generate.

* Use generated copies of files.

* Update miekg to latest.

Fixes a problem with `go generate`.

* Set GO15VENDOREXPERIMENT.

* Build in letsencrypt/boulder.

* fix travis more.

* Exclude vendor instead of godeps.

* Replace some ...

* Fix unformatted cmd

* Fix errcheck for vendorexp

* Add GO15VENDOREXPERIMENT to Makefile.

* Temp disable errcheck.

* Restore master fetch.

* Restore errcheck.

* Build with 1.6 also.

* Match statsd.*"

* Skip errcheck unles Go1.6.

* Add other ignorepkg.

* Fix errcheck.

* move errcheck

* Remove go1.6 requirement.

* Put godep-restore with errcheck.

* Remove go1.6 dep.

* Revert master fetch revert.

* Remove -r flag from godep save.

* Set GO15VENDOREXPERIMENT in Dockerfile and remove _worskpace.

* Fix Godep version.
2016-04-18 12:51:36 -07:00
Kane York 25b45a45ec Errcheck errors fixed (#1677)
* Fix all errcheck errors
* Add errcheck to test.sh
* Add a new sa.Rollback method to make handling errors in rollbacks easier.
This also causes a behavior change in the VA. If a HTTP connection is
abruptly closed after serving the headers for a non-200 response, the
reported error will be the read failure instead of the non-200.
2016-04-12 16:54:01 -07:00
Jacob Hoffman-Andrews ecc04e8e61 Refactor log package (#1717)
- Remove error signatures from log methods. This means fewer places where errcheck will show ignored errors.
- Pull in latest cfssl to be compatible with errorless log messages.
- Reduce the number of message priorities we support to just those we actually use.
- AuditNotice -> AuditInfo
- Remove InfoObject (only one use, switched to Info)
- Remove EmergencyExit and related functions in favor of panic
- Remove SyslogWriter / AuditLogger separate types in favor of a single interface, Logger, that has all the logging methods on it.
- Merge mock log into logger. This allows us to unexport the internals but still override them in the mock.
- Shorten names to be compatible with Go style: New, Set, Get, Logger, NewMock, etc.
- Use a shorter log format for stdout logs.
- Remove "... Starting" log messages. We have better information in the "Versions" message logged at startup.

Motivation: The AuditLogger / SyslogWriter distinction was confusing and exposed internals only necessary for tests. Some components accepted one type and some accepted the other. This made it hard to consistently use mock loggers in tests. Also, the unnecessarily fat interface for AuditLogger made it hard to meaningfully mock out.
2016-04-08 16:12:20 -07:00
Roland Bracewell Shoemaker 800b5b0cbf Switch to using a wrapped statter that provides PID
* Switch to using a wrapped statter that provides PID

* Fix tests and change some types to interfaces

* Add hostname to suffix + update comment
2016-04-01 15:43:35 -07:00
Kane York 6024176ee0 Add calls to PerformValidation in the RA 2016-03-23 10:39:23 -07:00
Jacob Hoffman-Andrews bc28bfe906 Implement reloadable JSON blacklist.
This eliminates the need the a database to store the hostname policy,
simplifying deployment. We keep the database for now, as part of our
deployability guidelines: we'll deploy, then switch config to the new style.

This also disables the obsolete whitelist checking code, but doesn't yet change
the function signature for policy.New(), to avoid bloating the pull request.
I'll fully remove the whitelist checking code in a future change when I also
remove the policy database code.
2016-03-13 20:00:51 -07:00
Jessica Frazelle 7955e268c7
go lint fixes
Signed-off-by: Jessica Frazelle <acidburn@docker.com>
2016-02-17 11:36:06 -08:00
Hugo Landau f218e314f8 Add good key testing for ECDSA. 2016-01-07 22:48:38 +00:00
Jeff Hodges 116ce96326 add retries and context deadlines to DNSResolver
This provides a means to add retries to DNS look ups, and, with some
future work, end retries early if our request deadline is blown. That
future work is tagged with #1292.

Updates #1258
2016-01-04 14:59:10 -08:00
Jeff Hodges e36895c9c5 bring RTT metrics inside DNSResolver
This moves the RTT metrics calculation inside of the DNSResolver. This
cleans up code in the RA and VA and makes some adding retries to the
DNSResolver less ugly to do.

Note: this will put `Rate` and `RTT` after the name of DNS query
type (`A`, `MX`, etc.). I think that's fine and desirable. We aren't
using this data in alerts or many dashboards, yet, so a flag day is
okay.

Fixes #1124
2015-12-16 17:41:42 -08:00
Jeff Hodges b31165444f move dns code to dns pkg and rename to bdns
Moves the DNS code from core to dns and renames the dns package to bdns
to be clearer.

Fixes #1260 and will be good to have while we add retries and such.
2015-12-14 11:21:43 -08:00
Jeff Hodges c9010744b4 delete old challenge code
Specifically, delete the simpleHttp and dvsni.

Hooray!

Fixes #894
2015-12-10 15:41:40 -08:00
Jacob Hoffman-Andrews b71a850501 Fix DBConfig references. 2015-11-24 16:41:53 -08:00
Jacob Hoffman-Andrews 608a0c9073 Move file reading to DBConfig and out of SA.
Also remove AMQP secret changes: one thing at a time.
2015-11-23 22:09:54 -08:00
Jacob Hoffman-Andrews 5dd212dd47 Add ConfigSecret.
This allows secret values to be stored externally to the config file, so that
config files can be easily shared without revealing secret data.
2015-11-19 22:41:27 -08:00
Jacob Hoffman-Andrews 5fb7be64b0 Make ServiceQueue a separate config param.
Also, make clientName strings into constants.
2015-11-18 17:40:45 -08:00
Jacob Hoffman-Andrews 7dcfcd7864 Add configurable RPC timeouts per backend.
In the process, break out AMQP config into its own struct, one per service.
The AMQPConfig struct is included by composition in the config structs that need
it. If any given service lacks an AMQP config of its own, it gets a default
value from the top-level AMQP config struct, for deployability reasons.

Tightens the RPC code to take a specific AMQP config, not an over-broad
cmd.Config.

Shortens construction of specific RPC clients so they instatiate the generic
client connection themselves, simplifying per-service startup code.

Remove unused SetTimeout method on RPC clients.
2015-11-17 19:51:51 -08:00
Jacob Hoffman-Andrews 2fc0f3143e Improve logging.
Consolidate initialization of stats and logging from each main.go into cmd
package.

Define a new config parameter, `StdoutLevel`, that determines the maximum log
level that will be printed to stdout. It can be set to 6 to inhibit debug
messages, or 0 to print only emergency messages, or -1 to print no messages at
all.

Remove the existing config parameter `Tag`. Instead, choose the tag from the
basename of the currently running process. Previously all Boulder log messages
had the tag "boulder", but now they will be differentiated by process, like
"boulder-wfe".

Shorten the date format used in stdout logging, and add the current binary's
basename.

Consolidate setup function in audit-logger_test.go.

Note: Most CLI binaries now get their stats and logging from the parameters of
Action. However, a few of our binaries don't use our custom AppShell, and
instead use codegangsta/cli directly. For those binaries, we export the new
StatsAndLogging method from cmd.

Fixes https://github.com/letsencrypt/boulder/issues/852
2015-11-11 16:52:42 -08:00
Richard Barnes 174011f6d8 Move validation and defaults out of UnmarshalJSON 2015-11-09 15:30:13 -05:00