After the prev. cleanup of legacy authz1 bits the `authzLookupFunc`
interface and the associated `handleAuthorization` function are only
used in one place for handling authz2 resources. This commit cleans
this now unneeded abstraction up (and also removes the "V2" suffix
from the challenge and authz handlers).
The `codespell` tool will be run during the "lints" phase of `test.sh`.
See `.codespell.ignore.txt for ignored words. Note that these ignored
words should be listed one per-line, in **lowercase** form.
The boulder-tools `build.sh` script is updated to include `codespell` in
the tools image. I built and pushed new images with this script that are
ref'd by `docker-compose.yml`.
Resolves#4635
In 0804e97 we updated `github.com/go-sql-driver/mysql` to a pinned
commit (b4242bab7dc5) newer than the latest tagged release (v1.4.1) to
avoid needing to pull in an extra dep. that was removed since v1.4.1.
Unfortunately for reasons that are not perfectly clear updating
`github.com/google/certificate-transparency-go` is preferring v1.4.1
over the pseudo-version made from the commit newer than v1.4.1 that we
previously pinned.
Since there is movement on making a v1.5.0 go-sql-driver mysql release
tag and we can likely get ct-go to use that we'll temporarily accept
this downgrade to update ct-go.
Unit tests are confirmed to pass:
```
~/go/src/github.com/go-sql-driver/mysql$ git log --pretty=format:'%h' -n 1
72cd26f
~/go/src/github.com/go-sql-driver/mysql$ go test ./...
ok github.com/go-sql-driver/mysql 0.081s
```
Note: This dep bump introduces a harmless, but annoying, error log
to our service startup output of the form:
> E203318 boulder-ra 2PvBvwg [AUDIT] ccResolverWrapper: error parsing service config: no JSON service config provided
We previously addressed this with the upstream project
(30f4150eec)
but the problem has returned. Filed https://github.com/letsencrypt/boulder/issues/4628
as a follow-up to chase this down.
Unit tests are confirmed to pass:
```
~/go/src/google.golang.org/grpc$ git log --pretty=format:'%h' -n 1
1a3960e
~/go/src/google.golang.org/grpc$ go test ./...
ok google.golang.org/grpc 18.163s
? google.golang.org/grpc/backoff [no test files]
? google.golang.org/grpc/balancer [no test files]
? google.golang.org/grpc/balancer/base [no test files]
ok google.golang.org/grpc/balancer/grpclb 15.491s
? google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 [no test files]
ok google.golang.org/grpc/balancer/roundrobin 0.349s
? google.golang.org/grpc/balancer/weightedroundrobin [no test files]
? google.golang.org/grpc/benchmark [no test files]
? google.golang.org/grpc/benchmark/benchmain [no test files]
? google.golang.org/grpc/benchmark/benchresult [no test files]
? google.golang.org/grpc/benchmark/client [no test files]
ok google.golang.org/grpc/benchmark/flags 0.001s
? google.golang.org/grpc/benchmark/grpc_testing [no test files]
ok google.golang.org/grpc/benchmark/latency 1.005s
ok google.golang.org/grpc/benchmark/primitives 0.001s [no tests to run]
? google.golang.org/grpc/benchmark/server [no test files]
? google.golang.org/grpc/benchmark/stats [no test files]
? google.golang.org/grpc/benchmark/worker [no test files]
? google.golang.org/grpc/binarylog/grpc_binarylog_v1 [no test files]
? google.golang.org/grpc/channelz/grpc_channelz_v1 [no test files]
ok google.golang.org/grpc/channelz/service 0.009s
ok google.golang.org/grpc/codes 0.002s
? google.golang.org/grpc/connectivity [no test files]
ok google.golang.org/grpc/credentials 0.017s
ok google.golang.org/grpc/credentials/alts 0.003s
? google.golang.org/grpc/credentials/alts/internal [no test files]
ok google.golang.org/grpc/credentials/alts/internal/authinfo 0.003s
ok google.golang.org/grpc/credentials/alts/internal/conn 0.079s
ok google.golang.org/grpc/credentials/alts/internal/handshaker 0.039s
ok google.golang.org/grpc/credentials/alts/internal/handshaker/service 0.007s
? google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp [no test files]
? google.golang.org/grpc/credentials/alts/internal/testutil [no test files]
? google.golang.org/grpc/credentials/google [no test files]
ok google.golang.org/grpc/credentials/internal 0.005s
? google.golang.org/grpc/credentials/oauth [no test files]
? google.golang.org/grpc/encoding [no test files]
? google.golang.org/grpc/encoding/gzip [no test files]
ok google.golang.org/grpc/encoding/proto 0.025s
? google.golang.org/grpc/examples/features/authentication/client [no test files]
? google.golang.org/grpc/examples/features/authentication/server [no test files]
? google.golang.org/grpc/examples/features/cancellation/client [no test files]
? google.golang.org/grpc/examples/features/cancellation/server [no test files]
? google.golang.org/grpc/examples/features/compression/client [no test files]
? google.golang.org/grpc/examples/features/compression/server [no test files]
? google.golang.org/grpc/examples/features/deadline/client [no test files]
? google.golang.org/grpc/examples/features/deadline/server [no test files]
? google.golang.org/grpc/examples/features/debugging/client [no test files]
? google.golang.org/grpc/examples/features/debugging/server [no test files]
? google.golang.org/grpc/examples/features/encryption/ALTS/client [no test files]
? google.golang.org/grpc/examples/features/encryption/ALTS/server [no test files]
? google.golang.org/grpc/examples/features/encryption/TLS/client [no test files]
? google.golang.org/grpc/examples/features/encryption/TLS/server [no test files]
? google.golang.org/grpc/examples/features/errors/client [no test files]
? google.golang.org/grpc/examples/features/errors/server [no test files]
? google.golang.org/grpc/examples/features/interceptor/client [no test files]
? google.golang.org/grpc/examples/features/interceptor/server [no test files]
? google.golang.org/grpc/examples/features/keepalive/client [no test files]
? google.golang.org/grpc/examples/features/keepalive/server [no test files]
? google.golang.org/grpc/examples/features/load_balancing/client [no test files]
? google.golang.org/grpc/examples/features/load_balancing/server [no test files]
? google.golang.org/grpc/examples/features/metadata/client [no test files]
? google.golang.org/grpc/examples/features/metadata/server [no test files]
? google.golang.org/grpc/examples/features/multiplex/client [no test files]
? google.golang.org/grpc/examples/features/multiplex/server [no test files]
? google.golang.org/grpc/examples/features/name_resolving/client [no test files]
? google.golang.org/grpc/examples/features/name_resolving/server [no test files]
? google.golang.org/grpc/examples/features/proto [no test files]
? google.golang.org/grpc/examples/features/proto/echo [no test files]
? google.golang.org/grpc/examples/features/reflection/server [no test files]
? google.golang.org/grpc/examples/features/retry/client [no test files]
? google.golang.org/grpc/examples/features/retry/server [no test files]
? google.golang.org/grpc/examples/features/wait_for_ready [no test files]
? google.golang.org/grpc/examples/helloworld/greeter_client [no test files]
? google.golang.org/grpc/examples/helloworld/greeter_server [no test files]
? google.golang.org/grpc/examples/helloworld/helloworld [no test files]
ok google.golang.org/grpc/examples/helloworld/mock_helloworld 0.003s
? google.golang.org/grpc/examples/route_guide/client [no test files]
ok google.golang.org/grpc/examples/route_guide/mock_routeguide 0.005s
? google.golang.org/grpc/examples/route_guide/routeguide [no test files]
? google.golang.org/grpc/examples/route_guide/server [no test files]
ok google.golang.org/grpc/grpclog 0.003s
? google.golang.org/grpc/grpclog/glogger [no test files]
ok google.golang.org/grpc/health 0.063s
? google.golang.org/grpc/health/grpc_health_v1 [no test files]
? google.golang.org/grpc/internal [no test files]
? google.golang.org/grpc/internal/backoff [no test files]
? google.golang.org/grpc/internal/balancerload [no test files]
ok google.golang.org/grpc/internal/binarylog 0.026s
ok google.golang.org/grpc/internal/buffer 0.002s
ok google.golang.org/grpc/internal/cache 0.653s
ok google.golang.org/grpc/internal/channelz 0.005s
? google.golang.org/grpc/internal/envconfig [no test files]
? google.golang.org/grpc/internal/grpcrand [no test files]
ok google.golang.org/grpc/internal/grpcsync 0.002s
ok google.golang.org/grpc/internal/grpctest 0.002s
ok google.golang.org/grpc/internal/leakcheck 4.083s
ok google.golang.org/grpc/internal/proto/grpc_service_config 0.002s
ok google.golang.org/grpc/internal/resolver/dns 1.620s
? google.golang.org/grpc/internal/resolver/passthrough [no test files]
? google.golang.org/grpc/internal/syscall [no test files]
ok google.golang.org/grpc/internal/testutils 0.002s
ok google.golang.org/grpc/internal/transport 81.078s
ok google.golang.org/grpc/internal/wrr 0.008s
? google.golang.org/grpc/interop [no test files]
? google.golang.org/grpc/interop/alts/client [no test files]
? google.golang.org/grpc/interop/alts/server [no test files]
? google.golang.org/grpc/interop/client [no test files]
? google.golang.org/grpc/interop/fake_grpclb [no test files]
? google.golang.org/grpc/interop/grpc_testing [no test files]
? google.golang.org/grpc/interop/http2 [no test files]
? google.golang.org/grpc/interop/server [no test files]
? google.golang.org/grpc/keepalive [no test files]
ok google.golang.org/grpc/metadata 0.004s
ok google.golang.org/grpc/naming 0.156s
? google.golang.org/grpc/peer [no test files]
ok google.golang.org/grpc/reflection 0.010s
? google.golang.org/grpc/reflection/grpc_reflection_v1alpha [no test files]
? google.golang.org/grpc/reflection/grpc_testing [no test files]
? google.golang.org/grpc/reflection/grpc_testingv3 [no test files]
? google.golang.org/grpc/resolver [no test files]
? google.golang.org/grpc/resolver/dns [no test files]
? google.golang.org/grpc/resolver/manual [no test files]
? google.golang.org/grpc/resolver/passthrough [no test files]
? google.golang.org/grpc/serviceconfig [no test files]
ok google.golang.org/grpc/stats 0.046s
? google.golang.org/grpc/stats/grpc_testing [no test files]
ok google.golang.org/grpc/status 0.008s
? google.golang.org/grpc/stress/client [no test files]
? google.golang.org/grpc/stress/grpc_testing [no test files]
? google.golang.org/grpc/stress/metrics_client [no test files]
? google.golang.org/grpc/tap [no test files]
ok google.golang.org/grpc/test 30.190s
ok google.golang.org/grpc/test/bufconn 0.204s
? google.golang.org/grpc/test/codec_perf [no test files]
? google.golang.org/grpc/test/go_vet [no test files]
? google.golang.org/grpc/test/grpc_testing [no test files]
? google.golang.org/grpc/xds/experimental [no test files]
ok google.golang.org/grpc/xds/internal 0.003s
ok google.golang.org/grpc/xds/internal/balancer 5.113s
ok google.golang.org/grpc/xds/internal/balancer/edsbalancer 1.264s
ok google.golang.org/grpc/xds/internal/balancer/lrs 0.246s
ok google.golang.org/grpc/xds/internal/balancer/orca 0.002s
ok google.golang.org/grpc/xds/internal/client 0.004s
? google.golang.org/grpc/xds/internal/proto [no test files]
? google.golang.org/grpc/xds/internal/proto/udpa/data/orca/v1 [no test files]
? google.golang.org/grpc/xds/internal/proto/udpa/service/orca/v1 [no test files]
? google.golang.org/grpc/xds/internal/proto/udpa/type/v1 [no test files]
ok google.golang.org/grpc/xds/internal/resolver 0.004s
```
Updates https://github.com/letsencrypt/boulder/issues/4548
Unit tests are confirmed to pass:
```
~/go/src/github.com/miekg/pkcs11$ git log --pretty=format:'%h' -n 1
210dc1e
~/go/src/github.com/miekg/pkcs11$ go test ./...
ok github.com/miekg/pkcs11 0.645s
? github.com/miekg/pkcs11/p11 [no test files]
```
Unit tests are confirmed to pass:
```
~/go/src/golang.org/x/crypto$ git log --pretty=format:'%h' -n 1
e1110fd
~/go/src/golang.org/x/crypto$ go test ./...
ok golang.org/x/crypto/acme 6.879s
ok golang.org/x/crypto/acme/autocert 1.213s
? golang.org/x/crypto/acme/autocert/internal/acmetest [no test files]
? golang.org/x/crypto/acme/internal/acmeprobe [no test files]
ok golang.org/x/crypto/argon2 0.084s
ok golang.org/x/crypto/bcrypt 2.224s
ok golang.org/x/crypto/blake2b 0.049s
ok golang.org/x/crypto/blake2s 0.034s
ok golang.org/x/crypto/blowfish 0.005s
ok golang.org/x/crypto/bn256 0.311s
ok golang.org/x/crypto/cast5 2.527s
ok golang.org/x/crypto/chacha20 0.013s
ok golang.org/x/crypto/chacha20poly1305 0.423s
ok golang.org/x/crypto/cryptobyte 0.002s
? golang.org/x/crypto/cryptobyte/asn1 [no test files]
ok golang.org/x/crypto/curve25519 0.017s
ok golang.org/x/crypto/ed25519 0.047s
? golang.org/x/crypto/ed25519/internal/edwards25519 [no test files]
ok golang.org/x/crypto/hkdf 0.009s
ok golang.org/x/crypto/internal/subtle 0.011s
ok golang.org/x/crypto/md4 0.001s
ok golang.org/x/crypto/nacl/auth 4.920s
ok golang.org/x/crypto/nacl/box 0.019s
ok golang.org/x/crypto/nacl/secretbox 0.002s
ok golang.org/x/crypto/nacl/sign 0.002s
ok golang.org/x/crypto/ocsp 0.020s
ok golang.org/x/crypto/openpgp 3.302s
ok golang.org/x/crypto/openpgp/armor 0.001s
ok golang.org/x/crypto/openpgp/clearsign 13.182s
ok golang.org/x/crypto/openpgp/elgamal 0.008s
? golang.org/x/crypto/openpgp/errors [no test files]
ok golang.org/x/crypto/openpgp/packet 0.115s
ok golang.org/x/crypto/openpgp/s2k 5.114s
ok golang.org/x/crypto/otr 0.163s
ok golang.org/x/crypto/pbkdf2 0.025s
ok golang.org/x/crypto/pkcs12 0.036s
ok golang.org/x/crypto/pkcs12/internal/rc2 0.001s
ok golang.org/x/crypto/poly1305 0.025s
ok golang.org/x/crypto/ripemd160 0.018s
ok golang.org/x/crypto/salsa20 0.029s
ok golang.org/x/crypto/salsa20/salsa 0.009s
ok golang.org/x/crypto/scrypt 0.384s
ok golang.org/x/crypto/sha3 0.121s
ok golang.org/x/crypto/ssh 2.779s
ok golang.org/x/crypto/ssh/agent 0.460s
ok golang.org/x/crypto/ssh/knownhosts 0.018s
ok golang.org/x/crypto/ssh/terminal 0.006s
ok golang.org/x/crypto/ssh/test 2.059s
ok golang.org/x/crypto/tea 0.003s
ok golang.org/x/crypto/twofish 0.013s
ok golang.org/x/crypto/xtea 0.009s
ok golang.org/x/crypto/xts 0.001s
```
Unit tests are confirmed to pass:
```
~/go/src/golang.org/x/net$ git log --pretty=format:'%h' -n 1
2180aed
~/go/src/golang.org/x/net$ go test ./...
ok golang.org/x/net/bpf 0.494s
ok golang.org/x/net/context 0.058s
ok golang.org/x/net/context/ctxhttp 0.104s
? golang.org/x/net/dict [no test files]
ok golang.org/x/net/dns/dnsmessage 0.074s
ok golang.org/x/net/html 0.097s
ok golang.org/x/net/html/atom 0.002s
ok golang.org/x/net/html/charset 0.020s
ok golang.org/x/net/http/httpguts 0.028s
ok golang.org/x/net/http/httpproxy 0.003s
ok golang.org/x/net/http2 125.352s
ok golang.org/x/net/http2/h2c 0.015s
? golang.org/x/net/http2/h2i [no test files]
ok golang.org/x/net/http2/hpack 0.042s
ok golang.org/x/net/icmp 0.002s
ok golang.org/x/net/idna 0.012s
? golang.org/x/net/internal/iana [no test files]
ok golang.org/x/net/internal/socket 4.560s
ok golang.org/x/net/internal/socks 0.222s
ok golang.org/x/net/internal/sockstest 0.015s
ok golang.org/x/net/internal/timeseries 0.020s
ok golang.org/x/net/ipv4 0.053s
ok golang.org/x/net/ipv6 0.043s
ok golang.org/x/net/nettest 1.057s
ok golang.org/x/net/netutil 0.819s
ok golang.org/x/net/proxy 0.039s
ok golang.org/x/net/publicsuffix 0.146s
ok golang.org/x/net/trace 0.007s
ok golang.org/x/net/webdav 0.091s
ok golang.org/x/net/webdav/internal/xml 0.010s
ok golang.org/x/net/websocket 0.026s
ok golang.org/x/net/xsrftoken 0.019s
```
Unit tests are confirmed to pass:
```
~/go/src/gopkg.in/yaml.v2$ git log --pretty=format:'%h' -n 1
f90ceb4
~/go/src/gopkg.in/yaml.v2$ go test ./...
ok gopkg.in/yaml.v2 2.873s
```
Unit tests confirmed to pass:
```
~/go/src/github.com/golang/mock$ git log --pretty=format:'%h' -n 1
d74b935
~/go/src/github.com/golang/mock$ go test ./...
go: downloading golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: extracting golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: finding golang.org/x/tools v0.0.0-20190425150028-36563e24a262
ok github.com/golang/mock/gomock 0.003s
? github.com/golang/mock/gomock/internal/mock_gomock [no test files]
ok github.com/golang/mock/mockgen 0.008s
ok github.com/golang/mock/mockgen/internal/tests/aux_imports_embedded_interface 0.002s
? github.com/golang/mock/mockgen/internal/tests/aux_imports_embedded_interface/faux [no test files]
? github.com/golang/mock/mockgen/internal/tests/copyright_file [no test files]
? github.com/golang/mock/mockgen/internal/tests/custom_package_name/client/v1 [no test files]
ok github.com/golang/mock/mockgen/internal/tests/custom_package_name/greeter 0.003s
? github.com/golang/mock/mockgen/internal/tests/custom_package_name/validator [no test files]
? github.com/golang/mock/mockgen/internal/tests/dot_imports [no test files]
? github.com/golang/mock/mockgen/internal/tests/empty_interface [no test files]
ok github.com/golang/mock/mockgen/internal/tests/generated_identifier_conflict 0.006s
? github.com/golang/mock/mockgen/internal/tests/import_source [no test files]
? github.com/golang/mock/mockgen/internal/tests/import_source/definition [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg/reflect_output [no test files]
? github.com/golang/mock/mockgen/internal/tests/internal_pkg/subdir/internal/pkg/source_output [no test files]
ok github.com/golang/mock/mockgen/internal/tests/mock_in_test_package 0.045s [no tests to run]
ok github.com/golang/mock/mockgen/internal/tests/test_package 0.002s [no tests to run]
ok github.com/golang/mock/mockgen/internal/tests/unexported_method 0.002s
? github.com/golang/mock/mockgen/internal/tests/vendor_dep [no test files]
? github.com/golang/mock/mockgen/internal/tests/vendor_dep/source_mock_package [no test files]
? github.com/golang/mock/mockgen/internal/tests/vendor_pkg [no test files]
ok github.com/golang/mock/mockgen/model 0.007s
ok github.com/golang/mock/sample 0.003s
ok github.com/golang/mock/sample/concurrent 0.002s
? github.com/golang/mock/sample/concurrent/mock [no test files]
? github.com/golang/mock/sample/imp1 [no test files]
? github.com/golang/mock/sample/imp2 [no test files]
? github.com/golang/mock/sample/imp3 [no test files]
? github.com/golang/mock/sample/imp4 [no test files]
? github.com/golang/mock/sample/mock_user [no test files]
```
The RA should set the expiry of valid authorizations based only on the current time and the configured authorizationLifetime. It should not extend the pending authorization's lifetime by the authorizationLifetime.
Resolves#4617
I didn't gate this with a feature flag. If we think this needs an API announcement and gradual rollout (I don't personally think this change deserves that) then I think we should change the RA config's authorizationLifetimeDays value to 37 days instead of adding a feature flag that we'll have to clean up after the flag date. We can change it back to 30 after the flag date.
In a handful of places I've nuked old stats which are not used in any alerts or dashboards as they either duplicate other stats or don't provide much insight/have never actually been used. If we feel like we need them again in the future it's trivial to add them back.
There aren't many dashboards that rely on old statsd style metrics, but a few will need to be updated when this change is deployed. There are also a few cases where prometheus labels have been changed from camel to snake case, dashboards that use these will also need to be updated. As far as I can tell no alerts are impacted by this change.
Fixes#4591.
Fixes an issue introduced in #4573 that could cause the CA orphan
queue to spin endlessly.
The bug introduced in #4573 was that while the precertificate insertion
and other operations were using a transaction in AddPrecertificate,
the certificate status insertion wasn't. This meant that if the
certificate status call succeeded but one of the preceding operations
didn't, all of the other insertions would be rolled back, but the
certificate status insertion wouldn't. This would cause a scenario
where a certificate status row existed for a precertificate that didn't
have a matching row in the precertificates table. Any preceding call to
AddPrecertificate would then fail on the certificate status insertion,
as there was already an existing duplicate row, which would prevent
any of the other insertions in the transactions from being applied.
This change also refactors the duplicate error check in ca/ca.go as
it is unclear from the error message it causes which of the two RPCs
(AddPrecertificate or AddCertificate) failed.
Incorporates square/go-jose#282.
$ go test gopkg.in/square/go-jose.v2
go: finding gopkg.in/square/go-jose.v2 v2.4.1
ok gopkg.in/square/go-jose.v2 46.790s
* deps: update publicsuffix-go to 342bab7
This updates `github.com/weppos/publicsuffix-go` to 342bab7, the tip of
master at the time of writing.
Unit tests are confirmed to pass:
```
~/go/src/github.com/weppos/publicsuffix-go$ git log --pretty=format:'%h' -n 1
342bab7
~/go/src/github.com/weppos/publicsuffix-go$ go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.023s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.015s
? github.com/weppos/publicsuffix-go/publicsuffix/generator [no test files]
```
* deps: update zmap/zlint to 71201e7
This updates `github.com/zmap/zlint` to 71201e7, the tip of master at
the time of writing.
Unit tests are confirmed to pass:
```
~/go/src/github.com/zmap/zlint$ git log --pretty=format:'%h' -n 1
71201e7
~/go/src/github.com/zmap/zlint$ go test ./...
ok github.com/zmap/zlint 0.205s
? github.com/zmap/zlint/cmd/zlint [no test files]
? github.com/zmap/zlint/cmd/zlint-gtld-update [no test files]
ok github.com/zmap/zlint/lints 0.214s
ok github.com/zmap/zlint/util 0.014s
```
This is a breaking API change: pkcs11key now takes as input a public key rather than
a private key label. In order to find the private key, it first finds the public key's CKA_ID
in the token, then looks for a private key with the same CKA_ID. From ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-30/pkcs-11v2-30b-d6.pdf:
> The CKA_ID field is intended to distinguish among multiple keys. In
the case of public and private keys, this field assists in handling
multiple keys held by the same subject; the key identifier for a
public key and its corresponding private key should be the same.
This does require that both the public key and private key are present and have
appropriate CKA_IDs set. I've verified this is the case in prod. In our integration
testing environment it was not the case, so I've tweaked entrypoint.sh to load
public keys into SoftHSM and set their CKA_ID.
The initial part of this change was written by @cpu. I've reviewed and approved
those commits.
* cmd: update prometheus.NewProcessCollector args.
There's a new struct `prometheus.ProcessCollectorOpts` that is expected
to be used as the sole argument to `prometheus.NewProcessCollector`. We
don't need to specify `os.Getpid` as the `PidFn` of the struct because
the default is to assume `os.Getpid`. Similarly we don't need to set the
namespace to `""` explicitly, it is the default.
* SA: reimplement db metrics as custom collector.
The modern Prometheus golang API supports translating between legacy
metric sources on the fly with a custom collector. We can use this
approach to collect the metrics from `gorp.DbMap`'s via the `sql.DB`
type's `Stats` function and the returned `sql.DbStats` struct.
This is a cleaner solution overall (we can lose the DB metrics updating
go routine) and it avoids the need to use the now-removed `Set` method
of the `prometheus.Counter` type.
* test: Update CountHistogramSamples.
The `With` function of `prometheus.HistogramVec` types we tend to use as
the argument to `test.CountHistogramSamples` changed to return
a `prometheus.Observer`. Since we only use this function in test
contexts, and only with things that cast back to
a `prometheus.Histogram` we take that approach to fix the problem
without updating call-sites.
New types and related infrastructure are added to the `db` package to allow
wrapping gorp DbMaps and Transactions.
The wrapped versions return a special `db.ErrDatabaseOp` error type when errors
occur. The new error type includes additional information such as the operation
that failed and the related table.
Where possible we determine the table based on the types of the gorp function
arguments. Where that isn't possible (e.g. with raw SQL queries) we try to use
a simple regexp approach to find the table name. This isn't great for general
SQL but works well enough for Boulder's existing SQL queries.
To get additional confidence my regexps work for all of Boulder's queries
I temporarily changed the `db` package's `tableFromQuery` function to panic if
the table couldn't be determined. I re-ran the full unit and integration test
suites with this configuration and saw no panics.
Resolves https://github.com/letsencrypt/boulder/issues/4559
The AddCertificate processing related to updating the fqdnSets and certificatesPerNames tables can be done in a separate transaction from the inserts to issuedNames and certificates.
This has the advantage of letting the overall AddCertificate request succeed when the primary transaction succeeds but the rate limit update transaction fails. We are OK with slightly incorrect rate limit results if it means more AddCertificate requests succeed and there are fewer orphaned final certificates.
To maintain visibility we audit log when the rate limit transaction fails and also increment a new failedAddCertRLTransactions prometheus counter.
Resolves#4566
In Python3, the output of subprocess.check_output is of type bytes.
That means calling print() on the output will print \n instead of an
actual newline. This PR adds decoding to the output of mysql in the slow
query test, bringing it into line with other check_output calls.
This also removes a redundant "def run" that is shadowed by the
definition in helpers.py (and was also missing a decode() call).
* SA: add unit test for auto_increment schemas.
`TestAutoIncrementSchema` uses a root user connection to the
`information_schema` MariaDB database to try and find table columns from
the Boulder schemas that are both `auto_increment` and not `int64`.
* SA: rename _db-next RemoveOCSPResponses.sql migration.
Based on the order that we apply migrations the
`RemoveOCSPResponses.sql` migration with its old prefix
(`20181101105733`) was never being applied. That in turn caused the new
`TestAutoIncrementSchema` unit test to fail because the old
`ocspResponses` table has an `id` field that is `auto_increment` but
`sized `int(11)`.
Renaming the migration with a newer prefix solves the problem. The
`ocspResponses` table ends up dropped when `config-next` is used.
Afterwards the `TestAutoIncrementSchema` unit test passes again.
Python 2 is over in 1 month 4 days: https://pythonclock.org/
This rolls forward most of the changes in #4313.
The original change was rolled back in #4323 because it
broke `docker-compose up`. This change fixes those original issues by
(a) making sure `requests` is installed and (b) sourcing a virtualenv
containing the `requests` module before running start.py.
Other notable changes in this:
- Certbot has changed the developer instructions to install specific packages
rather than rely on `letsencrypt-auto --os-packages-only`, so we follow suit.
- Python3 now has a `bytes` type that is used in some places that used to
provide `str`, and all `str` are now Unicode. That means going from `bytes` to
`str` and back requires explicit `.decode()` and `.encode()`.
- Moved from urllib2 to requests in many places.
We intentionally use a SLEEP in a SQL query to trigger timeout behavior.
This caused integration tests failures locally (where unittests are run
in the same session as integration tests).
Prev. we inserted data for tracking issued names into the `issuedNames` table
during `sa.AddCertificate`. A more robust solution is to do this during
`sa.AddPrecertificate` since this is when we've truly committed to having
issued for the names.
The new SA `WriteIssuedNamesPrecert` feature flag enables writing this table
during `AddPrecertificate`. The legacy behaviour continues with the flag
enabled or disabled but is updated to tolerate duplicate INSERT errors so that
it is possible to deploy this change across multiple SA instances safely.
Along the way I also updated `SA.AddPrecertificate` to perform its two
`INSERT`s in a transaction using the `db.WithTransaction` wrapper.
Resolves https://github.com/letsencrypt/boulder/issues/4565
In the process I tweaked a few variable names in GetAuthorizations2 to
refer to just "authz" instead of "authz2" because it made things
clearer, particularly in the case of authz2IDMap, which is a map of
whether a given ID exists, not a map from authz's to IDs.
Fixes#4564
In the deep dark history of Boulder we ended up jamming contacts into
a VARCHAR db field. We need to make sure that when contacts are
marshaled the resulting bytes will fit into the column or a 500 will
be returned to the user when the SA RPC fails.
One day we should fix this properly and not return a hacky error message
that's hard for users to understand. Unfortunately that will likely
require a migration or a new DB table. In the shorter term this hack
will prevent 500s which is a clear improvement.
Prev. we weren't checking the domain portion of an email contact address
very strictly in the RA. This updates the PA to export a function that
can be used to validate the domain the same way we validate domain
portions of DNS type identifiers for issuance.
This also changes the RA to use the `invalidEmail` error type in more
places.
A new Go integration test is added that checks these errors end-to-end
for both account creation and account update.
We need the RA's `NewOrder` RPC to return a `berrors.Malformed` instance
when there are too many identifiers. A bare error will be turned into
a server internal problem by the WFE2's `web.ProblemDetailsForError`
call while a `berrors.Malformed` will produce the expected malformed
problem.
This commit fixes the err, updates the unit test, and adds an end-to-end
integration test so we don't mess this up again.
This updates the `github.com/eggsampler/acme` dependency used in our Go-based
integration tests to v3. Notably this fixes a data race we encountered in CI.
With the data race fixed this branch can also revert
54a798b7f6 and resolve
https://github.com/letsencrypt/boulder/issues/4542
I ran a `go mod tidy` to cleanup the old `v2` copy of the dep and it also
removed a few stale cfssl/mysql items from the `go.mod`.
Upstream library's tests are confirmed to pass:
```
~/go/src/github.com/eggsampler/acme$ git log --pretty=format:'%h' -n 1
b581dc6
~/go/src/github.com/eggsampler/acme$ make pebble
mkdir -p /home/daniel/go/src/github.com/letsencrypt/pebble
git clone --depth 1 https://github.com/letsencrypt/pebble.git /home/daniel/go/src/github.com/letsencrypt/pebble \
|| (cd /home/daniel/go/src/github.com/letsencrypt/pebble; git checkout -f master && git reset --hard HEAD && git pull -q)
fatal: destination path '/home/daniel/go/src/github.com/letsencrypt/pebble' already exists and is not an empty directory.
Already on 'master'
Your branch is up-to-date with 'le/master'.
HEAD is now at 6c2d514 wfe: compare Identifier.Type with acme.IndentifierIP (#287)
docker-compose -f /home/daniel/go/src/github.com/letsencrypt/pebble/docker-compose.yml up -d
Creating network "pebble_acmenet" with driver "bridge"
Creating pebble_challtestsrv_1 ... done
Creating pebble_pebble_1 ... done
while ! wget --delete-after -q --no-check-certificate "https://localhost:14000/dir" ; do sleep 1 ; done
go clean -testcache
go test -race -coverprofile=coverage_18.txt -covermode=atomic github.com/eggsampler/acme/v3
ok github.com/eggsampler/acme/v3 24.292s coverage: 83.0% of statements
docker-compose -f /home/daniel/go/src/github.com/letsencrypt/pebble/docker-compose.yml down
Stopping pebble_pebble_1 ... done
Stopping pebble_challtestsrv_1 ... done
Removing pebble_pebble_1 ... done
Removing pebble_challtestsrv_1 ... done
Removing network pebble_acmenet
```
When a nag group hits capacity we set the nagsAtCapacity gauge to 1.
This gauge also needs to be reset to 0 when the nag group is no longer
at capacity.
Updates `github.com/weppos/publicsuffix-go` to 3dd5f42, and
`github.com/zmap/zlint` to eea5fe8. Both hashes are the tip of master at
the time of writing.
Unit tests are confirmed to pass:
```
~/go/src/github.com/weppos/publicsuffix-go$ git log --pretty=format:'%h' -n 1
3dd5f42
~/go/src/github.com/weppos/publicsuffix-go$ go test ./...
? github.com/weppos/publicsuffix-go/cmd/load [no test files]
ok github.com/weppos/publicsuffix-go/net/publicsuffix 0.008s
ok github.com/weppos/publicsuffix-go/publicsuffix 0.005s
? github.com/weppos/publicsuffix-go/publicsuffix/generator [no test files]
~/go/src/github.com/zmap/zlint$ git log --pretty=format:'%h' -n 1
eea5fe8
~/go/src/github.com/zmap/zlint$ go test ./...
ok github.com/zmap/zlint 0.240s
? github.com/zmap/zlint/cmd/zlint [no test files]
? github.com/zmap/zlint/cmd/zlint-gtld-update [no test files]
ok github.com/zmap/zlint/lints 0.156s
ok github.com/zmap/zlint/util 0.020s
```
I couldn't get this to work cleanly with
`--log-queries-not-using-indexes` because a couple of queries show up
during integration test runs, seemingly because the tables involved are
small enough that the optimizer finds it faster to skip the index.
Some possible followups:
- Allow list those queries, or
- Preload the DB with a certain number of certificates before the start
of testing.
This is a small clean-up I spotted while migrating the `WithTransaction` wrapper
out of the `sa` package into `db` during #4544.
The `admin-revoker` util. was using bare transactions with the `db.Rollback`
(prev `sa.Rollback`) helper function instead of the newly exported
`db.WithTransaction` wrapper. The latter is safer so we should use it here too.
After this change all of the external consumers of the `Rollback` function have
been switched to using `WithTransaction` so we can unexport `Rollback`.
The `orphans` Prometheus `CounterVec` is used to count orphans that
couldn't be confirmed saved by the SA and were queued by the CA.
The `adopted_orphans` `CounterVec` is used to count orphans pulled from
the queue by the CA and successfully integrated through to the SA.
Both counter stats are labelled by "type", e.g. "precert" or "cert".
Based on the volume of data Boulder supports we use `BIGINT(20)` for
database ID fields throughout all of our tables except for two that were
missed: `fqdnSets` and `issuedNames`. Prior to this migration both were
using `INT(11)`, allowing only values up to 2,147,483,647. After the
migration is applied the `BIGINT(20)` type allows values up to 2^63-1.
This avoids needing to send the entire certificate in OCSP generation
RPCs.
Ended up including a few cleanups that made the implementation easier.
Initially I was struggling with how to derive the issuer identification info.
We could just stick the full SPKI hash in certificateStatus, but that takes a
significant amount of space, we could configure unique issuer IDs in the CA
config, but that would require being very careful about keeping the IDs
constant, and never reusing an ID, or we could store issuers in a table in the
database and use that as a lookup table, but that requires figuring out how to
get that info into the table etc. Instead I've just gone with what I found to
be the easiest solution, deriving a stable ID from the cert hash. This means we
don't need to remember to configure anything special and the CA config stays
the same as it is now.
Fixes#4469.
We've found we need the context offered from logging the error closer to when it
happens in the `bdns` package rather than in the `va`. Adopting the function
requires adapting it slightly. Specifically in the new location we know it won't
be called with any timeout results, with a non-dns error, or with a nil
underlying error.
Having the logging done in `bdns` (and specifically from `exchangeOne`) also
lets us log the wire format of the query and response when we get a `dns.ErrId`
error indicating a query/response ID mismatch. A small unit test is included
that ensures the logging happens as expected.
In case it proves useful for matching against other metrics the DNS ID mismatch
error case also now increments a dedicated prometheus counter vector stat,
`dns_id_mismatch`. The stat is labelled by resolver and query type.
Resolves https://github.com/letsencrypt/boulder/issues/4532
The most recent tagged release of mysql is v1.4.1, from a year ago. It
also happens to pull in an unwanted dependency (appengine) that the
latest commit does not.
Tests pass:
$ go test -count=1 github.com/go-sql-driver/mysql
ok github.com/go-sql-driver/mysql 0.068s
Fixes#4530
Daniel McCarney