Bumps
[github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2)
from 1.27.1 to 1.30.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md">github.com/aws/aws-sdk-go-v2/service/s3's
changelog</a>.</em></p>
<blockquote>
<h1>Release (2023-01-10)</h1>
<h2>Module Highlights</h2>
<ul>
<li><code>github.com/aws/aws-sdk-go-v2/service/location</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/location/CHANGELOG.md#v1210-2023-01-10">v1.21.0</a>
<ul>
<li><strong>Feature</strong>: This release adds support for two new
route travel models, Bicycle and Motorcycle which can be used with Grab
data source.</li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/rds</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/rds/CHANGELOG.md#v1400-2023-01-10">v1.40.0</a>
<ul>
<li><strong>Feature</strong>: This release adds support for configuring
allocated storage on the CreateDBInstanceReadReplica,
RestoreDBInstanceFromDBSnapshot, and RestoreDBInstanceToPointInTime
APIs.</li>
</ul>
</li>
</ul>
<h1>Release (2023-01-09)</h1>
<h2>Module Highlights</h2>
<ul>
<li><code>github.com/aws/aws-sdk-go-v2/service/ecrpublic</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/ecrpublic/CHANGELOG.md#v1150-2023-01-09">v1.15.0</a>
<ul>
<li><strong>Feature</strong>: This release for Amazon ECR Public makes
several change to bring the SDK into sync with the API.</li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/kendraranking</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/kendraranking/CHANGELOG.md#v100-2023-01-09">v1.0.0</a>
<ul>
<li><strong>Release</strong>: New AWS service client module</li>
<li><strong>Feature</strong>: Introducing Amazon Kendra Intelligent
Ranking, a new set of Kendra APIs that leverages Kendra semantic ranking
capabilities to improve the quality of search results from other search
services (i.e. OpenSearch, ElasticSearch, Solr).</li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/networkfirewall</code>:
<a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/networkfirewall/CHANGELOG.md#v1230-2023-01-09">v1.23.0</a>
<ul>
<li><strong>Feature</strong>: Network Firewall now supports the Suricata
rule action reject, in addition to the actions pass, drop, and
alert.</li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/workspacesweb</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/workspacesweb/CHANGELOG.md#v190-2023-01-09">v1.9.0</a>
<ul>
<li><strong>Feature</strong>: This release adds support for a new portal
authentication type: AWS IAM Identity Center (successor to AWS Single
Sign-On).</li>
</ul>
</li>
</ul>
<h1>Release (2023-01-06)</h1>
<h2>Module Highlights</h2>
<ul>
<li><code>github.com/aws/aws-sdk-go-v2/service/acmpca</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/acmpca/CHANGELOG.md#v1210-2023-01-06">v1.21.0</a>
<ul>
<li><strong>Feature</strong>: Added revocation parameter validation:
bucket names must match S3 bucket naming rules and CNAMEs conform to
RFC2396 restrictions on the use of special characters in URIs.</li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/auditmanager</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/auditmanager/CHANGELOG.md#v1230-2023-01-06">v1.23.0</a>
<ul>
<li><strong>Feature</strong>: This release introduces a new data
retention option in your Audit Manager settings. You can now use the
DeregistrationPolicy parameter to specify if you want to delete your
data when you deregister Audit Manager.</li>
</ul>
</li>
</ul>
<h1>Release (2023-01-05)</h1>
<h2>General Highlights</h2>
<ul>
<li><strong>Dependency Update</strong>: Updated to the latest SDK module
versions</li>
</ul>
<h2>Module Highlights</h2>
<ul>
<li><code>github.com/aws/aws-sdk-go-v2/service/accessanalyzer</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/accessanalyzer/CHANGELOG.md#v1190-2023-01-05">v1.19.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/account</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/account/CHANGELOG.md#v180-2023-01-05">v1.8.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/acm</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/acm/CHANGELOG.md#v1170-2023-01-05">v1.17.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/acmpca</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/acmpca/CHANGELOG.md#v1200-2023-01-05">v1.20.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/alexaforbusiness</code>:
<a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/alexaforbusiness/CHANGELOG.md#v1150-2023-01-05">v1.15.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/amp</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/amp/CHANGELOG.md#v1160-2023-01-05">v1.16.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/amplify</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/amplify/CHANGELOG.md#v1130-2023-01-05">v1.13.0</a>
<ul>
<li><strong>Feature</strong>: Add
<code>ErrorCodeOverride</code><code>aws/smithy-go#401</code></li>
</ul>
</li>
<li><code>github.com/aws/aws-sdk-go-v2/service/amplifybackend</code>: <a
href="https://github.com/aws/aws-sdk-go-v2/blob/main/service/amplifybackend/CHANGELOG.md#v1140-2023-01-05">v1.14.0</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="116a622a26"><code>116a622</code></a>
Release 2023-01-05</li>
<li><a
href="ce039452b6"><code>ce03945</code></a>
Regenerated Clients</li>
<li><a
href="095bbfff59"><code>095bbff</code></a>
Update API model</li>
<li><a
href="2998a9800a"><code>2998a98</code></a>
Regenerate clients with <code>ErrorCodeOverride</code> (<a
href="https://github-redirect.dependabot.com/aws/aws-sdk-go-v2/issues/1969">#1969</a>)</li>
<li><a
href="1b0a07d93d"><code>1b0a07d</code></a>
Release 2023-01-04</li>
<li><a
href="ff5b1c7a27"><code>ff5b1c7</code></a>
Regenerated Clients</li>
<li><a
href="cabea36bb4"><code>cabea36</code></a>
Update API model</li>
<li><a
href="cd385dc3b8"><code>cd385dc</code></a>
Update links to point to smithy.io</li>
<li><a
href="4dd79b8978"><code>4dd79b8</code></a>
Rename SyntheticClone to Synthetic</li>
<li><a
href="b302f0a86c"><code>b302f0a</code></a>
Release 2023-01-03</li>
<li>Additional commits viewable in <a
href="https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.27.1...service/s3/v1.30.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This policy existed to ensure that, when updating a dependency to a
non-release-tagged commit, we weren't accidentally bringing in a
broken version of that dependency. Since it was instituted, we have
greatly reduced the number of non-versioned dependencies. And we
have extensive tests of our own that should detect if any of the behavior
we use from one of these dependencies is broken. Therefore this
requirement has fallen by the wayside.
Update the document to reflect current practices.
These helpers are only called in one place each, and provide no real
value via abstraction. Also, they're performing gymnastics in order to
be able to use `base64.URLEncoding`, instead of simply using
`base64.RawURLEncoding`.
The core.CertificateRequest wrapper type only had two fields: an
x509.CertificateRequest, and a slice of bytes representing the unparsed
version of the same request. However, x509.CertificateRequest carries
around its own field .Raw, which serves the same purpose. Also, we
weren't even referencing the .Bytes field anyway. Therefore, get rid of
this redundant wrapper type.
This also makes it clearer just how far the original CSR makes it
through our system. I'd like a future change to remove the CSR from the
request to the CA service, replacing it with a structured object that
only exposes exactly the fields of the CSR we care about, such as names,
public key, and whether or not the must-staple extension should be set.
When attempting to add TLS probe monitoring, got the error `TLS is not a
registered Prober type`. This PR adds TLS Prober to `observer.go` to
complete its registration and adds TLS Prober to the observer README.
Co-authored-by: Samantha <hello@entropy.cat>
We use this pattern in several places: there is a query that needs to
have a variable number of placeholders (question marks) in it, depending
on how many items we are inserting or querying for. For instance, when
issuing a precertificate we add that precertificate's names to the
"issuedNames" table. To make things more efficient, we do that in a
single query, whether there is one name on the certificate or a hundred.
That means interpolating into the query string with series of question
marks that matches the number of names.
We have a helper type MultiInserter that solves this problem for simple
inserts, but it does not solve the problem for selects or more complex
inserts, and we still have a number of places that generate their
sequence of question marks manually.
This change updates addIssuedNames to use MultiInserter. To enable that,
it also narrows the interface required by MultiInserter.Insert, so it's
easier to mock in tests.
This change adds the new function db.QuestionMarks, which generates e.g.
`?,?,?` depending on the input N.
In a few places I had to rename a function parameter named `db` to avoid
shadowing the `db` package.
Although we have a metric for counting how often we reuse a valid
authorization, no code has been incrementing that metric since 2021.
Re-enable it.
Additionally, that metric doesn't help us know how old authorizations
are when they get reused, which is useful information when deciding
whether or how much to shrink our authorization reuse lifetime. Add a
new histogram metric to track the ages of authorizations when they're
attached to an order.
In live.go we use a semaphore to limit how many inflight signing
requests we can have, so a flood of OCSP traffic doesn't flood our CA
instances. If traffic exceeds our capacity to sign responses for long
enough, we want to eventually start fast-rejecting inbound requests that
are unlikely to get serviced before their deadline is reached. To do
that, add a MaxSigningWaiters config field to the OCSP responder.
Note that the files in //semaphore are forked from x/sync/semaphore,
with modifications to add the MaxWaiters field and functionality.
Fixes#6392
Add a new kind of prober to boulder-observer which makes a TLS
connection to the target hostname and expects the certificate presented
for the TLS handshake to have certain properties, such as being valid,
expired, or revoked.
Part of #5927
We use partitioning to be able to clean up old data, and partitioning is
incompatible with unique indexes. We still have a unique index on
`serial`, and these tables are downstream from there. There still may
some duplicates, like when a certificate is treated as orphaned but was
actually successfully added to the DB; when we later go to incorporate
it a duplicate will show up.
This reflects changes already made in prod.
This PR removes a unittest that coincidentally relied on these indexes
to generate an error case it needed: `TestAddPrecertificateStatusFail`.
That test was added in #5918. We can bring that test back with a
significant refactoring to change `*db.WrappedMap` to an interface, but
in the meantime we're prioritizing landing this PR so we have a more
realistic integration test environment.
Adding an insecure option to HTTP prober so that it can still check the
status of sites that we expect to be insecure (e.g. expired sites).
Co-authored-by: Aaron Gable <aaron@aarongable.com>
The SA's GetOrder method issues four separate database queries:
- get the Order object itself
- get the list of Names associated with it
- get the list of Authorization IDs associated with it
- get the contents of those Authorizations themselves
These four queries can hit different database replicas with different
amounts of replication lag, and therefore different views of the
universe. This can result in inconsistent results, such as the Order
existing, but not having any Authorization IDs associated with it.
Conduct these four queries all within a single transaction, to ensure
they all hit a single read-replica with a consistent view of the world.
Fixes#6540
Originally, WFEs had a built-in nonce service. Then we added a "remote
nonce service" via gRPC, but we kept a fallback path for when the remote
nonce service was not configured, to use a built-in nonce service. This
PR removes that fallback path.
Since the fallback path was relied on by the unittests, this also
refactors the unittests to use a gRPC-style nonce service (but in-memory
for the unittests).
Fixes#6530
Fixes#6520.
Note: The unittest for this would be fairly simple - add a duplicate
entry in the certificateStatus table. But we can't do that because our
dev environment currently has a UNIQUE KEY on that table. So adding a
unittest update for this is blocked on #6519.
Delete the NewOrder and NewAuthorizations2 methods from the SA's gRPC
interface. These methods have been replaced by the unified
NewOrderAndAuthzs method, which performs both sets of insertions in a
single transaction.
Also update the SA and RA unittests to not rely on these methods for
setting up test data that other functions-under-test rely on. In most
cases, replace calls to NewOrder with calls to NewOrderAndAuthzs. In the
SA tests specifically, replace calls to NewAuthorizations2 with a
streamlined helper function that simply does the single necessary
database insert.
Fixes#6510Fixes#5816
Update our implementation of ARI to return a renewal window entirely in
the past (i.e., suggesting immediate renewal) if the certificate in
question has been revoked for any reason. This will allow clients which
implement ARI to discover that they need to replace their certificate
without having to query OCSP directly, especially as we move into a
future where OCSP is mostly supplanted by aggregated CRLs.
Fixes#6503
In the WFE, ocsp-responder, and crl-updater, switch from using
StorageAuthorityClients to StorageAuthorityReadOnlyClients. This ensures
that these services cannot call methods which write to our database.
Fixes#6454
The `digest` value in AddCertificate's response message is never used by
any callers. Remove it, replacing the whole response message with
google.protobuf.Empty, to mirror the AddPrecertificate method.
This swap is safe, because message names are not sent on the network,
and empty message fields are omitted from the wire format entirely, so
sending the predefined Empty message is identical to sending an empty
AddCertificateResponse message. Since no client is inspecting the
response to access the digest field, sending an empty response will not
break any clients.
Fixes#6498
When a client cancels their HTTP request, that propagates into
gRPC-level cancellation. But we don't want those canceled RPCs to show
up as 500s, we want them to show up as 408s. We do that by producing a
special ProblemDetails at the gRPC level. However, for that trick to
work, we need to make sure errors from RPC methods get passed through
web.ProblemDetailsForError. There were some places where we didn't do
this and instead created a from-scratch ProblemDetails. This resulted in
spurious 500s.
My methodology was to look at every method call on each of the WFE's
fields that represents a gRPC backend: `ra`, `sa`, `accountGetter`,
`nonceService`, `remoteNonceServe`. If the error handling for that call
did not use web.ProblemDetailsForError, I changed it to use that.
Fixes#6524
In draft because still needs tests.
Followup from #6506.
As noted in that PR: I'm not aware of any way to trigger invalid UTF-8
from the layers below this, so I can't think of a good way to unittest
it.
This avoids serialization errors passing through gRPC.
Also, add a pass-through path in replaceInvalidUTF8 that saves an
allocation in the trivial case.
Fixes#6490
The things we need in crl/checker really only need x509.Certificate.
This allows us to remove a dependency on pkcs11key from the crl checker,
and transitively on CGO.
I've confirmed that `CGO_ENABLED=0 go build ./crl/checker` succeeds on
this branch, while it fails on main.
Make minor changes to our implementation of CAA Account and Method
Binding, as a result of reviewing the code in preparation for enabling
it in production. Specifically:
- Ensure that the validation method and account ID are included at the
request level, rather than waiting until we perform the checks which use
those parameters;
- Clean up code which assumed the validation method and account ID might
not be populated;
- Use the core.AcmeChallenge type (rather than plain string) for the
validation method everywhere;
- Update comments to reference the latest version and correct sections
of the CAA RFCs; and
- Remove the CAA feature flags from the config integration tests to
reflect that they are not yet enabled in prod.
I have reviewed this code side-by-side with RFC 8659 (CAA) and RFC 8657
(ACME CAA Account and Method Binding) and believe it to be compliant
with both.
We have a new method, omitZero, which is to allow `max_statement_time`
and `long_query_time` to be zeroed out, but copy/paste error left
`long_query_time` out. Fix it.
The SA's NewOrder and NewOrderAndAuthzs methods both write new rows (the
order itself, new authorizations, and new OrderToAuthz relations) to the
database, and then quickly turn around and query the database for a
bunch of authz rows to compute the status of the new order. This is
necessary because many orders are created already referencing existing
authorizations, and the state of those authorizations is not known at
order creation time, but does affect the order's status.
Due to replication lag, it can cause issues if the database writes go to
the primary database, but the follow-up read goes to a read-only
database replica which may be lagging slightly. To prevent this issue,
conduct the reads on the same transaction as the writes.
Fixes#6511
This requires setting --issuer-file and --url, but it allows (for
instance) collecting a big pile of serial numbers for a known issuer,
rather than having to keep whole certificates.
- Replace `-1` in return values with `0`. No callers were depending on
`-1`.
- Replace `count(` with `COUNT(` for the sake of readability.
- Replace `COUNT(1)` with `COUNT(*)` (https://mariadb.com/kb/en/count).
Both
versions provide identical outputs but let's standardize on the docs.
Fixes#6494
Right now the expiration mailer does one big SELECT on
`certificateStatus` to find certificates to work on, then several
thousand SELECTs of individual serial numbers in `certificates`.
Since it's more efficient to get that data as a stream from a single
query, rather than thousands of separate queries, turn that into a JOIN.
NOTE: We used to use a JOIN, and switched to the current approach in
#2440 for performance reasons. I _believe_ part of the issue was that at
the time we were not using READ UNCOMMITTED, so we may have been slowing
down the database by requiring it to keep copies of a lot of rows during
the query. Still, it's possible that I've misunderstood the performance
characteristics here and it will still be a regression to use JOIN. So
I've gated the new behavior behind a feature flag.
The feature flag required extracting a new function, `getCerts`. That in
turn required changing some return types so we are not as closely tied
to `core.Certificate`. Instead we use a new local type named
`certDERWithRegId`, which can be provided either by the new code path or
the old code path.
Deprecate these feature flags, which are consistently set in both prod
and staging and which we do not expect to change the value of ever
again:
- AllowReRevocation
- AllowV1Registration
- CheckFailedAuthorizationsFirst
- FasterNewOrdersRateLimit
- GetAuthzReadOnly
- GetAuthzUseIndex
- MozRevocationReasons
- RejectDuplicateCSRExtensions
- RestrictRSAKeySizes
- SHA1CSRs
Move each feature flag to the "deprecated" section of features.go.
Remove all references to these feature flags from Boulder application
code, and make the code they were guarding the only path. Deduplicate
tests which were testing both the feature-enabled and feature-disabled
code paths. Remove the flags from all config-next JSON configs (but
leave them in config ones until they're fully deleted, not just
deprecated). Finally, replace a few testdata CSRs used in CA tests,
because they had SHA1WithRSAEncryption signatures that are now rejected.
Fixes#5171Fixes#6476
Part of #5997
Move ocsp-responder's creation of its clock below its setup of metrics
and logging.
The call to `cmd.Clock()` gets the default logger and prints to the log
if the clock is being set to a fake time using an environment variable,
as we do in our integration tests. The call to `cmd.StatsAndLogging()`
unconditionally creates a logger and sets it as the default logger.
However, because `cmd.Clock()` has already used a (default) logger to
print its warning, the latter call was running into an error.
This actually means that, in our integration tests, ocsp-responder has
not been using the correctly-configured logger; it has been using the
default logger.
Fixes#6491
Create a new gRPC service named StorageAuthorityReadOnly which only
exposes a read-only subset of the existing StorageAuthority service's
methods.
Implement this by splitting the existing SA in half, and having the
read-write half embed and wrap an instance of the read-only half.
Unfortunately, many of our tests use exported read-write methods as part
of their test setup, so the tests are all being performed against the
read-write struct, but they are exercising the same code as the
read-only implementation exposes.
Expose this new service at the SA on the same port as the existing
service, but with (in config-next) different sets of allowed clients. In
the future, read-only clients will be removed from the read-write
service's set of allowed clients.
Part of #6454
Now that we have the ability to easily add multiple gRPC services to the
same server, and control access to each service individually, use that
capability to expose the CA's CertificateAuthority, OCSPGenerator, and
CRLGenerator services all on the same address/port. This will make
establishing connections to the CA easier, but no less secure.
Part of #6448
Add a new gRPC server interceptor (both unary and streaming) which
verifies that the mTLS info set on the persistent connection has a
client cert which contains a name which is allowlisted for the
particular service being called, not just for the overall server.
This will allow us to make more services -- particularly the CA and the
SA -- more similar to the VA. We will be able to run multiple services
on the same port, while still being able to control access to those
services on a per-client basis. It will also let us split those services
(e.g. into read-only and read-write subsets) much more easily, because a
client will be able to switch which service it is calling without also
having to be reconfigured to call a different address. And finally, it
will allow us to simplify configuration for clients (such as the RA)
which maintain connections to multiple different services on the same
server, as they'll be able to re-use the same address configuration.
dependabot[bot]