Commit Graph

5254 Commits

Author SHA1 Message Date
John Cormie 3ff23d3684
Synchronize access to acknowledgedOutgoingBytes/transmitWindowFull. (#8547)
Fixes #8536
2021-09-22 12:19:48 -07:00
John Cormie 46dbac3eb6
Make manifest usable with android_instrumentation_test()s in google3 (#8545) 2021-09-22 11:53:16 -07:00
Jan Tattermusch 3049c2c147 reenable previously disabled aarch64 tests 2021-09-22 10:53:10 -07:00
markb74 e244065b0c
Fix flakey security policy tests. (#8550)
Using ShadowProcess to set the processes uID doesn't help since SecurityPolicies class fetches the ID in a static initializer, and it may have already been loaded.

Instead, just rely on whatever the uID is already, and ensure the other UIDs we test with are offset from that first value.
2021-09-22 19:19:10 +02:00
Zhouyihai Ding 5396a1de3d
grpclb: remove redundant logs and add a system property to hide server lists in logs
The server list updates are very verbose and currently logged every second, causing a huge log spam if `ChannelLogger` is completely enabled. For debugging an internal issue, we need to turn on `ChannelLogger` but hide the server list updates from the logs to keep the log size reasonable.
2021-09-22 10:13:42 -07:00
Zhouyihai Ding e41df60bea
core: change the mapping from ChannelLogLevel to java.util.logging.Level
Instead of `ChannelLogLevel.{DEBUG,INFO}` mapping to the same java level, `ChannelLogLevel.{WARNING,ERROR}` will shame the same java level. This allows us to be able to independently control the visibility of `ChannelLogLevel.DEBUG` logs which are the most verbose.
2021-09-22 09:43:08 -07:00
yifeizhuang f33daf0d9e
xds: implement equals hashcode in rbac matcher tree (#8546) 2021-09-21 16:29:07 -07:00
Piotr Morgwai Kotarbinski a6abb1b8d9
stub: add ServerCallStreamObserver.setOnCloseHandler(...) (#8452)
This allows for user code to be notified when the messages are actually
put on the wire and the stream is closed.

Fixes #5895
2021-09-21 11:31:04 -07:00
ZHANG Dapeng 29d238afca
api,stub: clarify StreamObserver and Listener param type (#8544) 2021-09-21 10:38:41 -07:00
ZHANG Dapeng 25022f6846
dep: bump netty to 4.1.63.Final and tcnative to 2.0.38.Final (#8167)
Upgrade Netty. This should also resolve #7830.
2021-09-21 09:14:54 -07:00
yifeizhuang e4a13778e0
xds: disable rbac by default (#8537) 2021-09-20 13:46:36 -07:00
ZhenLian 838438cedb
AdvancedTls: add functions to load credentials from static files (#8525)
* AdvancedTls: add functions to load credentials from static files
2021-09-17 09:45:41 -07:00
yifeizhuang 38a554c23a
xds: implement RBAC gRFC misc cases (#8518) 2021-09-16 16:12:52 -07:00
yifeizhuang fcf13952bb
xds, rbac: build per route serverInterceptor for httpConfig (#8524) 2021-09-16 12:35:09 -07:00
Eric Anderson 9d9d8ec66b
xds: Fix test compilation for confused javac
The internal build fails with "reference to assertThat is ambiguous". It
isn't clear why the internal build fails while the external one is okay,
but it is clear that the wildcard T return of readOutbound() is probably
confusing things as javac is considering assertThat(BigDecimal) as a
possible match.

The T return type is a hidden, convenience cast. We force the type
passed to assertThat() to be Object to avoid any ambiguity.
2021-09-16 12:09:15 -07:00
sanjaypujare 49842d2af1
xds: add hashCode and equals back to SslContextProviderSupplier (#8528) 2021-09-15 15:46:22 -07:00
Eric Anderson 7669656725 Revert "netty: Requests with Connection header are malformed"
This reverts commit 6e89919e32.

This was found to break a test proxy. We'll work on fixing the proxy and
then roll this forward again.
2021-09-15 15:20:40 -07:00
Eric Anderson 43b507160f xds: Drain old server connections on Listener updates
This is necessary to make sure all connections are using the new
configuration.
2021-09-15 10:08:28 -07:00
Eric Anderson 5307b69c9e netty: Allow protocol negotiators to shut down transport, with grace period
This will be used for draining old connections when xDS configuration
changes.
2021-09-15 10:08:28 -07:00
Eric Anderson 122b3b2f7c
netty: Support Host header on server-side
We want to know the single, unambiguous authority for the request. If
there is no authority, we use host instead. While authority would be
most typical for HTTP/2, requests proxied from HTTP/1 may use host
instead of authority.

This is generally useful, but the impetus is RBAC. See gRFC A41.
2021-09-15 09:40:56 -07:00
ZHANG Dapeng 3b237339c7
core: discard outbound content-length header (#8522)
Since netty version v4.1.67, content-lenght header validation will be enforced. So once grpc upgrades netty to that version or above, RPCs with invalid content-length header will fail.

Some libraries such as HTTP to gRPC adapters blindly copy all HTTP headers to gRPC metadata, but the content-length header is one of those that shouldn't be forwarded because gRPC uses different encoding. This mistake has already been in existence for a long time.

Discard outbound content-length headers in gRPC, so that users who encounter invalid content-length issue when upgrading grpc-java version on server/client side would be able to workaround by upgrading grpc-java on client/server side as well without fixing the HTTP adapter.
2021-09-13 17:15:45 -07:00
Terry Wilson 876f56e2ea
api: Stabilize the Status.asException() call. (#8520)
Removes the ExperimentalApi annotation from this call.

Contributes to: #4683
2021-09-13 14:54:25 -07:00
Eric Anderson 6e89919e32
netty: Requests with Connection header are malformed
Although this is part of HTTP/2 and should have already been handled
already, it was noticed as part of RBAC work to avoid matching
hop-by-hop headers. See gRFC A41.

Also add a warning if creating Metadata.Key for "Connection". Use this
to try to help diagnose a client if it happens to blindly copy headers
from HTTP/1, as PROTOCOL_ERROR is hard to debug.
2021-09-13 11:30:19 -07:00
ZHANG Dapeng 7c6f53ab79
all: add internal API to disable retry stats (#8510)
Resolves b/197648853 for internal performance regression. Reporting retry stats caused significant amount of performance overhead internally.
2021-09-13 09:12:04 -07:00
ZHANG Dapeng 9ff54059d8
xds: populate envoy RetryPolicy with no retryOn to resolver (#8511)
Envoy RetryPolicy with empty retryOn should not be ignored as no retry config when selecting Route config. Therefore, if xDS update for a route contains a RetryPolicy that has no RetryOn value that we support, but the virtual host config does, xds client should choose the Envoy RetryPolicy from the route (even with no RetryOn), rather than choosing the one from virtual host, and try to convert it into grpc RetryPolicy, and end up with no retry.
2021-09-13 08:31:00 -07:00
ZHANG Dapeng 7a65c74283
xds: apply valid resources while NACKing update (#8506)
Implementing [gRFC A46](https://github.com/grpc/proposal/pull/260)
2021-09-11 21:57:47 -07:00
yifeizhuang 7ad7876e99
fix header matcher for null value (#8503) 2021-09-09 12:15:27 -07:00
yifeizhuang a6df9de7bb
xds: add terminal http filter verification, remove lame route filter, add hcm as terminal network filter verification (#8342)
* xds: add terminal filter verification, remove lame route filter

* move last filter check inline

* add server validate terminal filter
2021-09-09 09:55:27 -07:00
Benjamin Peterson 67d5f1b0d6 stub: update CallStreamObserver stabilization issue 2021-09-09 09:53:47 -07:00
yifeizhuang be7aa50441
xds: referenciate server routing config (#8491)
* routing config ref

* atomic ref virtual host list

* Revert "routing config ref"

This reverts commit cbcad5744f.

* test: noop config non-static, better validation
2021-09-08 18:32:26 -07:00
Eric Anderson 9870db1f47 stub: Document that noop onCancelHandler is useful
setOnCancelHandler tells gRPC that the application is handling
cancellation. But it's fine to have noop behavior within the handler
itself if the application doesn't need it. It is just a way to opt-in to
the more recent no-exception-from-onNext behavior. Let's mention this
use-case in the docs to make it more obvious it is a possibility.

Came up as part of #8409.
2021-09-08 16:48:20 -07:00
sanjaypujare 22603810b9
xds: use the new cert-provider instances if present (#8494) 2021-09-08 16:06:21 -07:00
sanjaypujare f71eedff40
xds: remove hashCode() and equals() for SslContextProviderSupplier (#8496) 2021-09-08 15:38:26 -07:00
ZhenLian fb00463001
fix a flaky test in advanced TLS (#8474)
* fix a flaky test in advanced tls
2021-09-08 11:43:23 -07:00
Eric Anderson 1f1396f3f0 Start 1.42.0 development cycle 2021-09-08 09:10:20 -07:00
ZHANG Dapeng cd346832ba
rls: migrate deprecated server/path to extraKeys (#8469)
The [`server` and `path` fields](https://github.com/grpc/grpc-java/blob/v1.40.1/rls/src/main/proto/grpc/lookup/v1/rls.proto#L25-L32) in `RouteLookupRequest` are deprecated. Instead, we will send the server/path information in side of [`key_map`](https://github.com/grpc/grpc-java/blob/v1.40.1/rls/src/main/proto/grpc/lookup/v1/rls.proto#L45).

The keys for the server, service and method in the `key_map` will be the _values_ of `host`, `service`, `method` fields respectively in [`extraKeys`](https://github.com/grpc/grpc-java/blob/v1.40.1/rls/src/main/proto/grpc/lookup/v1/rls_config.proto#L69) in RlsConfig.

We will also include all entries in the [`constantKey`](https://github.com/grpc/grpc-java/blob/v1.40.1/rls/src/main/proto/grpc/lookup/v1/rls_config.proto#L80) in RlsConfig into `RouteLookupRequest`.


Other changes:

- Add AutoValue library for ExtraKeys class, just like data classes used in grpc-xds. Will migrate other data classes to AutoValue as well.
- Not to keep `targetType` field in the route lookup request data class, because we always use "grpc" as targetType.
2021-09-07 21:32:33 -07:00
sanjaypujare 5dc6e0ca54
xds: update Envoy protos to a later revision for the new CertificateProvider definitions (#8490) 2021-09-07 14:27:49 -07:00
Sergii Tkachenko 6cd911757a
census: make internal linter happy
TODO is preferred to FIXME.
2021-09-03 13:26:43 -07:00
ZHANG Dapeng 5475cf12bb
xds: fix parsing retryOn values (#8477)
- Envoy ignores white spaces in `retryOn` field
https://github.com/envoyproxy/envoy/blob/v1.19.1/source/common/router/retry_state_impl.cc#L166

  We should do the same.

- Envoy ignores unsupported values https://github.com/envoyproxy/envoy/blob/v1.19.1/source/common/router/config_impl.cc#L89-L90
  and we should do the same.
2021-09-03 12:47:38 -07:00
sanjaypujare 4828698bec
xds: enable PSM security by default (#8478) 2021-09-03 12:38:26 -07:00
Sergii Tkachenko a91cc85dfd
Revert "core/auth: Remove CallCredentials2 (#8464)"
This reverts commit 7cde473efa.
2021-09-02 17:20:20 -07:00
Brice Jaglin 62fafe7eda core: clarify exception message
Reformulate message to highlight that SizeEnforcingInputStream is
applied on the message size of the message after decompression.
2021-09-02 15:18:00 -07:00
Daniel Zou ffebe231c0
netty-shaded: Rename the directory of netty shaded resources to avoid collisions 2021-09-02 18:12:10 -04:00
zpencer 0838b73674
netty: remove unneeded TransportTracer null checks 2021-09-02 12:01:44 -07:00
ZHANG Dapeng 07747c59a2
xds: Fix WeakReference bug in SharedCallCounterMap (#8466)
Fixes #8397.
#8397 is caused by mistakenly clearing up a map entry right after the entry is recreated after gc. Reproduced in regression test.
2021-09-02 10:25:15 -07:00
ZHANG Dapeng 2faa748797
census: Fix retry stats data race (#8459)
There is data race in `CensusStatsModule. CallAttemptsTracerFactory`:

If client call is cancelled while an active stream on the transport is not committed, then a [noop substream](https://github.com/grpc/grpc-java/blob/v1.40.0/core/src/main/java/io/grpc/internal/RetriableStream.java#L486) will be committed and the active stream will be cancelled. Because the active stream cancellation triggers the stream listener closed() on the _transport_ thread, the closed() method can be invoked concurrently with the call listener onClose(). Therefore, one `CallAttemptsTracerFactory.attemptEnded()` can be called concurrently with `CallAttemptsTracerFactory.callEnded()`, and there could be data race on RETRY_DELAY_PER_CALL. See also the regression test added.

The same data race can happen in hedging case when one of hedges is committed and completes the call, other uncommitted hedges would cancel themselves and trigger their stream listeners closed() on the transport_thread concurrently. 

Fixing the race by recording RETRY_DELAY_PER_CALL once both the conditions are met: 
- callEnded is true 
- number of active streams is 0.
2021-09-02 10:24:22 -07:00
Anuraag Agrawal 522b37bc3b
Fix drift in MessageFramer comment (#8427) 2021-09-02 08:56:56 -07:00
sanjaypujare b0b250024f
xds: fix implementation to comply with gRFC for security (#8468) 2021-09-01 10:49:33 -07:00
Terry Wilson 7cde473efa
core/auth: Remove CallCredentials2 (#8464)
- Removes CallCredentials2
- Removes CallCredentials2ApplyingTest
- Adds two tests from CallCredentials2ApplyingTest to CallCredentialsApplyingTest
- Updates GoogleAuthLibraryCallCredentials to extend from CallCredentials instead of CallCredentials2
2021-09-01 09:49:20 -07:00
Sergii Tkachenko 4fa612ae3d
xds: fix java style 2021-08-31 16:45:37 -07:00