In core, add a new enum element to `RpcProgress` for the case that the stream is closed even before anything leaves the client. `RetriableStream` will do unlimited transparent retry for this type of `RpcProgress` since they are local-only.
In netty, call `tranportReportStatus()` for pending streams on failure.
Also fixes#8394
Previous versions of error prone were incompatible with Java 17 javac.
In grpc-api, errorprone is now api dependency because it is on a public
API. I was happy to see that Gradle failed the build without the dep
change, although the error message wasn't super clear as to the cause.
It seems that previously -PerrorProne=false did nothing. I'm guessing
this is due to a behavior change of Gradle at some point. Swapping to
using the project does build without errorProne, although the build
fails with Javac complaining certain classes are unavailable. It's
unclear why. It doesn't seem to be caused by the error-prone plugin.
I've left it failing as a pre-existing issue.
ClientCalls/ServerCalls had Deprecated removed from some methods because
they were only deprecated in the internal class, not the API. And with
Deprecated, InlineMeSuggester complained.
I'm finding InlineMeSuggester to be overzealous, complaining about
package-private methods. In time we may figure out how to use it better,
or we may request changes to the checker in error-prone.
e0dca93c broke the interop-testing script for unix because Gradle
changed the scripts for
https://github.com/gradle/gradle/security/advisories/GHSA-6j2p-252f-7mw8
The solution here looks weird, but we are inserting the replacement
string into a single-quoted string, so we stop that string, start a
double-quoted string to allow the variable replacement, and then resume
the previous string.
These changes make the build compatible with Gradle 7, except for
Android which requires plugin updates.
I removed animalsniffer from binder because it did nothing (as there
were no signatures) and it was failing after setting toolVersion. It
failed because animalsniffer is only compatible with java plugin. After
this change I put the withId(animalsniffer) loading inside the
withId(java) to avoid a plugin ordering failure. That made it safe again
for binder to load animalsniffer, but it is still best to remove the
plugin from binder as it is misleading.
I did not upgrade Android plugin versions as newer versions (even 3.6)
require dealing with androidx (#8421).
We are setting up fallback test based on TD. Currently the test client is compiled in google3, so we must run it in a container so that the client can have the GRTE dependency. However, container does not have `ip`, `iptables`, etc network command, so we plan to run the network command outside of the container. To do this, add a new flag `skipNetCmd` to skip network commands inside the test client.
Add AbstractXdsInteropTest, XdsTestControlPlaneService and only ping-pong testcase in initial implementation.
AbstractXdsInteropTest sets up the test control plane, create xdsClient and xdServer using bootstrap override, test case extending AbstractXdsInteropTest is supposed to override the control plane config and run the verification.
XdsTestControlPlaneService only has static xds configurations, not able to keep states.
How to run:
./gradlew :grpc-interop-testing:installDist -PskipCodegen=true
./interop-testing/build/install/grpc-interop-testing/bin/xds-e2e-test-client
There is data race in `CensusStatsModule. CallAttemptsTracerFactory`:
If client call is cancelled while an active stream on the transport is not committed, then a [noop substream](https://github.com/grpc/grpc-java/blob/v1.40.0/core/src/main/java/io/grpc/internal/RetriableStream.java#L486) will be committed and the active stream will be cancelled. Because the active stream cancellation triggers the stream listener closed() on the _transport_ thread, the closed() method can be invoked concurrently with the call listener onClose(). Therefore, one `CallAttemptsTracerFactory.attemptEnded()` can be called concurrently with `CallAttemptsTracerFactory.callEnded()`, and there could be data race on RETRY_DELAY_PER_CALL. See also the regression test added.
The same data race can happen in hedging case when one of hedges is committed and completes the call, other uncommitted hedges would cancel themselves and trigger their stream listeners closed() on the transport_thread concurrently.
Fixing the race by recording RETRY_DELAY_PER_CALL once both the conditions are met:
- callEnded is true
- number of active streams is 0.
We don't want other APIs to copy the stub-based API to attach the
interceptor. The API has a shorter name, but isn't actually all that
easier to use and isn't fluent like using the interceptor API.
These are _very_ old methods, so we won't be quick to delete them. Seems
we should have them deprecated at least a year or two; they are easy to
maintain in the mean time.
See API Review notes in #1789
Rebased PR #8343 into the first commit of this PR, then (the 2nd commit) reverted the part for metric recording of retry attempts. The PR as a whole is mechanical refactoring. No behavior change (except that some of the old code path when tracer is created is moved into the new method `streamCreated()`).
The API change is documented in go/grpc-stats-api-change-for-retry-java
The unit tests in RetriableStreamTest do not really test buffer limit from end to end, because the buffer limit is implemented using ClientStreamTracer.Factory, and the tracer callback outboundMessageSize() is only triggered in AbstractClientStream after message serialization. In fact, it was broken without failing any existing tests (#8343 (comment))
This PR adds a retry buffer limit test that runs through the AbstractClientStream code path.
Extend XdsTestServer features as specified in go/xds-retry-interop-test
See also xds retry interop test case implementation grpc/grpc#26746, grpc/grpc#26791
Previously, rpc-behavior values in the request headers are handled in tow different places, one in interceptor and the other in service implementation via Context. I moved all the rpc-behavior handling in interceptor, Context is not needed any more.
This can be used by annotation processors to avoid processing the
gRPC-generated code. The normal Generated annotation only has SOURCE
retention, so isn't available to annotation processors.
I don't include the service name within the annotation as that assumes
we'll never have need for any other type of generated class. If there's
a request for exposing service name via an annotation in the future, we
can make an RpcService annotation or the like.
Fixes#8158
Because we are emulating aarch64 during testing the tests run slower
than normal. Bumping the timeout seems easy and obvious for
StressTestClientTest. The warmup for BinlogHelperTest mirrors what we've
done with timing-based tests in the past. Hopefully that is enough to
lower variance so that it passes consistently; seems likely. I'd rather
not increase the timing fuzziness if I can avoid it, as that reduces the
test's ability to detect issues.
Fixes#8135 (mostly+hopefully)
failOnVersionConflict has never been good for us. It is equivalent to
Maven dependencyConvergence which we discourage our users to use because
it is too tempermental and _creates_ version skew issues over time.
However, we had no real alternative for determining if our deps would be
misinterpeted by Maven.
failOnVersionConflict has been a constant drain and makes it really hard
to do seemingly-trivial upgrades. As evidenced by protobuf/build.gradle
in this change, it also caused _us_ to introduce a version downgrade.
This introduces our own custom requireUpperBoundDeps implementation so
that we can get back to simple dependency upgrades _and_ increase our
confidence in a consistent dependency tree.
To separately manage services/classes with and without protobuf dependency in services package, we are moving classes with protobuf dependency into io.grpc.protobuf.services. This includes healthchecking, reflection, channelz, and binlogging.
Forwarding classes are created to avoid breaking existing users, while they are marked as deprecated to notify users to migrate.
Updates TestServiceClient to support creating channel without target port (mostly useful for xDS that uses the channel target as the resource name).
Adds an env var for overriding the TD URI used in cloud-to-prod test.
Starting in Netty 4.1.60, Netty will validate Content-Length headers
using getAll() and setLong(). While getAll() was documented as only used
in tests, it doesn't appear it was currently used in any tests.
While Http2NettyTest.contentLengthPermitted() was added to confirm that
Content-Length works, it won't actually exercise any interesting
behavior until we upgrade to Netty 4.1.60. However, I did test with
Netty 4.1.60 and it reproduced the failure in
https://github.com/grpc/grpc-java/issues/7953 and passed with this
change.
Since Netty is now observing/modifying the headers, it would seem
appropriate to implement a substantial portion of the Http2Headers API.
However, the surface is much larger than we'd want to implement for a
'quick fix' that could be backported. In addition, it seems much of the
API is just convenience methods, so it is probably appropriate to split
out a AbstractHeaders class from DefaultHeaders in Netty that doesn't
make any assumptions about the header storage mechanism.
Changes the xDS interop test client to support timeout test.
- Synced xDS test proto messages with grpc-proto.
- Changed RpcConfig to be the configuration for per test method type. Added timeoutSec for its deadline configuration.
- Changed accumulated stats to include RPC status instead of just succeeded/failed.
Resolves#7741
Some of the static methods in generated code have the same method name but different package name, such `ClientCalls.asyncClientStreamingCall` and `ServerCalls.asyncClientStreamingCall`. It's less readable using static import than using full-qualified method name in-place.
The proto field is named as num_failures but its comment is saying it is for number of RPCs that failed to record a remote peer. RPC failed == RPC failed to record a remote peer was true previously (so no existing tests should be affected by this changed) as server completed RPCs immediately. It is no longer true with server capability to keep the call open/delayed.
This change clarifies the proto definition for stats RPC. rpcs_by_peer is for recording RPCs succeeded and num_failures is for RPCs failed. RPCs in the flight when the stats call times out are not counted towards any of the stats.
Update xDS interop test proto to aggregate accumulated stats based on RPC methods (mirroring 643e5bcd1e8db931cf76a3be19cd9bba223ee987 in C-core's change). Updated the xDS interop test client to support querying accumulated stats aggregated to RPC methods.
ZHANG Dapeng
Eric Anderson
Jintao
Mohan Li
sanjaypujare
yifeizhuang
apolcyn
Chengyuan Zhang
Sergii Tkachenko
Doug Fawley