Adds support for specifying either google default or compute engine
"custom" credentials on the command line. This works like it does in
TestServiceClient.
Another feature from TestServiceClient is also included - the channel
builder is created differently when the server port is 0. This avoids
some ipv6 address parsing shenanigans.
Introduces a flag to enable the logging of collected metrics. In some
cases it can be more practical to analyze the metrics from a log file
after the fact, instead of connecting to the metrics server to collect
the metrics as the client is running.
This breaks the ABI of the classes listed below.
Users that recompiled their code using grpc-java [`v1.36.0`]
(https://github.com/grpc/grpc-java/releases/tag/v1.36.0) (released on
Feb 23, 2021) and later, ARE NOT AFFECTED.
Users that compiled their source using grpc-java earlier than
[`v1.36.0`]
(https://github.com/grpc/grpc-java/releases/tag/v1.36.0) need to
recompile when upgrading to grpc-java `v1.59.0`. Otherwise the code
will fail on runtime with `NoSuchMethodError`. For example, code:
```java
NettyChannelBuilder.forTarget("localhost:100").maxRetryAttempts(2);
```
Will fail with
> `java.lang.NoSuchMethodError: 'io.grpc.internal.AbstractManagedChannelImplBuilder
io.grpc.netty.NettyChannelBuilder.maxRetryAttempts(int)'`
**Affected classes**
Class `AbstractManagedChannelImplBuilder` is deleted, and no longer in
the class hierarchy of the channel builders:
- `io.grpc.netty.NettyChannelBuilder`
- `io.grpc.okhttp.OkhttpChannelBuilder`
- `grpc.cronet.CronetChannelBuilder`
OverrideAuthority() is the modern alternative to rewriting the
InetAddress, and can also work if we change the channel creation to use
target strings instead. We can use test CAs now without resorting to the
Netty-specific APIs. After this change, only 4 classes in
interop-testing depend on Netty.
Interop-testing is both binaries and a library. It hadn't been updated
to use java-library and to expose API-surface dependencies to
dependents.
This fixes the error:
```
> Task :grpc-gae-interop-testing-jdk8:javadoc FAILED
javadoc: error - An internal exception has occurred.
(com.sun.tools.javac.code.Symbol$CompletionFailure: class file for io.grpc.internal.testing.StreamRecorder not found)
Please file a bug against the javadoc tool via the Java bug reporting page
(http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com)
for duplicates. Include error messages and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for io.grpc.internal.testing.StreamRecorder not found
1 error
```
Most changes are migrating from conventions to the equivalent
extensions. JMH, AppEngine, and Jib plugins trigger future compatibility
warnings with `--warning-mode all`.
The movement of configurations was to allow sourceSets to create the
configurations and then we just configure them. When configurations were
before sourceSets, we'd implicitly create the configuration.
The examples were _not_ updated to the newer Gradle, although the
non-Android examples work with the newer Gradle. The Android examples
use an older Android Gradle Plugin which will need to be upgraded first.
https://github.com/grpc/grpc-java/issues/10445
Currently, the gRPC compiler isn't properly using the fully qualified
string name `java.lang.String` instead of `String`. Update the generator
to use the `$String$` alias to avoid compile issues with protobuf
messages called String.
Fixes#10316.
* context, all: move Context classes to grpc-api
clean up grpc-context since it has no source code: only add dep on grpc-api
add exclusion for all transitive deps of grpc-api - only guava
exclude grpc-context as a dependency from grpc-alts because all context code is in grpc-api now
api: 1.7 as target Java version for Context source-set of grpc-api
* core, census: fix the issues with android project pulling in old grpc-context version
* api,context: make changes to bazel build files to account for context code moving from context to api
These code locations just needed a generic gRPC server, without any
transport-specific configuration. Use vanilla ManagedServerBuilder
instead of hard-coding to Netty, as we would suggest to our users.
The flag was documented in https://github.com/grpc/grpc/pull/33180 .
This flag will be useful for some RLS integration tests where we need
to have the client set certain headers for routing purposes.
Ciphers have been "fast enough" for testing since early Java 8 updates;
we haven't needed to override ciphers since we dropped Java 7 support.
Java 8u252 had ALPN, so Conscrypt or Jetty ALPN hasn't been necessary
for basic testing for a while. We still want specialized testing for
Conscrypt, but only tests testing Conscrypt need to care.
This avoids the (often missing) evaluationDependsOn and fixes using
results from other projects without propagating those through
Configuration. It also reduces the number of useless classes pulled in
by down-stream tests, reducing the probability of rebuilds.
The expectation of fixtures is they help testing down-stream code that
use the classes in main. That applies to all the classes here except for
FakeClock and StaticTestingClassLoader. It would also apply to many
internal classes in grpc-testing, but let's consider cleaning that up
future work.
TlsTesting.loadCert() is a public API and so should be preferred over
our internal utility. It avoids creating a temp file that has to be
deleted by a shutdown hook. Usages that needed a file were not migrated.
The internal build rule for this client already adds these dependencies, and is what related tests for these dependencies have been relying on for a while.
We're starting to use the per-release published interop client docker images for those tests too now, and these use the gradle build rules.
The xds server can take a really long time to start if the xds resources
are slow to load. Ideally the management server would be available
during this time so we can inspect the server. The server health still
won't go to SERVING until the xds server starts, which is appropriate.
LoadWorkerTest.runUnaryBlockingClosedLoop and Http2NettyTest.tlsInfo are
failing every CI run. It appears they are the unfortunate tests run
first, so are slowest to start as classloading proceeds. There's
definitely other tests that probably need adjustment, but fixing these
two gives us some hope of having a green run occasionally.
After #9937 was merged, the Java observability tests start to fail.
This System.exit(0) call in the existing Interop client main() method
prevented execution to continue in the new combined Observability
Interop test binary here. (The new binary is calling the old binary's
main() method.)
gcp-observability has many dependencies so is a bit annoying in some
build systems to get working... just for it not to be used in
non-observability scenarios.
grpc-go and c core are using separate binaries for gcp-observability
interop testing, so do the same in Java, which makes interop-testing a
bit lighter.
Introduce an AsyncService interface in the generated code and move the methods from <service>ImplBase to default implementation of the interface.
* update pom files to allow java 1.8
* Add a bindService(<service>Async) method
* Change TestServiceImpl to use the interface and include a bind method instead of extending TestServiceImplBase.
* Correct value being passed to throttler which had been backwards.
* Fix flaky test.
* Add a test using AdaptiveThrottler with a CachingRlsLBClient.
* Address test flakiness.
Full end to end implementation of gRPC server as a Servlet including tests and examples
Co-authored-by: Penn (Dapeng) Zhang <zdapeng@google.com>
Co-authored-by: Chengyuan Zhang <chengyuanzhang@google.com>
This client can be used as a part of XDS federation integration tests. It can concurrently perform RPCs with different stubs using different underlying XDS servers.
For example, one might perform proxyless service mesh and DirectPath RPCs in the same process with flags like:
```
--server_uris=xds:///${PSM_TARGET},google-c2p:///${DIRECTPATH_TARGET} \
--credentials_types=INSECURE_CREDENTIALS,compute_engine_channel_creds \
--test_case=rpc_soak \
--soak_iterations=10 \
--soak_max_failures=0 \
--soak_per_iteration_max_acceptable_latency_ms=2500 \
--soak_overall_timeout_seconds=300
```
Trying to upgrade Gradle to 7.6 improved the checkstyle plugin such that
it appears to have been running in new occasions. That in turn exposed
us to https://github.com/checkstyle/checkstyle/issues/5088. That bug was
fixed in 8.28, which also fixed lots of other bugs. So now we have
better checking and some existing volations needed fixing. Since the
code style fixes generated a lot of noise, this is a pre-fix to reduce
the size of a Gradle upgrade.
I did not upgrade past 8.28 because at some point some other bugs were
introduced, in particular with the Indentation module. I chose the
oldest version that had the particular bug impacting me fixed. Upgrading
to this old-but-newer version still makes it easier to upgrade to a
newer version in the future.
Add big negative integer to pending stream count when cancelled. The count is used to delay closing master listener until streams fully drained.
Increment pending stream count before creating one. The count is also used to indicate callExecutor is safe to be used. New stream will not be created if big negative number was added, i.e. stream cancelled. New stream is created if not cancelled, callExecutor is safe to be used, because cancel will be delayed.
Create new streams (retry, hedging) is moved to the main thread, before callExecutor calls drain.
Minor refactor the masterListener.close() scenario.
rls: Update implementation to match spec.
* Cleanup cache if exceeds max size when add an entry. Make cache entry size calculations more accurate
* Trigger pending RPC processing if unexpired backoff entries were removed from the cache by triggering helper to call it's parent updateBalancingState with the same state and picker
* Introduce minimum time before eviction (5 seconds)
* Change default accept ratio for AdaptiveThrottler from 1.2 -> 2.0
* Configuration validation
* When checking key names for duplicates also look at headers
* Check extra keys for duplicates
See analysis of implementation versus spec at https://docs.google.com/spreadsheets/d/18w5s1TEebRumWzk1pvWnjiHFGKc6MW-vt8tRLY4eNs0/
Introduces a new acceptResolvedAddresses() to the LoadBalancer.
This will now be the preferred way to handle addresses from the NameResolver. The existing handleResolvedAddresses() will eventually be deprecated.
The new method returns a boolean based on the LoadBalancers ability to use the provided addresses. If it does not accept them, false is returned. LoadBalancer implementations using the new method should no longer implement the canHandleEmptyAddressListFromNameResolution(), which will eventually be removed, along with handleResolvedAddresses().
Backward compatibility will be maintained so existing load balancers using handleResolvedAddresses() will continue to work.
Additionally the previously deprecated handleResolvedAddressGroups() method is removed.
%s is fairly safe (requires a Formattable to use Locale), so %d is the
main risk item. Places that really didn't need to use String.format()
were converted to plain string concatenation. Logging locations were
generally converted to using the log infrastructure's delayed
formatting, which is generally locale-sensitive but we're okay with
that. That wasn't done in okhttp, however, because Android frequently
doesn't use MessageFormat so we'd lose the parameters. Everywhere else
was explicitly defined to be Locale.US, to be consistent independent of
the default system locale.
The two checker tasks run quickly so don't gain much from UP-TO-DATE,
but it is convenient to not see them in the noise (checkUpperBoundDeps
in particular). Gradle only performs UP-TO-DATE checks (on the inputs)
if the task has both inputs and outputs defined.
The biggest saving was for distZip/distTar/shadowDistZip/shadowDistTar
which were using the same name for the non-shadow and shadow versions.
Thus the output file would always be out-of-date because it had been
rewritten and was invalid. This is worrisome because we could have
"randomly" been using the shadow Zip/Tar at times and the non-shadow
ones at others, although I think in practice the shadow tasks always run
last and so those are the files we'd see. Changing the classifier avoids
the colliding file names. These tasks took ~7 seconds, so incremental
builds are considerably shorter now.
RuntimeOnly dependencies have been missing since 3624d59. This is
because the implementation configuration extendsFrom the shadow
configuration, so any of the things like runtimeOnly are being lost.
This change isn't "correct" but it stops the bleeding with minimal cost.
It is probably incorrect to be using shadow plugin in interop-testing at
all.
INVALID_ARGUMENT is propagated to the data plane if no previous config
is available. INVALID_ARGUMENT is reserved for application use; LBs
should pretty much use UNAVAILABLE exclusively.
While most of the changes are in xds, there do not appear to be likely
xds code paths that would propagate a bad status to the data plane.
Internal policies either don't use parseLoadBalancingPolicyConfig() and
instead have their configuration objects constructed directly or are
constructed transitively through the cluster manager which uses INTERNAL
if there's a child failure. There was a worrisome hole before this
commit for StatusRuntimeExceptions received by the cluster manager, but
the audit didn't find any locations throwing such an exception.
User-selected policies produce a NACK and are protected from the
existing xds client watcher paths. The worst that appears could happen
is the channel could panic (which uses INTERNAL) if a bug let a bad
configuration through.
This can avoid creating an additional 736 tasks (previously 502 out of
1591 were not created). That's not all that important as the build time
is essentially the same, but this lets us see the poor behavior of the
protobuf plugin in our own project and increase our understanding of how
to avoid task creation when developing the plugin. Of the tasks still
being created, protobuf is the highest contributor with 165 tasks,
followed by maven-publish with 76 and appengine with 53. The remaining
59 are from our own build, but indirectly caused by maven-publish.
all: Update netty to 4.1.77.Final and netty_tcnative to 2.0.53.Final
Also switches to a non-release version of rules_jvm_external to allow Bazel build to work with artifact classifiers.
This moves our depedencies into a plain file that can be read and
updated by tooling. While the current tooling is not particularly better
than just using gradle-versions-plugin, it should put us on better
footing. gradle-versions-plugin is actually pretty nice, but will be
incompatible with Gradle 8, so we need to wait a bit to see what the
future holds.
Left libraries as an alias for libs to reduce the commit size and make
it easier to revert if we don't end up liking this approach.
We're using Gradle 7.3.3 where it was an incubating fetaure. But in
Gradle 7.4 is became stable.
JVM startup costs that happen when dependencies are loaded for the
first time can consume a lot of time (we've occasionally observed
around ~5 seconds of CPU time); this causes frequent test flakes
with xds (google-c2p) when using the current 5 second deadline.
Increasing to 15 seconds should give enough time.
Two main incompatibilities existed in the copy of protos in grpc-proto:
no SimpleContext and an Empty method argument was replaced with a
message. "Context" is a very old word for "Metadata" back from the days
before the current gRPC protocol. We don't need that message in
particular, and well-known protos actually works in Protobuf Lite these
days, so we can swap to wrappers.proto's StringValue and don't need to
upstream a change to grpc-proto. The argument problem is fixed just by
changing the type in the Java code.
With the incompatibilities fixed, do a sync from grpc-proto and include
interop-testing.
When a problem happens, it will now report back quickly instead of
waiting until the timeout expires. The timeout exception will also
report each RPC's state.
This is to help diagnose aarch64 test failures.
This reverts commit 0963f3151d. This
causes dependency problems when importing into Google, as
google-auth-library-java needs to be upgraded and that requires an
upgrade to google-http-java-client to bring in
https://github.com/googleapis/google-http-java-client/pull/1505 .
Reverting for now and will roll forward once those upgrades are
performed.
Retryable was added in google-auth-library 1.5.3 to make clear the
situations that deserve a retry of the RPC. Bump to that version and
swap away from the imprecise IOException heuristic.
go/auth-correct-retry
Fixes#6808
* okhttp: forked required files to make okhttp dep compile only
* okhttp: forked missing file to make okhttp dep compile only
* okhttp: moved url and request files to proxy packge
* okhttp: removed unused methods from forked files; fixed build
In core, add a new enum element to `RpcProgress` for the case that the stream is closed even before anything leaves the client. `RetriableStream` will do unlimited transparent retry for this type of `RpcProgress` since they are local-only.
In netty, call `tranportReportStatus()` for pending streams on failure.
Also fixes#8394
Previous versions of error prone were incompatible with Java 17 javac.
In grpc-api, errorprone is now api dependency because it is on a public
API. I was happy to see that Gradle failed the build without the dep
change, although the error message wasn't super clear as to the cause.
It seems that previously -PerrorProne=false did nothing. I'm guessing
this is due to a behavior change of Gradle at some point. Swapping to
using the project does build without errorProne, although the build
fails with Javac complaining certain classes are unavailable. It's
unclear why. It doesn't seem to be caused by the error-prone plugin.
I've left it failing as a pre-existing issue.
ClientCalls/ServerCalls had Deprecated removed from some methods because
they were only deprecated in the internal class, not the API. And with
Deprecated, InlineMeSuggester complained.
I'm finding InlineMeSuggester to be overzealous, complaining about
package-private methods. In time we may figure out how to use it better,
or we may request changes to the checker in error-prone.
e0dca93c broke the interop-testing script for unix because Gradle
changed the scripts for
https://github.com/gradle/gradle/security/advisories/GHSA-6j2p-252f-7mw8
The solution here looks weird, but we are inserting the replacement
string into a single-quoted string, so we stop that string, start a
double-quoted string to allow the variable replacement, and then resume
the previous string.
These changes make the build compatible with Gradle 7, except for
Android which requires plugin updates.
I removed animalsniffer from binder because it did nothing (as there
were no signatures) and it was failing after setting toolVersion. It
failed because animalsniffer is only compatible with java plugin. After
this change I put the withId(animalsniffer) loading inside the
withId(java) to avoid a plugin ordering failure. That made it safe again
for binder to load animalsniffer, but it is still best to remove the
plugin from binder as it is misleading.
I did not upgrade Android plugin versions as newer versions (even 3.6)
require dealing with androidx (#8421).
We are setting up fallback test based on TD. Currently the test client is compiled in google3, so we must run it in a container so that the client can have the GRTE dependency. However, container does not have `ip`, `iptables`, etc network command, so we plan to run the network command outside of the container. To do this, add a new flag `skipNetCmd` to skip network commands inside the test client.
Add AbstractXdsInteropTest, XdsTestControlPlaneService and only ping-pong testcase in initial implementation.
AbstractXdsInteropTest sets up the test control plane, create xdsClient and xdServer using bootstrap override, test case extending AbstractXdsInteropTest is supposed to override the control plane config and run the verification.
XdsTestControlPlaneService only has static xds configurations, not able to keep states.
How to run:
./gradlew :grpc-interop-testing:installDist -PskipCodegen=true
./interop-testing/build/install/grpc-interop-testing/bin/xds-e2e-test-client
There is data race in `CensusStatsModule. CallAttemptsTracerFactory`:
If client call is cancelled while an active stream on the transport is not committed, then a [noop substream](https://github.com/grpc/grpc-java/blob/v1.40.0/core/src/main/java/io/grpc/internal/RetriableStream.java#L486) will be committed and the active stream will be cancelled. Because the active stream cancellation triggers the stream listener closed() on the _transport_ thread, the closed() method can be invoked concurrently with the call listener onClose(). Therefore, one `CallAttemptsTracerFactory.attemptEnded()` can be called concurrently with `CallAttemptsTracerFactory.callEnded()`, and there could be data race on RETRY_DELAY_PER_CALL. See also the regression test added.
The same data race can happen in hedging case when one of hedges is committed and completes the call, other uncommitted hedges would cancel themselves and trigger their stream listeners closed() on the transport_thread concurrently.
Fixing the race by recording RETRY_DELAY_PER_CALL once both the conditions are met:
- callEnded is true
- number of active streams is 0.
We don't want other APIs to copy the stub-based API to attach the
interceptor. The API has a shorter name, but isn't actually all that
easier to use and isn't fluent like using the interceptor API.
These are _very_ old methods, so we won't be quick to delete them. Seems
we should have them deprecated at least a year or two; they are easy to
maintain in the mean time.
See API Review notes in #1789
Terry Wilson
Sergii Tkachenko
Eric Anderson
Mohan Li
Philip K. Warren
sanjaypujare
apolcyn
Terry Wilson
Daniel Liu
yifeizhuang
Stanley Cheung
Benjamin Peterson
Larry Safran
Colin Alworth
Chris Povirk
yifeizhuang
amirhadadi
Michael Lumish
DNVindhya
Ran
beatrausch
ZHANG Dapeng
Jintao