Commit Graph

3699 Commits

Author SHA1 Message Date
Chengyuan Zhang 213b91b165
xds: refactor XdsLoadReportClient and XdsLoadStatsStore in order to integrate with XdsLoadBalancer (part 1) (#5863)
* extract self-defined Locality into XdsLocality class

* separate out functionalities for recording client load from lrsClient, xds load balancer will directly interact with XdsLoadStatsStore to set up locality counters

* added GRPC to constant TRAFFICDIRECTOR_HOSTNAME_FIELD name to better match that in XdsComms

* fixed bug of using the wrong cluster name in load report's ClusterStats, it should be GSLB service name, which is responsed by load report response (same as that in EDS response).

* added a new line to the end of files.

* Revert "fixed bug of using the wrong cluster name in load report's ClusterStats, it should be GSLB service name, which is responsed by load report response (same as that in EDS response)."

This reverts commit 6097dd4066.

* rephrase interface comment for StatsStore

* added equality and hashCode test for XdsLocality
2019-06-11 09:43:15 -07:00
Chengyuan Zhang c98fb2d03e
xds: fix bug of missing total_dropped_requests field in ClusterStats proto (#5862) 2019-06-10 14:22:16 -07:00
ZHANG Dapeng f7077a565a
xds: cleanup XdsLbStateTest
The test case `XdsLbStateTest.handleSubchannelState()` was introduced before `LocalityStore` refactored out of `XdsLbState`. After `LocalityStore` refactored out, the test case should not be in `XdsLbStateTest` anymore. The test case is already covered in `LocalityStoreTest`.
2019-06-10 14:19:27 -07:00
ZHANG Dapeng b69b15fddb
javadoc: exclude internal APIs
Fixes #5858 in master
2019-06-10 13:36:21 -07:00
ZHANG Dapeng 33c30db42c
xds: allow grpclb balancer addresses for backward compatibility
During migration, the name resolver may not know when the client has been upgraded to xds, so it may still send grpclb v1 addresses with a list of policies including both grpclb v1 and xds.
2019-06-10 11:27:42 -07:00
Carl Mastrangelo 2db3abc9ad
Revert "core, netty: add io.perfmark Annotations" (#5853)
This causes internal breakage which needs to be resolved before continuing.

This reverts commit 71967622d6.
2019-06-07 17:23:49 -07:00
Carl Mastrangelo 44ecdf3649
interop-testing: fix tests on Android 2019-06-07 15:01:21 -07:00
Kun Zhang c6f15162ff
services: fix HealthCheckingLoadBalancer.shutdown(). (#5848)
HealthCheckingLoadBalancer.shutdown() calls
hcState.onSubchannelState(SHUTDOWN) which removes that hcState from
helper.hcStates.  Therefore, if more than one Subchannels are present,
ConcurrentModificationException will be thrown.

Since HealthCheckingLoadBalancer.shutdown() will clear the hcStates
set after the loop, it's unnecessary to do the deletion within the
loop.  However, when a Subchannel is shutdown by LoadBalancer, its
HcState still needs to be removed.  To do that, change moves the
deletion to Subchannel.shutdown().
2019-06-07 09:32:55 -07:00
Eric Anderson 26bd76fa76
Upgrade to Gradle 5 2019-06-07 08:40:53 -07:00
Eric Anderson a284cff892 java_grpc_library: Swap to descriptor_set_in to protoc
This avoids re-parsing the proto files and allows proto_library
to enforce more checks.

This is an export of cl/197343148
2019-06-07 07:27:01 -07:00
Carl Mastrangelo 71967622d6
core, netty: add io.perfmark Annotations
This add perfmark annotations in some key places, notably on transport/application boundaries, and thread hop locations. Perfmark records to a thread-local buffer the events that happen in each thread. Perfmark is disabled by default, and will compile to a noop unless Perfmark.setEnabled is invoked. This should make it free when disable, and pretty fast when it is enabled.

It is important that started tasks are ended, so several places in our code are moved to either try-finally blocks, or moved into a private method. I realize this is ugly, but I think it is manageable. In the future, we can look at making an agent or compiler plugin that simplifies the recording.

Linking between threads is done with a Link object, which is created on the "outbound" task, and used on the "inbound" task. This is slightly more verbose, and does has a small amount of runtime overhead, even when disabled. (for null checks, slightly higher memory usage, etc.) I think this is okay to, because it makes other optimizations much easier.
2019-06-06 17:58:49 -07:00
Eric Anderson 5d0c283b46 java_grpc_library.bzl: Support alternative javac toolchains
Depending on jdk:toolchain causes java_grpc_library to always use the
_default_ toolchain, even if the user tried to override it. Changing to
:current_java_toolchain allows the rule to use the user-selected
toolchain when overridden.

Tested by adding to BUILD:
load("@bazel_tools//tools/jdk:default_java_toolchain.bzl", "default_java_toolchain")
default_java_toolchain(
    name = "mychain",
    misc = ["-Amy=flag"],
    visibility = ["//visibility:public"],
)

And then verifying -Amy=flag is in the output of:
bazel aquery --java_toolchain=:mychain services:_reflection_java_grpc

Fixes #5841
2019-06-06 15:31:31 -07:00
Eric Anderson 63a6e26f39 Hard-code Netty's epoll classifier
There's only one classifier that we use today, and we really want the
compilation results to be the same independent of which machine you
compiled on.
2019-06-06 15:28:11 -07:00
Carl Mastrangelo dcd68e5b57
context: Extend raw ComparableSubject instead of supplying type parameters 2019-06-06 15:00:03 -07:00
Carl Mastrangelo f8ba38a0e4
alts: ensure only the first few bytes of key are used
Fixes grpc/grpc#19271
2019-06-06 14:59:22 -07:00
Eric Anderson be819fa3fd
android: Convert to maven-publish
com.github.dcendents:android-maven-gradle-plugin is incompatible with
Gradle 5 and the project hasn't seen any activity in over a year, so it
seems unlikely to get fixed.

We want to use maven-publish anyway, since that's what we use elsewhere.
2019-06-06 14:53:27 -07:00
Carl Mastrangelo 8536832232
core,netty: expose server stream id 2019-06-06 13:52:22 -07:00
Eric Anderson ee5731cc18 interop-testing: Only set okhttp's sslSocketFactory for test CA
We want the interop client to be configured like a normal user would,
and a normal user wouldn't call sslSocketFactory to use the default
roots.
2019-06-06 10:30:10 -07:00
Carl Mastrangelo 9ef0e9fc1b
interop-testing: disable timeout when debugging 2019-06-05 23:39:43 -07:00
ZHANG Dapeng 6aadaf0a64
core,services: cleanup io.grpc.internal.IoUtils 2019-06-05 17:31:46 -07:00
Carl Mastrangelo 7657523b28
all: update to error prone 2.3.3 2019-06-05 15:28:43 -07:00
ZHANG Dapeng 16de96befe
xds: Add gogoproto dependency to xds
The generated grpc services are not changed.
2019-06-05 10:13:19 -07:00
Carl Mastrangelo 409afe5867
all: update to truth 0.45 2019-06-04 21:12:21 -07:00
Jihun Cho 23170c298e
alts: add TsiPeer boolean property (#5824) 2019-06-03 16:29:48 -07:00
Chengyuan Zhang f81201024e
upgrade netty version to 4.1.35 and netty-tcnative version to 2.0.25 (#5818) 2019-06-03 11:40:59 -07:00
Ran 81ba42a1d6
core: expose some of AutoConfiguredLoadBalancer because some internal tests need to access them (#5821)
* core: revert some changes to fix tests

* fix style
2019-06-03 09:34:04 -07:00
Chengyuan Zhang 93551719b9
xds: integrate backend metric API to client load reporting (#5797)
* augmented ClientLoadCounter with backend metrics

* added a listener implementation for receiving backend metrics and aggregate in ClientLoadCounter
2019-05-31 14:28:23 -07:00
Kun Zhang 276b7d8512
interop-testing: create GrpclbLongLivedAffinityTestClient (#5817)
This is a long-running stand-alone test client for a specific customer that uses GRPCLB's pick_first mode.
2019-05-31 12:54:05 -07:00
Manuel Kollus e526891a2b api,protobuf-lite: solve code style issues 2019-05-31 12:49:30 -07:00
Eric Anderson c6c2ee876a core: Remove unnecessary SuppressWarnings from JsonParser 2019-05-31 10:09:39 -07:00
Jihun Cho d37f87abce
core: Migrate InternalSubchannel to use SynchronizedContext (#5555) 2019-05-30 18:40:50 -07:00
ZHANG Dapeng d8aa42723d
xds: fix bug in XdsLoadBalancerProvider.parseLoadBalancingConfigPolicy
Resolves #5804
2019-05-30 16:37:08 -07:00
ZHANG Dapeng f9decbf69d
xds: remove unused variables 2019-05-30 14:38:43 -07:00
Eric Anderson 3c931b40b0 api: Mention similarity of synccontext to a dedicated thread
This is the conceptual model we use. Document it to help aid others'
understanding and make it easier to understand when it is appropriate to
use.
2019-05-30 10:52:22 -07:00
Kun Zhang eff13a9ec8
core: only let ManagedChannelImpl convert empty resolution result to error (#5803)
Previously, AutoConfiguredLoadBalancer was also handling it, but it
doesn't trigger retries.  By returning true for
canHandleEmptyAddressListFromNameResolution(),
AutoConfiguredLoadBalancer effectively by-passed the empty-result
handling logic in ManagedChannelImpl, thus resolution retries were
never triggered.

This change requires AutoConfiguredLoadBalancer to stop being a
LoadBalancer, for its tryHandleResolvedAddresses().  It doesn't
cause any trouble because AutoConfiguredLoadBalancer has become
less and less like a LoadBalancer during the service config changes.
2019-05-30 09:48:30 -07:00
Kun Zhang af2c16d301
api: deprecate Helper.updateSubchannelAddresses() and add equivalent on Subchannel (#5802)
Resolves #5676
2019-05-30 09:16:38 -07:00
Ryan Michela 9b4c958201 Explain why client stub mocking is discouraged (#5796) 2019-05-30 08:52:13 -07:00
Eric Anderson bc2e1764f6 api,stub: Clarify isReady()/onReady() interaction semantics 2019-05-29 17:28:45 -07:00
Tim van der Lippe ad0893737e Migrate org.mockito.Matchers#any* to org.mockito.ArgumentMatchers
The former is deprecated and replaced by the latter in Mockito 2.
However, there is a functional difference: ArgumentMatchers will reject
`null` and check the type if the matcher specified a type (e.g.
`any(Class)` or `anyInt()`). `any()` will remain to accept anything.
2019-05-29 16:53:01 -07:00
Carl Mastrangelo 4e5e19f6fd
Revert "netty: change server to new protocol negotiator model" (#5798)
This reverts commit f94b77c87f.
2019-05-28 21:12:25 -07:00
Eric Anderson 2cbc540cb3 Remove deprecated enableKeepAlive API from transports
As mentioned in 5188[1], the default used with the enableKeepAlive API
conflicted with the default server enforcement. Instead of fixing it,
remove it. These APIs were deprecated in v1.3.0 in April 2017.

1. https://github.com/grpc/grpc-java/issues/5188#issuecomment-482269303
2019-05-28 16:37:02 -07:00
ZHANG Dapeng 77a512551f
xds: handle 100% drop for fallback mode
- Cancel fallback timer and/or exit fallback mode once receiving an EDS response indicating 100% drop.
- Also update balancing state once receiving the first EDS response with drop information when the channel is at the initial IDLE state.
2019-05-24 20:54:37 -07:00
Chengyuan Zhang 7fd5f261b4
xds: implement lb policy backend metric api (#5639)
* implemented utility methods to create ClientStreamTracer.Factory with OrcaReportListener installed for retrieving per-request ORCA data

* added unit tests

* use delegatesTo instead of spy

* implemented OrcaReportingHelper delegating to some original Helper for load balancing policies accessing OOB metric reports

* added unit tests for out-of-band ORCA metric accessing API in a separate test class

* rebase to master, resolve the breaking change of StreamInfo class being final with builder

* trashed hashCode/equal for OrcaReportingConfig

* changed log level and channel trace event level to ERROR as required by design doc

* added OrcaReportingHelperWrapper layer to allow updating report interval at any time

* reverse the naming of parent/child helper, child helper is the outer-most helper in the wrapping structure

* changed orca listener interface to use separate listener interfaces for per-request and out-of-band cases

* added more comprehensive unit tests

* added test case for per-request reporting that parent creates its own stream tracer

* fixed bug of directly assign reporting config, which would cause it be mutated later

* separate test cases for updating reporting config at different time

* fixed lint style error

* polish comments

* minor polish in unit tests

* refactor OrcaUtil class into OrcaOobUtil and OrcaPerRequestUtil and get rid of static methods for easier user testing

* hide BackoffPolicyProvider and Stopwatch supplier in OrcaOobUtil's public API

* add javadoc for getInstance() methods

* ensure the same Subchannel instance created by the helper that has corresponding OrcaOobReportListener registered are passed to the listener callback

* removed costNames foe OrcaReportingConfig

* removed redundant checks

* reformated the OrcaOobUtilTest class to put helper methods in the bottom

* fixed impl with changes made on Subchannel (SubchannelStateListener now ties with Subchannel)

* fixed comments

* added usage examples in javadoc for OrcaUtils

* add method comments for OrcaUtil's listener API threading

* make fields in OrcaReportingConfig final

* fixed OrcaOobUtilTest for calling setOrcaReportingConfig inside syncContext

* added ExperimentalApi annotation for Orca utils
2019-05-24 15:12:22 -07:00
Yang Song 46f34a513f core: Make OpenCensus tags non-propagating. (#5689)
The `method` and `status` shouldn't be propagated in the first place,
but in previous OpenCensus implementation all tags are propagating by
default. Now with the TagMetadata it may make sense to change them to
local tags.

This will be a breaking change to users who depend on the behavior that
these tags propagate through process boundaries.
2019-05-24 14:32:26 -07:00
Chengyuan Zhang d86d3dd363
all: fix lint and revert redundant lint fixes in #5570 (#5787)
* Revert "all: fix lint (#5770)"

This reverts commit 00d4cc29ad.

* all: fix lint and revert redundant lint fix in #5570
2019-05-24 01:02:12 -07:00
ZHANG Dapeng 2180fcd113
xds: fix protobuf fields can not be null 2019-05-23 18:27:09 -07:00
Kun Zhang 881dbeb3a2
core: fix a deterministic race between the old createSubchannel() and Subchannel.requestConnection(). (#5785)
The old createSubchannel() doesn't require being called from
sync-context, thus its implementation schedules start() on
sync-context because start() has such requirement.  However, if a
LoadBalancer call createSubchannel() in sync-context, start() is
queued and only called after the control exits the sync-context.  If
createSubchannel() is immediately followed by other Subchannel methods
that requires start(), such as requestConnection(), they will fail.
This will be a very common issue as most LoadBalancers call
createSubchannel() in the sync-context.

This fix splits out the real work of start() into internalStart() which
is called by the old createSubchannel().
2019-05-23 17:10:27 -07:00
Eric Anderson d7ef437dac RELEASING.md: Update locations for new grpc.io website 2019-05-23 13:31:47 -07:00
Eric Anderson 5a4178a74e Update README to reference 1.21.0 2019-05-23 13:31:25 -07:00
Eric Anderson 58c0ea5961 RELEASING.md: Use search.maven.org instead of mvnrepository.com
search.maven.org is more authoritative than mvnrepository.com
2019-05-23 12:56:29 -07:00