This is a refactor of the existing xds policy to use XdsClient. It does neither create a copy of EDS policy as in #6371 nor re-implement an EDS policy. This should be similar to the idea of https://github.com/grpc/grpc/pull/20368 in C-core.
Here it refactors `XdsComms2` to an implementation of `XdsClient`, which can be drop-in replaced by `XdsClientImp` when it's available.
Let child balancer (currently is Round Robin) handle the address list with only healthy addresses for its locality. If no healthy address available, let the child balancer handleNameResolutionError.
This change cleans up the definitions of gRPC's data types for envoy's proto messages. These data types carry only information needed for gRPC's logic extracted from envoy's proto messages. Each data type has a static method to convert from its corresponding envoy's proto.
Existing Bootstrapper is an eagerly loaded singleton and reading bootstrap file is done when first time referencing Boostrapper class. This is not use friendly for error propagation.
This change added an instance method Bootstrapper#readBootstrap() for reading the bootstrap file. Instead of making the Bootstrapper instance a singleton, make the bootstrap data (aka, BootstrapInfo) a singleton. Now reading the bootstrap file happens when the first Bootstrapper instance first time calling Bootstrapper#readBootstrap().
Defines an XdsClient interface, which is the blueprint for the class that encapsulates all of the logic for communicating with the xDS server. Also defines watcher interfaces for watching CDS/EDS updates.
`XdsNameResolver` will load bootstrap file and populate `Node` reference to the `XdsLoadBalancer` through attributes of `ResolvedAddresses`. This is only for demo purpose until `XdsClient` is available, whence `XdsNameResolver` will populate `XdsClient` reference to `XdsLoadBalancer` through attributes instead.
Moved `ManagedChannel initLbChannel(Helper helper, String balancerName)` out of `LookasideChannelLb` because `LoadReportClientImpl` need channel be initialized already.
Migration will be in an upcoming PR.
This change reimplements Bootstrapper with new design of bootstrap file format, with no longer using ApiConfigSource proto. The new JSON format of the bootstrap file contains a top level "node" and a "xds_server" object, which contains a "server_uri" string and a `channel_creds` list for channel credentials.
Unknown fields in the bootstrap file are allowed by ignored so that the implementation does not break when new fields are added in the future. Therefore, we cannot simply create a custom proto file and use proto util to parse it.
- Implementing priority failover
- Removed `localityWeight` field from `LocalityLbInfo` because this is always available and up to date in `edsResponsLocalityInfo`, otherwise need to create a new instance of `LocalityLbInfo` each time weight is updated.
- Introduced `PriorityManager` that manages all the priority failover logic.
The fileTree is supposed to be relative to the current project, which in
this case would be the child project, but apparently it isn't. Poked
around but couldn't find out why. In any case this fixes a regression
introduced in 4215b80b where checkstyle would no longer file any Java
files and thus it would not notice any failures.
* Defined a proto message that maps the bootstrap JSON file for convenient parsing purpose.
* Implemented a Bootstrapper class which reads a local bootstrap file.
* Added unit test for Bootstrapper.
* Let xDS load balancer bootstrap from a bootstrap file. Currently not use configurations from bootstrap in xDS communication as the xDS load balancer implementation is undergoing changes. We will integrate it later.
* Added newline at the end of files.
* Added json_name proto field option.
* Replaced all RuntimeException with IOException for bootstrap data errors.
* Changed to use JUnit exception rules in test.
* Use StandardCharSets.UTF_8
* Revert "Let xDS load balancer bootstrap from a bootstrap file. Currently not use configurations from bootstrap in xDS communication as the xDS load balancer implementation is undergoing changes. We will integrate it later."
This reverts commit 37200cdd3c.
* Use initialization-on-demand holder idiom for instantiating Bootstrapper instance.
* Fixed usage of JUnit exception rules.
* Changed lazily instantiated variable to camel case
* Removed unnecessary constructor.
- Get rid of `XdsLbState` and make config change handling into two layers: `LookasideLb` (handles balancer name) and `LookasideChannelLb` (handles child policy), under `XdsLoadBalanecer` (fallback manager layer)
- Move `XdsComms`/`AdsStream` to a layer under `LookasideChannelLb`. They don't keep the helper, but only `SyncCtx` and `ChannelLogger`
- For each layer, we pass in a `LoadBalancer.Factory` for the next/child layer. In test,
+ we mock/fake the factory, so we don't care about the implementation details of the child layer.
+ we capture the helper of `factory.newBalancer(factory)`, so we can mimic `updateBalancingState() `from the child layer.
Part 1 contains fallback management logic. There is no change in fallback management logic.
Examples and android projects were left unchanged. They can be changed
later.
No plugin versions were changed, to make this as non-functional of a
change as possible. Upgrading Gradle to 5.6 was necessary for
pluginManagement in settings.gradle.
* Implemented an XdsNameResolver that always returns a hard-coded service config
* Implemented XdsNameResolverProvider
* Added unit tests for XdsNameResolver and XdsNameResolverProvider
* Added META-INF file for XdsNameReresolverProvider
* Removed balancer name field in hard-coded service config
* Changed URI scheme to xds-experimental.
* Deleted unnecessary executors for running name resolution in a separate thread.
* Fixed nits.
* Fixed usage of GrpcUtil.getDefaultProxyDetector() as it was deleted.
* Removed unnecessary shutdown implementation.
* Replaced return with AssertionError as it hard-coded service config should never have error.
* Removed unused name resolver args.
* Added tail blank line.
* xds: do not add a new CallMetricRecorder instance to context if there is already one
* save a local reference to CallMetricRecorder instance to allow ServerCall#close() being called in a different context
* added test coverage for having ServerInterceptor/ServerStreamTracerFactory both install CallMetricRecorder
* Added methods in CallMetricRecorder accessor for accessing context key and creating new CallMetricRecorder instance.
* Added impl for a server interceptor that records and sends custom metric values to client side.
* Added unit test for the server interceptor.
* Modified dependencies to have xds package depends on grpc-services (for CallMetricRecorder) and grpc-testing-proto (for testing proto).
* Changed usage of GrpcServerRule to GrpcCleanupRule.
* make constant fields static
* use GrpcCleanupRule to register InProcessChannel to avoid manual shutdown.
* Make class public and match the usage of grpclb's load reporting server tracer.
* added ExperimentalApi annotation
* compiler: Use 'SERVICE_NAME' instead of duplicated '$Package$$service_name$'
* compiler: Align indentation
* Fix typo
* Add modified golden files and all re-generated code to meet Travis CI and Windows build requirements
See PR #5943
* Polishing
* Fixed LocalityStore test relying on WeightChildPicker list order for mapping to its corresponding locality.
* Fixed bug for not updating new weights for existing localities.
* xds/third_party: import envoy proto based on envoy's commit eefcd0e6fcbeba446454bd5396a34c69348338eb
* xds: fixed breaking changes introduced by changes of fields in ORCA proto message.
* Cleaned up XdsLoadStatsStore.
- Renamed the StatsStore interface to XdsLoadStatsStore and its corresponding implementation is XdsLoadStatsStoreImpl.
- Revised/reworded specification for XdsLoadStatsStore.
* Cleaned up ClientLoadCounter specification.
Reworded specification for ClientLoadCounter
* Cleaned up XdsLoadReportClient, reworded specifications, formatted tests.
* Removed Xds prefix from LoadStatsStore.
* Removed Xds prefix from XdsLoadReportClient.
* augmented xds child balancer helper with orcaOobReportingHelperWrapper and added an interface method in LocalityStore for updating out-of-band backend metrics reporting interval
* added an callback for passing load reporting interval received from lrs response to xDS load balancer, thus, LocalityStore for configuring OOB backend metrics reporting
* moved XdsLoadReportCallback interface into XdsLoadReportClient
* Redefined StatsStore interface.
- Removed interface method StatsStore#interceptPickResult and implementation do not take the resposibility intercepting PickResult with locality-level load recording.
- Introduce a wrapper class for SubchannelPicker to let users wrap SubchannelPicker by themselves, with client side load recording logic.
- Associate the corresponding locality counter with child helper when it is created, child helper will intercept the SubchannelPicker it creates.
* Renamed backend metrics listener class to be more abstract, hides the implementation detail of doing locality-level aggregation.
* Integrate client load recording and backend metrics recording with xDS load balancer.
- Created LoadRecordingSubchannelPicker class for applying XdsClientLoadRecorder that records client load to PickResult.
- Created MetricsObservingSubchannel class for applying OrcaReportingTracerFactory that takes listener to receive ORCA reports to PickResult.
- In xDS load balancer LocalityStore, the original picker is wrapped two layers inside the above wrappers.
* Renamed XdsClientLoadRecorder to ClientLoadRecorder. It should only be used for testing, xDS load balancer should use SubchannelPicker wrappers instead of this load recorder directly.
* Removed redudent layer of wrapping for SubchannelPicker in LocalityStore
* Added toString for SubchannelPicker wrapper classes.
* Rename ClientLoadRecorder to LoadRecordingStreamTracerFactory.
* Renamed StreamInstrumentedSubchannelPicker to TracerWrappingSubchannelPicker.
* Eliminate duplicated code in LocalityStoreTest, put them into a loop.
* fixed bug of intercepting a PickResult with no Subchannel, it should just return the original PickResult. Also, the test was not correct, fixed it.
* changed ClientLoadCounter to a mock in XdsLoadStatsStoreTest, it's not necessary to instantiate a real instance.
* added a TODO comment for suggesting a warning for desired locality counter missing when intercepting a PickResult
* use isSameInstanceAs for verifying intercepting invalid PickResult instead of isEqualTo.
* integrate recordDropRequest in LocalityStore
* integrated StatsStore#addLocality and StatsStore#removeLocality in LocalityStore in handling EDS response.
* integrated picker interception in LocalityStore
* integrate XdsLoadReportClient in XdsLoadBalancer
* put removing locality counters after updating subchannl pickers to narrow down race window
* fixed modifier for XdsLoadReportClientFactory
* refactor handleNewConfig method in XdsLoadBalancer for better readability
* edited message for closing lb rpc when balancer name changes
* weaker the specification of XdsLoadReportClient to allow start/stop be called multiple times.
* removed lrsWorking flag as we relaxed precondition of calling start/stop on XdsLoadReportClient
* refactor initLbChannel to be a factory method for better readability
* added comment for the case when child policy changes, lrs should not be affected
* changed comments for eliminating potential load lose upon locality update.
* make lb RPC cancellation message more informative
* extract self-defined Locality into XdsLocality class
* separate out functionalities for recording client load from lrsClient, xds load balancer will directly interact with XdsLoadStatsStore to set up locality counters
* added GRPC to constant TRAFFICDIRECTOR_HOSTNAME_FIELD name to better match that in XdsComms
* fixed bug of using the wrong cluster name in load report's ClusterStats, it should be GSLB service name, which is responsed by load report response (same as that in EDS response).
* added a new line to the end of files.
* Revert "fixed bug of using the wrong cluster name in load report's ClusterStats, it should be GSLB service name, which is responsed by load report response (same as that in EDS response)."
This reverts commit 6097dd4066.
* rephrase interface comment for StatsStore
* added equality and hashCode test for XdsLocality
The test case `XdsLbStateTest.handleSubchannelState()` was introduced before `LocalityStore` refactored out of `XdsLbState`. After `LocalityStore` refactored out, the test case should not be in `XdsLbStateTest` anymore. The test case is already covered in `LocalityStoreTest`.
During migration, the name resolver may not know when the client has been upgraded to xds, so it may still send grpclb v1 addresses with a list of policies including both grpclb v1 and xds.
- Cancel fallback timer and/or exit fallback mode once receiving an EDS response indicating 100% drop.
- Also update balancing state once receiving the first EDS response with drop information when the channel is at the initial IDLE state.
* implemented utility methods to create ClientStreamTracer.Factory with OrcaReportListener installed for retrieving per-request ORCA data
* added unit tests
* use delegatesTo instead of spy
* implemented OrcaReportingHelper delegating to some original Helper for load balancing policies accessing OOB metric reports
* added unit tests for out-of-band ORCA metric accessing API in a separate test class
* rebase to master, resolve the breaking change of StreamInfo class being final with builder
* trashed hashCode/equal for OrcaReportingConfig
* changed log level and channel trace event level to ERROR as required by design doc
* added OrcaReportingHelperWrapper layer to allow updating report interval at any time
* reverse the naming of parent/child helper, child helper is the outer-most helper in the wrapping structure
* changed orca listener interface to use separate listener interfaces for per-request and out-of-band cases
* added more comprehensive unit tests
* added test case for per-request reporting that parent creates its own stream tracer
* fixed bug of directly assign reporting config, which would cause it be mutated later
* separate test cases for updating reporting config at different time
* fixed lint style error
* polish comments
* minor polish in unit tests
* refactor OrcaUtil class into OrcaOobUtil and OrcaPerRequestUtil and get rid of static methods for easier user testing
* hide BackoffPolicyProvider and Stopwatch supplier in OrcaOobUtil's public API
* add javadoc for getInstance() methods
* ensure the same Subchannel instance created by the helper that has corresponding OrcaOobReportListener registered are passed to the listener callback
* removed costNames foe OrcaReportingConfig
* removed redundant checks
* reformated the OrcaOobUtilTest class to put helper methods in the bottom
* fixed impl with changes made on Subchannel (SubchannelStateListener now ties with Subchannel)
* fixed comments
* added usage examples in javadoc for OrcaUtils
* add method comments for OrcaUtil's listener API threading
* make fields in OrcaReportingConfig final
* fixed OrcaOobUtilTest for calling setOrcaReportingConfig inside syncContext
* added ExperimentalApi annotation for Orca utils
This is the implementation of the Fallback-at-Startup mode in the design doc.
- The Fallback-After-Startup mode is not implemented.
- Drop related behavior is not implemented.
This is a revised version of #5503 (62b03fd), which was rolled back in f8d0868. The newer version passes SubchannelStateListener to Subchannel.start() instead of SubchannelCreationArgs, which allows us to remove the Subchannel argument from the listener, which works as a solution for #5676.
LoadBalancers that call the old createSubchannel() will get start() implicitly called with a listener that passes updates to the deprecated LoadBalancer.handleSubchannelState(). Those who call the new createSubchannel() will have to call start() explicitly.
GRPCLB code is still using the old API, because it's a pain to migrate the SubchannelPool to the new API. Since CachedSubchannelHelper is on the way, it's easier to switch to it when it's ready. Keeping
GRPCLB with the old API would also confirm the backward compatibility.
* added counts for recently issued calls in client side load reporting
* use recordCallStarted/Finished to manipulate counter instead of explicitly incr/decr methods
Summary of PR:
- XdsLbState now assumes standard mode only.
- Will not send CDS request. A EDS request will be sent at the constructor of `AdsStream`.
- Added a method to `LocalityStore`
- `void updateLocalityStore(Map<Locality, LocalityInfo> localityInfoMap);`
- When a EDS response is received. `LocalityStore.updateLocalityStore()` will be called.
- `LocalityStoreImpl` maintains a map `Map<Locality, LocalityLbInfo> localityMap`.
- `LocalityStoreImpl.updateLocalityStore()` will create a child balancer for each locality, with a `ChildHelper`. Then each child balancer will call `handleResolvedAddresses()`.
- `LocalityStoreImpl.updateLocalityStore()` will update `childPickers`.
- `ChildHelper.updateBalancingState()` will update `childPickers` and then delegate to parent `helper.updateBalancingState()`.
- `XdsLbState.handleSubchannelState()` will delegate to `childBalancer.handleSubchannelState()` where the subchannel belongs to the childBalancer's locality.
* make ClientLoadCounter as a separate class, added unit tests for it as it now counts quite many stats
* add MetricListener class that takes in a ClientLoadCounter and updates metric counts from received OrcaLoadReport
* refactor XdsClientLoadRecorder into XdsLoadReportStore for better integrity
* move interceptPickResult implementation to XdsLrsClient, no delegated call
* added unit test annotation
* created a StatsStore interface for better modularize LrsClient and LoadReportStore
* add more tests to ClientLoadCounter to increase coverage
* added tests for add/get/remove locality counter
* refactored tests for XdsLoadReportStore, with newly added abstract base class for ClientLoadCounter, real counter data is not involved, only stubbed snapshot is needed
* comparing doubles doing arithmetic is not recommended, but we are fine here as we are manually repeating the computation exactly
* added test case for two metric listeners with the same counter, metric values should be aggregated to the same counter
* fixed exception message and comment to only refer to interface
* removed unused variables
* cleaned up unused mock init
* removed unnecessary ClusterStats comparison helper method, as we are really comparing with the object manually created, order is deterministic
* trashed stuff for backend metrics, it should be in a separate PR
* added toString test
* remove Duration dependency in LoadReportStore
* use ThreadLocalRandom to generate positive double randoms directly
* rename XdsLoadReportStore to XdsLoadStatsStore
* rename XdsLrsClient to XdsLoadReportClient
* refactor ClientLoadSnapshot to be an exact snapshoht of ClientLoadCounter, use getters for ClientLoadSnapshot and avoid touching fields directly
* renamed XdsLoadStatsManager to XdsLoadReportClient and XdsLoadReportClient to XdsLoadReportClientImpl
* make fields final in ClientLoadSnapshot
* use a constant noop client stream tracer instead of creating new one for each noop client stream tracer factory
* rename loadReportStore for abstraction
* Revert "xds/third_party: fixed compatibility issue of regex in BSD for import.sh sed command (#5613)"
This reverts commit affce636dd.
* added comment to avoid manual change as the script is synced with internal upstream
As we are now endorsing the wrapping of ClientStreamTracers by
providing ForwardingClientStreamTracer, there is a need for altering
StreamInfo, especially CallOptions before it's passed onto the
delegate. A Builder class and a toBuilder() provides a robust way
to copy the rest of the fields.
This is a breaking change for anybody who creates StreamInfo, which is
unlikely in non-test code, because StreamInfo was added as late as
1.20.0.
* Implement LRS client with backoff. No load data is invovled yet, only for load reporting interval updates. Unit test with load report interval update and streamClosed retry.
* use a separate stopwatch to manage actual load report interval
* refactor XdsLrsClientTest
* LRS response will only receive exactly one cluster name for grpc use case
* add more XdsLrsClientTest
* change class modifier
* fixed class comment
* renamed TRAFFICDIRECTOR_HOSTNAME_FIELD
* removed self-implemented Duration util methods, instead use methods in com.google.protobuf.util.Durations
* starting LrsStream's stopwatch inside LrsStream's start method
* fixed bug of using the wrong stopwatch for XdsLrsClient retrying
* removed try-catch around request StreamObserver
* polished code by eliminating unnecessary operations
* log an error instead of crash the thread when receiving LRS response for different cluster name
* created a XdsLoadStatsManager interface, XdsLrsClient implements it
* added XdsLoadStatsStore component in XdsLrsClient
* specify thread safety in XdsLoadStatsManager
* fixed style and convention issues
* added test case for verifying recorded load data by manually crafting load data
* added thread-safety in interface specification
* minor polish with adding debug logs to LRS client
* Implemented XdsCliendLoadRecorder which is a ClientStreamTracer.Factory that takes a counter and produces ClientStreamTracer aggregating the counter in callback.
* WIP: add tests for XdsLoadReportStore
* fix query count logic, use an atomic in-progress call counter instead of callsStarted with manual computation at snapshot
* make XdsLoadReportStore threadsafe
* fix class and field modifiers
* make iterating concurrentMap threadsafe
* fixed forgetting to call delegated streamClosed
* add a method to discard ClientLoadCounter for a given locality
* add test to guard interceptPickResult does not destroy original ClientStreamTracer
* added cluster wide dropCounters and method to be called to record dropped requests, tests to be added later.
* add methods for add/discard ClientLoadCounters for localities manually instead of implicitly added by interceptPickResult call. Unit tested.
* make a static noop ClientStreamTracer and ClientStreamTracer.Factory instead of creating one each every time need it
* refractor interceptPickResult
* modified ClientLoadCounter to allow continuing recording loads for localities no longer exposed by balancer while having ongoing loads
* refractor tests
* reworded method comment for calling in syncContext
* fixed issue of no setting dropCount to 0 after load reporting
* polish test
* added test coverage for recording dropped requests (not concurrent)
* added class comment for XdsClientLoadRecorder
io.grpc has fewer dependencies than io.grpc.internal. Moving it to a
separate artifact lets users use the API without bringing in the deps.
If the library has an optional dependency on grpc, that can be quite
convenient.
We now version-pin both grpc-api and grpc-core, since both contain
internal APIs.
I had to change a few tests in grpc-api to avoid FakeClock. Moving
FakeClock to grpc-api was difficult because it uses
io.grpc.internal.TimeProvider, which can't be moved since it is a
production class. Having grpc-api's tests depend on grpc-core's test
classes would be weird and cause a circular dependincy. Having
grpc-api's tests depend on grpc-core is likely possible, but weird and
fairly unnecessary at this point. So instead I rewrote the tests to
avoid FakeClock.
Fixes#1447
This class is used in other places than just NameResolver.Helper. It
should not be an inner class of Helper.
Strictly speaking this is an API-breaking change. However, this is
part of the service config error handling API that hasn't been done
yet. Nobody has a legitimate reason to use it.
This mainly avoids protoc from 3.7.0 which has a dependency on libatomic. Most
of our systems have libatomic, so it mostly works, but the interop docker
container does not, so building fails. Version 3.7.1 was rebuilt to avoid
needing the libatomic shared library.
This has the added benefit that Bazel is now on the same version as Gradle, as
3.7.1 included fixes for Bazel.
This commit swaps to using a Sync task to place generated code in the
src/generated folder instead of the gradle-protobuf-plugin's
generatedFilesBaseDir. This provides much nicer results on failed
builds, and you will no longer see all the generated files deleted.
But at the same time the Sync task makes it easy to only copy the
grpc-generated code. This was not previously done because we were lazy
and using generatedFilesBaseDir, which made it difficult to treat the
services differently from the messages.
The LoadBalancingConfig message, which looks like
```json
{
"policy_name" : {
"config_key1" : "config_value1",
"config_key2" : "config_value2"
}
}
```
appears multiple times. It gets super tedious and confusing to handle, because both the whole config and the value (in the above example is `{ "config_key1" : "config_value1" }`) are just `Map<String, Object>`, and each user needs to do the following validation:
1. The whole config must have exactly one key
2. The value must be a map
Here I define `LbConfig` that holds the policy name and the config value, and a method in `ServiceConfigUtil` that converts the parsed JSON format into `LbConfig`.
There is also multiple cases where you need to handle a list of configs (top-level balancing policy, child and fallback policies in xds, grpclb child policies). I also made another helper method in `ServiceConfigUtil` to convert them into `List<LbConfig>`.
Found and fixed a bug in the xds code, where the top-level balancer should pass the config value (excluding the policy name), not the whole config to the child balancers. Search for "supported_1_option" in the diff to see it in the tests.
added fallback handling
in addition:
- made XdsLbState not abstract for now
- did not include graceful swapping balancers when service config change, for now just shutdown the old one and use the new one.
* Import envoy proto file to the latest internal version, which has correct java proto options. (The PGV proto, `validate.proto`, doesn't have the correct and up-to-date java_package proto option yet, but as long as we don't use those generated classes, it seems fine.)
* Stop modifying java proto options by import.sh.
* Apply shadow plugin when publishing.
For Bazel, we upgrade to protobuf 3.6.1.2 and javalite HEAD to fix
incompatibilities in newer Bazel releases.
compiler/Dockerfile is unused, so it was removed instead of being updated.
protoc no longer includes codegen for nano, so we remain on the older protoc
any time nano is used.
Protobuf now requires C++11 when compiling, so windows was swapped to
VC 14.
- defined XdsLbState, playing a similar role to GrpclbState
- there are two modes of XdsLbState: STANDARD and CUSTOM
- on `XdsLoadBalancer.handleResolvedAddressGroups()`, the `xdsLoadBalancer` will update the `xdsLbState` based on the lb config in the attributes passed in
All files other than the following are generated by `import.sh`.
```
settings.gradle
xds/build.gradle
xds/third_party/envoy/import.sh
xds/third_party/protoc-gen-validate/import.sh
```
sanjaypujare
Chengyuan Zhang
ZHANG Dapeng
Jihun Cho
Ran
Eric Anderson
liym
Kun Zhang
Carl Mastrangelo
Tim van der Lippe