Before this change, in EDS balancer, child balancer's state update is handled synchronously, which includes priority handling. This would cause a deadlock if the child policy sends a state update inline when handling addresses (e.g. when roundrobin handles empty address list).
This change moves the child balancer state handling into a goroutine, to avoid the problem.
- More logs in xds bootstrap/resolver/cds/eds
- Bootstrap file content/error
- Request/response on ADS stream
- Actions by client/resolver/balancer
- Content of updates
- Logs prefixed with component name and id
- `[xds-bootstrap]`
- `[xds-client <address>]`
- `[cds-lb <address>]`
- `[eds-lb <address>]`
- The LRS client will find known clusters from the response, and send the loads
- The LRS client support only one cluster now, will be extended to support multiple
* Modified tests to use tlogger.
* Fail on errors, with error expectations.
* Added expects and MixedCapsed grpclb_config tests
* Moved tlogger to grpctest, moved leakcheck tester to grpctest.go
* Added ExpectErrorN()
* Removed redundant leak checks
* Fixed new test
* Made tlogger globals into tlogger methods
* ErrorsLeft -> EndTest
* Removed some redundant lines
* Fixed error in test and empty map in EndTest
edsBalancer (the old xds balancer) was in `package balancer`, one level above the eds implementation. It's a thin wrapper of the eds impl (and fallback in the future).
This change moves the thin wrapper to `package edsbalancer`, and also renames some structs.
Simplified the tests by only testing what is required and faking out
whatever can be faked out.
Also added a fakexds.Server implementation. Will switch other users of
the existing fakeserver implementation after this PR is merged.
Errors will be handled specifically, depending on whether it's a
connection error or other types of errors.
Without this fix, balancer's callback will be called with <nil> update,
causing nil panic later.
This PR removes the xds_client implementation from eds balancer, and replaces it with a xds_client wrapper. (The xds_client wrapper has very similar API as the old xds_client implementation, so the change in the eds balancer is minimal).
The eds balancer currently doesn't look for xds_client from attributes, and always creates a new xds_client. The attributes change will be done in a following up change.
The xds client will parse the EDS response, and give the parse result to eds balancer, so the balancer doesn't need to deal with proto directly.
Also moved `ClusterLoadAssignmentBuilder` to another pacakge to be shared by tests in different packages.
Generated protobuf messages contain internal data structures
that general purpose comparison functions (e.g., reflect.DeepEqual,
pretty.Compare, etc) do not properly compare. It is already the case
today that these functions may report a difference when two messages
are actually semantically equivalent.
Fix all usages by either calling proto.Equal directly if
the top-level types are themselves proto.Message, or by calling
cmp.Equal with the cmp.Comparer(proto.Equal) option specified.
This option teaches cmp to use proto.Equal anytime it encounters
proto.Message types.
Each priority maps to a balancer group.
When a priority is in use, its balancer group is started, and it will close the balancer groups with lower priorities. When a priority is down (no connection ready), it will start the next priority balancer group.
Fields are added in: https://github.com/grpc/grpc-proto/pull/64
Other changes:
- Move XDSConfig from internal to balancer
- Later we will add a separate config for CDS balancer
- generate service_config.pb.go and test with json generated from proto message
When a locality is removed from EDS response, it's corresponding
sub-balancer will be removed from balancer group.
With this change, the sub-balancer won't be removed immediately. It will
be kept in a cache (for 15 minutes by default). If the locality is
re-added within the timeout, the sub-balancer in cache will be picked
and re-used.
* Implement missing pieces for connection backoff.
Spec can be found here:
https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md
Summary of changes:
* Added a new type (marked experimental), ConnectParams, which contains
the knobs defined in the spec (except for minConnectTimeout).
* Added a new API (marked experimental), WithConnectParams() to return a
DialOption to dial with the provided parameters.
* Added new fields to the implementation of the exponential backoff in
internal/backoff which mirror the ones in ConnectParams.
* Marked existing APIs WithBackoffMaxDelay() and WithBackoffConfig() as
deprecated.
* Added a default exponential backoff implementation, for easy use of
internal callers.
Added a new backoff package which defines the backoff configuration
options, and is used by both the grpc package and the internal/backoff
package. This allows us to have all backoff related options in a
separate package.
* Incorporate changes to bootstrap file format.
The format of the bootstrap file will be as follows:
{
"xds_server": {
"server_uri": <string containing URI of xds server>,
// List of channel creds; client will stop at the first type it
// supports.
"channel_creds": [
{
"type": <string containing channel cred type>,
// The "config" field is optional; it may be missing if the
// credential type does not require config parameters.
"config": <JSON object containing config for the type>
}
]
},
"node": <JSON form of Node proto>
}
- Also, the bootstrap file will be read everytime a new xDS client is created.
- Change NewConfig() to not return error. Instead it just returns with
certain fields left unspecified if it encounters errors.
- Do not fail the bootstrap process if we see unknown fields in the
bootstrap file.
This is a preparing change to support priority failover. It adds start() and close() to balancer group, so we can have a balancer group that's not in use, but has all the data and is ready to be started (think about a lower priority when the higher priority is in use).
A balancer group is split into two parts: static and dynamic:
static: the data from EDS, and gets updated even if balancer group is closed
balancer IDs and builders, addresses for each balancer
dynamic: the sub-balancers
These are only created when the balancer group is started. They are closed when the balancer group is closed.
And only when the balancer group is started, the sub-balancers will get address updates.
As of this implementation, the bootstrap file will be provided in the
GRPC_XDS_BOOTSTRAP environment variable and will be read by the xDS
balancer. The file will be in JSON form with two top-level entities: a
Node proto and an ApiConfigSource proto. The overall JSON format will
be:
{
"node": <Node proto>,
"xds_server": <ApiConfigSource proto>
}
In the ApiConfigSource proto, We will support only one grpc_services
entry. The api_type field must be GRPC.
As for creds, we will use default TLS creds (which will use CAs from
system default directories, and will not use any client certificates)
for transport and default compute engine creds for call credentials.
This will evolve into something more configurable in the future.
Rename TestXdsBalanceHandleResolvedAddrs as TestXdsFallbackResolvedAddrs
and add comments to explain what the test is doing. Also added some
other comments and made minor formatting changes.
This resolver doesn't do much at this point, except returning an empty
address list and a hard-coded service config which picks the xds
balancer with a round_robin child policy.
Also moved the xdsConfig struct to the xds/internal package and exported
it as LBConfig, so that both the resolver and the balancer packages can
make use of this.
We will have a root level xds/ directory which will eventually contain
all xDS implementation including balancer, resolver, client etc.
The new structure looks something like this:
grpc/
|
+--xds/
|
+--internal/
| |
| +--balancer/
| |
| +--edsbalancer/
| |
| +--lrs/
| |
| +--orca/
|
+--experimental/
Users need to import grpc/xds/experimental package to get all xds
functionality, and this will eventually be moved to grpc/xds.
Also, moved grpc/balancer/internal/wrr to grpc/internal/wrr.
Menghan Li