Commit Graph

921 Commits

Author SHA1 Message Date
Anirudh Ramachandra fec7c2ea4a
Add fix for xdstp replacement for encoded authorities (#10571)
In ac35ab6 the logic in xDS Name resolver was changed to support encoded
authorities. This seems to cause an issue for xdstp replacements which
would percent encode the authority for the replacement causing double
encoding.

For example:

URI = xds:///path/to/service
Authority = path%2Fto%2Fservice
xdstp resource = xdstp:///envoy.config.listener.v3.Listener/path%252Fto%252Fservice

Here the authority is encoded due to slashes and during replacement we
percent encode it again causing %2F to change to %252F. To avoid this
issue, use the encoded authority only for the getServiceAuthority() API
and for all other use cases retain the unencoded authority.
2023-09-22 08:25:08 -07:00
Larry Safran e1334eae7b
Change Round Robin and WeightedRoundRobin into petiole policies (#10528)
* Change Round Robin and WeightedRoundRobin into petiole policies
2023-09-15 10:27:36 -07:00
Eric Anderson 67f4eabb42 xds: In tests, verify WRR static stride does not iterate too many times 2023-08-18 09:59:33 -07:00
Tony An 97f4f8687c
xds: WRR scheduler clips weights (#10480) 2023-08-17 17:11:55 -07:00
Eric Anderson 3e44bbfe4a Exclude Internal classes from javadoc 2023-08-16 15:38:30 -07:00
Eric Anderson e6c77a368a xds: Add missing ExperimentalApi to OrcaServiceImpl 2023-08-16 11:24:59 -07:00
Eric Anderson b5d7f1394d xds: Fix import scripts deleting the wrong files, use of trap, and full git clone
The scripts used `git rev-parse --show-toplevel` so it appeared they
could be used from any directory. But references to "GIT_BASE_DIR"
weren't absolute, so it did matter the starting directory. And it
mattered in a big way for xds/import.sh as if you ran it from the
grpc-java directory it would delete the xds directory in grpc-java, not
third_party.

The trap that deleted the GIT_BASE_DIR was very broken. In addition to
potentially deleting the wrong directory, it was unnecessary because
that directory was in tmpdir. But you can only have one trap per signal,
so this unnecessary trap disabled the trap that deleted tmpdir.

The script needed a full clone because it needed to check out a specific
commit. To work with --depth 1 you have to use some convoluted syntax.
But just downloading a tar.gz is easy and seems should work fine on Mac.

protoc-gen-validate/import.sh didn't have the trap problem, but seemed
to have drifted from the other scritps. All the scripts were synced to
match.
2023-08-15 16:54:32 -07:00
Mohan Li 5850de2f5f
pick_first: de-experiment pick first (#10475) 2023-08-15 15:48:24 -07:00
Larry Safran a0d8f2eb31
Create a shared LB base class for LBs with multiple children (#10448)
* Create a shared LB base class for LBs with multiple children and change ClusgterManagerLoadBalancer to use it.
2023-08-08 14:36:27 -07:00
Tony An 40bff673c0
xds: replace random with atomic sequence in WRR (#10458) 2023-08-07 16:19:50 -07:00
Halvard Skogsrud 4dc786b5be
Fix locality logging (#10423)
The bootstrapping code currently does not log zone and subZone from locality correctly, and only logs region. This commit fixes the logging message format.
2023-07-31 17:44:16 -07:00
Tony An 3c19325587
removed assert for flakey static stride scheduler with multiple threads (#10437)
on master branch
2023-07-31 14:03:00 -07:00
Larry Safran 621fbad78c
Reduce test flakiness (#10429) 2023-07-28 14:55:37 -07:00
sanjaypujare 41552bfd9a
all: generate automatic module name in the manifest (#10413) 2023-07-25 09:00:11 -07:00
Sergii Tkachenko 419767f968
Upgrade dependencies post v1.57.x branch cut (#10359)
### Dependency updates

#### Update successfully to the latest
- [x] `androidx.core:core 1.10.0 -> 1.10.1`
- [x] `com.google.api.grpc:proto-google-common-protos 2.17.0 -> 2.22.0`
- [x] `com.google.cloud:google-cloud-logging 3.14.5 -> 3.15.5`
- [x] `com.google.truth:truth 1.0.1 -> 1.1.5`
- [x] `com.puppycrawl.tools:checkstyle 8.28 -> 10.12.1`
- [x] `org.robolectric:robolectric 4.9.2 -> 4.10.3`
- [x] Auto-value
   - [x] `com.google.auto.value:auto-value 1.10.1 -> 1.10.2`
   - [x] `com.google.auto.value:auto-value-annotations 1.10.1 -> 1.10.2`
- [x] Protobuf
   - [x] `com.google.protobuf:protobuf-java 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protobuf-java-util 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protobuf-javalite 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protoc 3.22.3 -> 3.23.4`
- [x] Errorprone
   - [x] `com.google.errorprone:error_prone_annotations 2.18.0 -> 2.20.0`
   - [x] `com.google.errorprone:error_prone_core 2.18.0 -> 2.20.0`
   - ~`libs.checkstylejava8 = com.puppycrawl.tools:checkstyle 9.3 -> 10.12.1`~ -- pinned to last version supporting java8, update not needed

#### Updated to non-latest
- [x] `com.squareup.okio:okio 1.17.5 ->` ~`3.4.0`~ `2.10.0` -- updating to 3.x failed due to them [introducing gradle multiplatform artifacts](https://github.com/square/okio/blob/master/CHANGELOG.md#version-320). Error in [the comment below](https://github.com/grpc/grpc-java/pull/10359#issuecomment-1632853307.).
- [x] `org.checkerframework:checker-qual 3.33.0 ->` ~`3.36.0`~ -- removed, no longer needed
- [x] Mockito - updated to `4.11.0`. Versions 5.x break some tests. Errors in [the comment below](https://github.com/grpc/grpc-java/pull/10359#issuecomment-1632834435).
   - `org.mockito:mockito-android 3.12.4 ->` ~`5.4.0`~ `4.11.0`
   - `org.mockito:mockito-core 3.12.4 ->` ~`5.4.0`~ `4.11.0`

#### Not updated
- Cronet -- upgrade failed, created a blocker ticket https://github.com/grpc/grpc-java/issues/10396
   - ~`org.chromium.net:cronet-api 108.5359.79 -> 113.5672.61`~
   - ~`org.chromium.net:cronet-embedded 108.5359.79 -> 113.5672.61`~
- Google auth -- upgrade blocked by https://github.com/grpc/grpc-java/pull/9118
   - ~`com.google.auth:google-auth-library-credentials 1.4.0 -> 1.19.0`~
   - ~`com.google.auth:google-auth-library-oauth2-http 1.4.0 -> 1.19.0`~
- Guava -- not upgraded: 32.1.0 [broke gradle modules](https://github.com/google/guava/releases/tag/v32.1.0)
   - ~`libs.guava = com.google.guava:guava 32.0.1-android -> 32.1.1-jre`~
   - ~`libs.guava.jre = com.google.guava:guava 32.0.1-jre -> 32.1.1-jre`~
   - ~`libs.guava.testlib = com.google.guava:guava-testlib 32.0.1-android -> 32.1.1-jre`~


### checkForUpdates results
```
> Task :checkForUpdates
androidx.core:core 1.10.0 -> 1.10.1
com.google.auto.value:auto-value 1.10.1 -> 1.10.2
com.google.auto.value:auto-value-annotations 1.10.1 -> 1.10.2
org.checkerframework:checker-qual 3.33.0 -> 3.36.0
com.puppycrawl.tools:checkstyle 8.28 -> 10.12.1
org.chromium.net:cronet-api 108.5359.79 -> 113.5672.61
org.chromium.net:cronet-embedded 108.5359.79 -> 113.5672.61
com.google.errorprone:error_prone_annotations 2.18.0 -> 2.20.0
com.google.errorprone:error_prone_core 2.18.0 -> 2.20.0
com.google.errorprone:error_prone_core 2.10.0 -> 2.20.0
com.google.api.grpc:proto-google-common-protos 2.17.0 -> 2.22.0
com.google.auth:google-auth-library-credentials 1.4.0 -> 1.19.0
com.google.auth:google-auth-library-oauth2-http 1.4.0 -> 1.19.0
com.google.cloud:google-cloud-logging 3.14.5 -> 3.15.5
com.google.guava:guava 32.0.1-android -> 32.1.1-jre
com.google.guava:guava 32.0.1-jre -> 32.1.1-jre
com.google.guava:guava-testlib 32.0.1-android -> 32.1.1-jre
org.mockito:mockito-android 3.12.4 -> 5.4.0
org.mockito:mockito-core 3.12.4 -> 5.4.0
com.squareup.okio:okio 1.17.5 -> 3.4.0
com.google.protobuf:protobuf-java 3.22.3 -> 3.23.4
com.google.protobuf:protobuf-java-util 3.22.3 -> 3.23.4
com.google.protobuf:protobuf-javalite 3.22.3 -> 3.23.4
com.google.protobuf:protoc 3.22.3 -> 3.23.4
org.robolectric:robolectric 4.9.2 -> 4.10.3
com.google.truth:truth 1.0.1 -> 1.1.5
```
2023-07-21 10:15:05 -07:00
Larry Safran 864d30caed
Handle header with errors and endStream = true (#10384)
* Eliminate NPE by skipping further processing when stream is defined, but doesn't have a property for streamKey (header processing identified an error)

Fixes #10364

* Add unit test for missing content type
2023-07-20 15:31:46 -07:00
sanjaypujare 0f5f07f876
core, inprocess, util: move inprocess and util code into their own new artifacts grpc-inprocess and grpc-util (#10362)
* core, inprocess, util: move inprocess and util code into their own new artifacts grpc-inprocess and grpc-util
2023-07-17 11:45:31 -07:00
Anirudh Ramachandra ac35ab67f2
xds: Encode the service authority in XdsNameResolver (#10207)
Encode the service authority before passing it into gRPC util in the xDS name resolver to handle xDS requests which might contain multiple slashes. Example: xds:///path/to/service:port.

As currently the underlying Java URI library does not break the encoded authority into host/port correctly simplify the check to just look for '@' as we are only interested in checking for user info to validate the authority for HTTP.

This change also leads to few changes in unit tests that relied on this check for invalid authorities which now will be considered valid.

Just like #9376, depending on Guava packages such as URLEscapers or PercentEscapers leads to internal failures(Ex: Unresolvable reference to com.google.common.escape.Escaper from io.grpc.internal.GrpcUtil). To avoid these issues create an in house version that is heavily inspired by grpc-go/grpc.
2023-07-14 12:54:18 -07:00
Tony An 37351c690c
fixed StaticStrideScheduler looping too many times (#10370) 2023-07-13 10:24:07 -07:00
Daniel Liu 4fa2814d65
services, xds, orca: LRS named metrics support (#10282)
Implements gRFC A64: xDS LRS Custom Metrics Support
2023-07-11 13:29:31 -07:00
Larry Safran 8dbd47ceb5
Sort the policies in a rule by policy name when parsing from proto. (#10334)
* Sort the policies in a rule by policy name when parsing from proto.  This fixes the server sending a GOAWAY when an LDS update with no changes other than ordering is received.

* Remove use of deprecated method setSourceIp

* Fix style issues

* Update RbacFilterTest.java
2023-07-10 11:07:54 -07:00
Tony An 0b53dd7304
implemented and tested static stride scheduler for weighted round robin load balancing policy (#10272) 2023-07-06 10:03:08 -07:00
Larry Safran ed4cc896fc
require EDS service name in CDS resources with xdstp name (#10329)
* require EDS service name in CDS resources with xdstp name
2023-06-30 13:51:46 -07:00
Philip K. Warren 3808e707f9
compiler: Use fully qualified String in codegen (#10321)
Currently, the gRPC compiler isn't properly using the fully qualified
string name `java.lang.String` instead of `String`. Update the generator
to use the `$String$` alias to avoid compile issues with protobuf
messages called String.

Fixes #10316.
2023-06-29 10:50:13 -07:00
Eric Anderson b2327238c5 xds: Include third_party sources is sources jar 2023-06-27 09:48:54 -07:00
Eric Anderson ac8908ec7f xds: Plumb thirdparty classes through sourceSet instead of jar
Plumbing through sourceSet lets cross-project dependencies work the same
way as artifacts published to Maven. This fixes an issue for
interop-testing where build/install would include all the raw files from
thirdparty in addition to the grpc-xds.jar. For example:
build/install/grpc-interop-testing/lib/com/github/xds/data/orca/v3/OrcaLoadReport$1.class

b/288577812
2023-06-23 13:27:29 -07:00
Eric Anderson f46793e82d Remove unneeded dependency excludes
Since 44847bf4e, when we upgraded our JUnit version, the JUnit
exclusions have probably not been necessary. e0ac97c4f upgraded
Robolectric to a version that had the auto.service problem fixed.
2023-06-23 11:18:10 -07:00
Terry Wilson 31321562ee
xds: Use Rule order instead of RuleChain (#10289)
This is now the preferred way to specify rule execution order.
2023-06-16 13:39:47 -07:00
Larry Safran 99392004a0
Handle Aggregate cluster loops and dups (#10274)
* Suppress duplicate children and NACK if detect loops (child is an ancestors of the current CDS aggregate).  

* Handle diamond shaped aggregations (same cluster appears under 2 distinct parents that doesn't create a loop).
2023-06-15 10:24:23 -07:00
Daniel Liu 1daf0ff720
services, xds, orca: use application_utilization and fallback to cpu_utilization if unset in WRR (#10256)
Implements updates to [A51][] and [A58][].

Imported cncf/xds using import.sh script.

A51: https://github.com/grpc/proposal/pull/374
A58: https://github.com/grpc/proposal/pull/373
2023-06-08 16:03:50 -07:00
Eric Anderson 33a402d924 Avoid NettyServerBuilder when unnecessary
These code locations just needed a generic gRPC server, without any
transport-specific configuration. Use vanilla ManagedServerBuilder
instead of hard-coding to Netty, as we would suggest to our users.
2023-06-05 17:11:16 -07:00
Larry Safran 217103cd5e
Flip default for RLS on XDS being enabled to true. (#10248) 2023-06-05 15:49:30 -07:00
sanjaypujare 639fd8edf2
xds: remove env var for security enable/disable (#10243) 2023-06-02 09:24:51 -07:00
Daniel Liu 5a27e3ead2
services, xds, orca: support EPS in client-side WRR (#10177) 2023-05-26 12:52:42 -07:00
sanjaypujare e875d1b01c
xds: remove remaining occurences of SDS in the security code (#10219) 2023-05-25 10:28:37 -07:00
Terry Wilson 810d8cab09
xds: pick_first LB configuration (#10181)
Configure pick_first load balancing if the xDS PickFirst LB config
message is used in the Cluster load balancing config.
2023-05-17 10:19:29 -07:00
Eric Anderson 29b8483fd6
Use test fixtures instead of sourceSets.test.output
This avoids the (often missing) evaluationDependsOn and fixes using
results from other projects without propagating those through
Configuration. It also reduces the number of useless classes pulled in
by down-stream tests, reducing the probability of rebuilds.

The expectation of fixtures is they help testing down-stream code that
use the classes in main. That applies to all the classes here except for
FakeClock and StaticTestingClassLoader. It would also apply to many
internal classes in grpc-testing, but let's consider cleaning that up
future work.
2023-05-16 12:10:13 -07:00
Eric Anderson 2cc24a9bda gradle: Remove version pinning from POMs
The pinning is unreliable in Maven and ignored by Gradle. I'm not at all
convinced that we are pinning/not pinning in appropriate projects. The
pinning also serves less of a purpose since we started encouraging the
BOM and grpc-netty-shaded. Netty's HTTP/2 API has also become somewhat
stable compared to its earlier history. If we notice an up-tick in
version skew, we can reinstate it.

The pinning is annoying in the build.gradle code and causes Maven/Gradle
to download the version list once a day, which can be troublesome to
users unaware of how to tell the tools to work offline.

It also opens our users to platform issues like seen in #10043
and #10086 where Maven Central's version list was incorrectly generated.
Or like #9664 where Gradle Plugin's repository caches packages from
JCenter but the version list is not as cachable so exposed us to JCenter
instability.

This fixes #8357, by way of "we think we won't worry any more." See
90db93b9 when it was originally introduced. And issues
like #8337, #3634.
2023-05-15 13:08:40 -07:00
Terry Wilson 793fde840a
xds: Import latest Envoy protos (#10167) 2023-05-11 09:44:43 -07:00
Daniel Liu 6e486cba48
import cncf/xds using import.sh script (#10160) 2023-05-10 12:42:26 -07:00
Eric Anderson 02a4cb5c69 Fix throwing LBProvider.parseLoadBalancingConfig() impls
LoadBalancers in general should never throw, but
parseLoadBalancingConfig() in particular has a return value to
communicate the error. Throwing can be a bit unpredictable, but at its
most trivial form causes a channel panic. There's no reason to throw
explicitly and calls to JsonUtil have to be protected by a try-catch
because it can throw.
2023-05-10 12:42:05 -07:00
yifeizhuang 6f804331f7
xds: remove _experimental in wrr policy name (#10162) 2023-05-10 11:02:13 -07:00
Eric Anderson 74b515ecf7 Migrate many usages of TestUtils.loadCert() to the public TlsTesting
TlsTesting.loadCert() is a public API and so should be preferred over
our internal utility. It avoids creating a temp file that has to be
deleted by a shutdown hook. Usages that needed a file were not migrated.
2023-05-09 17:01:31 -07:00
Eric Anderson f229aed538 core: Hide RR's ReadyPicker and EmptyPicker
ReadyPicker hasn't been necessary since 111ff60e, when we stopped
calling super.pickSubchannel(). EmptyPicker was only used in tests, and
we can just compare the class name instead of doing an instanceof check.

Unfortunately, calling getClass() caused Java to start casting the
return value of pickerCaptor.getValue() based on its generics. Captors
don't verify the type they capture, so using any type other than
SubchannelPicker for the pickerCaptor is misleading and hides a cast.
2023-05-09 13:25:19 -07:00
Eric Anderson 847ea7cfc9 Upgrade Mockito to 3.12.4
MockitoAnnotations.initMocks() is deprecated.
2023-05-08 16:39:42 -07:00
Eric Anderson 180b3e9092 xds: Avoid default locale and reset()
Found via linter
2023-05-05 15:23:18 -07:00
Eric Anderson 94dec7c020 xds: Avoid WRR config synchronization 2023-05-05 14:43:26 -07:00
Terry Wilson b5800edd76
Importing latest xDS protos from envoyproxy/envoy (#10123) 2023-05-03 11:57:52 -07:00
sanjaypujare f24a5a7d0d
xds: handle the handlerRemoved callback to skip updateSslContext processing (#10118)
* xds: handle the handlerRemoved callback to skip updateSslContext processing
     In handlerAdded we submit a callback to updateSslContext but before the
     callback is executed the handler could be removed (e.g. bad connection)
     in which case the callback should skip all of the processing.
     Also added a unit test to check there is no exception.
2023-05-03 07:41:15 -07:00
yifeizhuang b249a5340e
services: allow cpu utilization greater than 1 (#10107) 2023-04-28 11:59:10 -07:00