Commit Graph

218 Commits

Author SHA1 Message Date
yifeizhuang 58cd6e1a7f
example: fix orca example to use new ORCA API (#9403) 2022-07-22 07:40:49 -07:00
Sergii Tkachenko 4aa9b92551
buildscripts: Fix kube contexts in the xds LB tests (#9389)
- The primary should've been `GKE_CLUSTER_PSM_LB`
- The secondary cluster was not activated for LB tests. This resulted
  in the failover test failing, as it relies on workloads running in
  different zones.
2022-07-18 17:40:46 -07:00
Sergii Tkachenko 10449d63bc Revert "buildscripts: Add missing secondary_kube_context to xds LB tests (#9380)"
This reverts commit e3e152a449.
2022-07-18 15:28:35 -07:00
Sergii Tkachenko e3e152a449
buildscripts: Add missing secondary_kube_context to xds LB tests (#9380)
Secondary cluster was not activated for LB tests. This resulted in the failover test failing, as it relies on workloads running in different zones.

ref b/238226704
2022-07-15 16:03:48 -07:00
Sergii Tkachenko d7a6c1ea31
xds: Allow Gradle to use more memory when building interop - GCE (#9354)
Same as #9347, but for GCE framework too (xds and xds_v3 jobs).

Should fix "Expiring Daemon because JVM heap space is exhausted".

PR #9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
2022-07-08 19:28:23 -07:00
Eric Anderson 5f9ef98173 xds: Allow Gradle to use more memory when building interop
Should fix "Expiring Daemon because JVM heap space is exhausted".

https://github.com/grpc/grpc-java/pull/9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
2022-07-08 12:59:27 -07:00
Eric Anderson 3de7e74c57
xds: Build third-party protos in separate build step
This dramatically shortens build time, even for full builds. A full
assemble of xds on my laptop goes from 1m 46s to 33s at least because
errorprone is disabled for the protos.
2022-07-07 07:26:38 -07:00
Sergii Tkachenko 91fcc33243
buildscripts: Fix Xmx JVM flag propagation in GRADLE_OPTS
* buildscripts: Fix Xmx JVM flag propagation in GRADLE_OPTS
* buildscripts: double Java memory allocation pool

To reduce periodic OOMs of the "GitHub Actions Linux Testing / tests (11) (pull_request)" job.
2022-06-30 14:39:03 -07:00
Eric Anderson c0790283ec
Bump protobuf to 3.21.1 (#9311)
Fixes #9264
2022-06-30 11:18:49 -07:00
sumitd2 04f9bdc1fd
compiler: Cross-compile for ppc64le platform (#9284)
Fixes #9274
2022-06-23 09:18:19 -07:00
Sergii Tkachenko 12984db6a7 kokoro: standard TESTING_VERSION in k8s framework build scripts
This Addresses the issue with skips not working due to the
missing/inconsistent `--testing_version` flag, ref b/235688697.

1. Uses the new `TESTING_VERSION` variable populated in the shared
grpc_xds_k8s_install_test_driver.sh - new approach for detecting
versions applicable to all languages.
2. Use `TESTING_VERSION` in all build files in `--testing_version` and
when tagging docker images. This will be backported to all active
test branches. Build Scripts in all other languages will be updated
as well.

Corresponding grpc core change: https://github.com/grpc/grpc/pull/30027
2022-06-21 09:08:24 -07:00
Lidi Zheng d6a23e4116 interop-test: Remove duplicated GCE tests 2022-05-25 15:56:29 -07:00
Eric Anderson 5bb721e217
interop-testing: Support syncing protos from grpc-proto
Two main incompatibilities existed in the copy of protos in grpc-proto:
no SimpleContext and an Empty method argument was replaced with a
message. "Context" is a very old word for "Metadata" back from the days
before the current gRPC protocol. We don't need that message in
particular, and well-known protos actually works in Protobuf Lite these
days, so we can swap to wrappers.proto's StringValue and don't need to
upstream a change to grpc-proto. The argument problem is fixed just by
changing the type in the Java code.

With the incompatibilities fixed, do a sync from grpc-proto and include
interop-testing.
2022-05-18 14:28:20 -07:00
Sergii Tkachenko 88770009fb
Increase memory in Linux aarch64 (emulated) builds (#9111)
Fix the issue with `Linux aarch64 (emulated)` builds failing with 

```
Expiring Daemon because JVM heap space is exhausted
Daemon will be stopped at the end of the build after running out of JVM memory
```

This fixes the build itself, however certain tests still fail.
2022-04-27 16:02:05 -07:00
Lidi Zheng d196e588a2
Add more xds_lb test cases (#8998)
This test suite is only fully enabled for C++ and Python at this
moment. This commit enables the rest of the test suite for grpc-java.

* change_backend_service_test
* failover_test
* remove_neg_test
* round_robin_test
* affinity_test
2022-03-21 12:17:26 -07:00
Lidi Zheng d71bef5afe
Redirect xDS tests to use grpc/grpc's master branch (#8982)
We are solving the issue of grpc/grpc delaying the release, causing GCE
tests to fail. Updating the master branch prevents similar cases from
happening.

Tested: [prod:grpc/java/master/branch/xds_v3](http://sponge/010f5353-e65e-4b1a-b6d3-f5a84e31546b)
2022-03-17 09:55:41 -07:00
yifeizhuang 467985e958
add kokoro config for basic tests xds k8s (#8888) 2022-02-04 11:28:13 -08:00
Eric Anderson 39cc44e38c
kokoro: Pretty test results
Previously, only Windows had the plumbing to rename test results for
the Kokoro result viewers to pretty-print.

macos.cfg was the only CI that lacked a corresponding .sh, which maked
unix.sh harder to reason about. Created macos.sh so that unix.sh is now
just a helper script and will not be called directly by Kokoro.

We now avoid "gradle clean" to avoid wiping results. Still clean compiler
since we do re-run the build multiple times with varying platforms.
Shouldn't be necessary, but "just in case" since I want this commit to
be low risk. This improves Windows to produce detailed results even
if the CI was successful.
2022-01-12 15:44:01 -08:00
sanjaypujare d7f951a9d8
all: update netty to 4.1.72.Final and tcnative to 2.0.46.Final (#8780) 2022-01-07 15:34:59 -08:00
ZHANG Dapeng 5ae6f9ca25
all: Upgrade Android plugin to 4.2.0
- bump android plugin version to 4.2.0
- migrate deprecated android.support dependencies to androidx dependencies
- bump `targetSdkVersion` to 29
- temporarily ignore lint error for 'MissingClass' due to #8799
- run android CIs with `-Pandroid.useAndroidX=true -Pandroid.enableJetifier=true` flags
- android examples are still using android.support dependencies, will not be updated in this PR.
2022-01-07 09:52:26 -08:00
Eric Anderson 80c3be0f80
Bump protobuf to 3.19.2 2022-01-06 09:08:50 -08:00
sanjaypujare e474e3dc7e
buildscripts: always make sure we are using jdk8 (#8800) 2022-01-04 13:01:02 -08:00
Lidi Zheng 80f1be03f0
Use xds-test-server-5 as the GCE interop server (#8777) 2021-12-21 16:39:31 -08:00
Eric Anderson efd968bcbb Upgrade Protobuf to 3.19.1 and Guava to 30.1.1
Protobuf uses Guava 30.1.1, so I upgrade it at the same time. It also
caused an update to rules_jvm_external and reworking the Bazel build.
Protobuf no longer requires bind() so they were dropped. Although
Protobuf's protobuf_deps() brings in rules_jvm_external, and so we don't
need to define it ourselves, it seems better to define it directly and
not depend on transitive deps since we use it directly.

Protobuf now has support for maven_install() by exposing
PROTOBUF_MAVEN_ARTIFACTS, which required reorganizing the WORKSPACE to
use maven_install() after loading protobuf. Protobuf still doesn't
define target overrides for itself so we still maintain those. When
reorganizing the WORKSPACE I noticed http_archive should ideally be
above io_grpc_grpc_java as most users will need it there, so I fixed
that since there were lots of other load()-reordering already.
2021-12-09 10:35:39 -08:00
markb74 ee581bfdfa
buildscripts: add config for building grpc-binder artifact (#8722) 2021-11-23 18:11:54 +01:00
sanjaypujare b746bab97b
buildscripts: rename xds-k8s to psm-security as part of tech-debt cleanup and name clarity (#8695) 2021-11-12 17:40:07 -08:00
Kevin Wooten 6518d7bd6d
Copy macOS x86 artifacts to aarch during upload (#8680) 2021-11-09 10:36:04 -08:00
Eric Anderson fe9026ed8a kokoro: Increase xds-k8s timeout to 3 hours
The addition of the authz tests in 0d345721 is causing the tests to
exceed their timeout. By itself, the authz test takes about an hour in
this environment. Before the authz tests, xds-k8s was taking an hour
and a half.
2021-11-05 10:04:48 -07:00
Eric Anderson 0d34572149 kokoro: Enable xds authz_test
It is successfully passing against prod:
https://source.cloud.google.com/results/invocations/e2be0996-ed4d-4a4c-90ad-20bc706f9f70/targets
2021-11-04 16:31:16 -07:00
Ivo List bb51bb6dfa
java_grpc_library.bzl: Fix parameters of java_common.compile (#7598)
Parameter host_javabase is removed.

This is preparation for flipping incompatible_java_common_parameters in
Bazel 5. See https://github.com/bazelbuild/bazel/issues/12373

Bazel versions prior to 4 require host_javabase, so are no longer supported.
2021-10-08 13:48:22 -07:00
Terry Wilson 83d36104e1
Source k8s test driver install script from core repo (#8573)
The test driver install script is read directly from the core repo master branch and the copy in the Java repo is deleted.
2021-10-07 13:05:03 -07:00
Lidi Zheng 8ac9a4e7bd [xDS interop] add Docker tagging logic to the xds_url_map job 2021-10-07 09:08:23 -07:00
markb74 f57de6bd03
Make binder instrumentation tests run on kokoro. (#8563)
The tests run as part of the existing android-interop-testing job.

We needed to modify the manifest of the apk built under android-interop-testing to declare Android Services used by the binder tests.
2021-09-29 11:28:14 +02:00
Lidi Zheng 0287d83182 Add testing_version flag 2021-09-27 15:19:41 -07:00
Jan Tattermusch 3049c2c147 reenable previously disabled aarch64 tests 2021-09-22 10:53:10 -07:00
Lidi Zheng 40f70ca3c1
Change to a non-workload-identity GKE cluster (#8461)
Part of grpc/grpc#27189 and b/198291728.

By disabling the workload identity, we should be able to run tests faster and avoid future IAM policy size issue.

Kokoro run: https://fusion2.corp.google.com/invocations/b52b1684-47de-406d-a9f6-644909755f34/targets
2021-08-31 10:35:51 -07:00
Lidi Zheng c54fcba2ee
Extend the xds_url_map job's timeout to 90 minutes (#8429)
As title. We recently had one flake caused by the Kokoro job timeout.
2021-08-20 12:12:54 -07:00
Eric Anderson 3e9488be25 buildscripts: Increase memory for Gradle in Android CI
We've still been seeing random memory-related failures with the Android
CI, but it is nowhere near as severe as it was. But even when running
locally with "-Xmx512m -XX:MaxMetaspaceSize=512m" I get failures. Our CI
environment has lots of RAM; let's use it.
2021-08-17 12:31:49 -07:00
Lidi Zheng 6a6a5279c0
Add a branch name in xds_url_map's CloudBuild (#8405) 2021-08-12 13:45:19 -07:00
Lidi Zheng 2a636420ef
Update xDS client/server image per-branch tag after build (#8400) 2021-08-11 14:01:21 -07:00
Eric Anderson 57bd087cdf buildscripts: Build android instrumentation tests in android CI
Binder's :build was missing. Cronet build failed without specifying
Java 8 because of the transitive Guava dependency.
2021-08-02 16:52:30 -07:00
Eric Anderson f781d24ddd Mostly revert "Run binderchannel android tests. (#8306)"
This partilaly reverts commit 5e18ff208a.
It leaves the compilation fix that was made to
BinderClientTransportTest.

Running instrumentation tests via firebase requires a `--app` argument.
However, we don't have such an app and it isn't immediately clear how
we'll go about making one. Revert the change to let android-testing to
start passing again.

This problem wasn't noticed before merging the original commit because
android-testing is a post-commit CI.
2021-08-02 16:13:33 -07:00
Lidi Zheng 39b7415bc0
Fix a typo in xds_url_map.cfg (#8344) 2021-07-26 16:41:34 -07:00
Lidi Zheng b3a26b732e
Add xDS k8s url-map Kokoro job (#8340)
Related PR grpc/grpc#26764
Related CL cl/386366746

This test suite will be run every 6 hours, and takes around 30 minutes. The script is copied from xds-k8s.sh, and removed the generation of server image.
2021-07-26 10:14:27 -07:00
markb74 5e18ff208a
Run binderchannel android tests. (#8306) 2021-07-14 13:10:28 +02:00
Eric Anderson 79e75bace4
Port from Travis-CI to GitHub Actions
Travis-CI no longer has a free tier (only a free trial). That was a
major reason we used Travis-CI, so that external contributors would be
able to run the CI on their forks. Iterating on a Travis config in a
personal repo was also quite convenient. The other reason was that
Travis-CI was safe to run even with untrusted code.

Since the introduction of the permissions field in workflows, GitHub
Actions appears safe to run untrusted code and has a free tier for
external contributors. GitHub Actions and Google Cloud Build are the
main contenders for a Kokoro replacement, but Cloud Build isn't safe for
untrusted code. Instead of migrating to Travis-CI.com from
Travis-CI.org, let's migrate to GitHub Actions and gain some familiarity.

I've really appreciated Travis-CI.org and have wanted to pay for it for
years but wasn't about to give it write permission to the repo. I'm
disappointed to migrate off it, now that the permissions issues have
been sorted out.
2021-06-17 13:32:56 -07:00
Sergii Tkachenko 814655cdde buildscripts: add option to use xds-k8s test driver from a fork 2021-06-16 19:09:04 -04:00
Eric Anderson 5642e01243
Replace failOnVersionConflict() with custom requireUpperBoundDeps
failOnVersionConflict has never been good for us. It is equivalent to
Maven dependencyConvergence which we discourage our users to use because
it is too tempermental and _creates_ version skew issues over time.
However, we had no real alternative for determining if our deps would be
misinterpeted by Maven.

failOnVersionConflict has been a constant drain and makes it really hard
to do seemingly-trivial upgrades. As evidenced by protobuf/build.gradle
in this change, it also caused _us_ to introduce a version downgrade.

This introduces our own custom requireUpperBoundDeps implementation so
that we can get back to simple dependency upgrades _and_ increase our
confidence in a consistent dependency tree.
2021-06-11 14:01:18 -07:00
Eric Anderson 29618a6bb6 Bump Protobuf to 3.17.2
Removed some references to the protobuf version from COMPILING.md and
compiler/README.md to make updates a tiny bit easier.
2021-06-07 11:20:48 -07:00
Menghan Li d4e90a78fd
buildscript: fold header/path matching tests to all (#8054) 2021-05-27 09:47:50 -07:00