Commit Graph

195 Commits

Author SHA1 Message Date
Eric Anderson efd968bcbb Upgrade Protobuf to 3.19.1 and Guava to 30.1.1
Protobuf uses Guava 30.1.1, so I upgrade it at the same time. It also
caused an update to rules_jvm_external and reworking the Bazel build.
Protobuf no longer requires bind() so they were dropped. Although
Protobuf's protobuf_deps() brings in rules_jvm_external, and so we don't
need to define it ourselves, it seems better to define it directly and
not depend on transitive deps since we use it directly.

Protobuf now has support for maven_install() by exposing
PROTOBUF_MAVEN_ARTIFACTS, which required reorganizing the WORKSPACE to
use maven_install() after loading protobuf. Protobuf still doesn't
define target overrides for itself so we still maintain those. When
reorganizing the WORKSPACE I noticed http_archive should ideally be
above io_grpc_grpc_java as most users will need it there, so I fixed
that since there were lots of other load()-reordering already.
2021-12-09 10:35:39 -08:00
markb74 ee581bfdfa
buildscripts: add config for building grpc-binder artifact (#8722) 2021-11-23 18:11:54 +01:00
sanjaypujare b746bab97b
buildscripts: rename xds-k8s to psm-security as part of tech-debt cleanup and name clarity (#8695) 2021-11-12 17:40:07 -08:00
Kevin Wooten 6518d7bd6d
Copy macOS x86 artifacts to aarch during upload (#8680) 2021-11-09 10:36:04 -08:00
Eric Anderson fe9026ed8a kokoro: Increase xds-k8s timeout to 3 hours
The addition of the authz tests in 0d345721 is causing the tests to
exceed their timeout. By itself, the authz test takes about an hour in
this environment. Before the authz tests, xds-k8s was taking an hour
and a half.
2021-11-05 10:04:48 -07:00
Eric Anderson 0d34572149 kokoro: Enable xds authz_test
It is successfully passing against prod:
https://source.cloud.google.com/results/invocations/e2be0996-ed4d-4a4c-90ad-20bc706f9f70/targets
2021-11-04 16:31:16 -07:00
Ivo List bb51bb6dfa
java_grpc_library.bzl: Fix parameters of java_common.compile (#7598)
Parameter host_javabase is removed.

This is preparation for flipping incompatible_java_common_parameters in
Bazel 5. See https://github.com/bazelbuild/bazel/issues/12373

Bazel versions prior to 4 require host_javabase, so are no longer supported.
2021-10-08 13:48:22 -07:00
Terry Wilson 83d36104e1
Source k8s test driver install script from core repo (#8573)
The test driver install script is read directly from the core repo master branch and the copy in the Java repo is deleted.
2021-10-07 13:05:03 -07:00
Lidi Zheng 8ac9a4e7bd [xDS interop] add Docker tagging logic to the xds_url_map job 2021-10-07 09:08:23 -07:00
markb74 f57de6bd03
Make binder instrumentation tests run on kokoro. (#8563)
The tests run as part of the existing android-interop-testing job.

We needed to modify the manifest of the apk built under android-interop-testing to declare Android Services used by the binder tests.
2021-09-29 11:28:14 +02:00
Lidi Zheng 0287d83182 Add testing_version flag 2021-09-27 15:19:41 -07:00
Jan Tattermusch 3049c2c147 reenable previously disabled aarch64 tests 2021-09-22 10:53:10 -07:00
Lidi Zheng 40f70ca3c1
Change to a non-workload-identity GKE cluster (#8461)
Part of grpc/grpc#27189 and b/198291728.

By disabling the workload identity, we should be able to run tests faster and avoid future IAM policy size issue.

Kokoro run: https://fusion2.corp.google.com/invocations/b52b1684-47de-406d-a9f6-644909755f34/targets
2021-08-31 10:35:51 -07:00
Lidi Zheng c54fcba2ee
Extend the xds_url_map job's timeout to 90 minutes (#8429)
As title. We recently had one flake caused by the Kokoro job timeout.
2021-08-20 12:12:54 -07:00
Eric Anderson 3e9488be25 buildscripts: Increase memory for Gradle in Android CI
We've still been seeing random memory-related failures with the Android
CI, but it is nowhere near as severe as it was. But even when running
locally with "-Xmx512m -XX:MaxMetaspaceSize=512m" I get failures. Our CI
environment has lots of RAM; let's use it.
2021-08-17 12:31:49 -07:00
Lidi Zheng 6a6a5279c0
Add a branch name in xds_url_map's CloudBuild (#8405) 2021-08-12 13:45:19 -07:00
Lidi Zheng 2a636420ef
Update xDS client/server image per-branch tag after build (#8400) 2021-08-11 14:01:21 -07:00
Eric Anderson 57bd087cdf buildscripts: Build android instrumentation tests in android CI
Binder's :build was missing. Cronet build failed without specifying
Java 8 because of the transitive Guava dependency.
2021-08-02 16:52:30 -07:00
Eric Anderson f781d24ddd Mostly revert "Run binderchannel android tests. (#8306)"
This partilaly reverts commit 5e18ff208a.
It leaves the compilation fix that was made to
BinderClientTransportTest.

Running instrumentation tests via firebase requires a `--app` argument.
However, we don't have such an app and it isn't immediately clear how
we'll go about making one. Revert the change to let android-testing to
start passing again.

This problem wasn't noticed before merging the original commit because
android-testing is a post-commit CI.
2021-08-02 16:13:33 -07:00
Lidi Zheng 39b7415bc0
Fix a typo in xds_url_map.cfg (#8344) 2021-07-26 16:41:34 -07:00
Lidi Zheng b3a26b732e
Add xDS k8s url-map Kokoro job (#8340)
Related PR grpc/grpc#26764
Related CL cl/386366746

This test suite will be run every 6 hours, and takes around 30 minutes. The script is copied from xds-k8s.sh, and removed the generation of server image.
2021-07-26 10:14:27 -07:00
markb74 5e18ff208a
Run binderchannel android tests. (#8306) 2021-07-14 13:10:28 +02:00
Eric Anderson 79e75bace4
Port from Travis-CI to GitHub Actions
Travis-CI no longer has a free tier (only a free trial). That was a
major reason we used Travis-CI, so that external contributors would be
able to run the CI on their forks. Iterating on a Travis config in a
personal repo was also quite convenient. The other reason was that
Travis-CI was safe to run even with untrusted code.

Since the introduction of the permissions field in workflows, GitHub
Actions appears safe to run untrusted code and has a free tier for
external contributors. GitHub Actions and Google Cloud Build are the
main contenders for a Kokoro replacement, but Cloud Build isn't safe for
untrusted code. Instead of migrating to Travis-CI.com from
Travis-CI.org, let's migrate to GitHub Actions and gain some familiarity.

I've really appreciated Travis-CI.org and have wanted to pay for it for
years but wasn't about to give it write permission to the repo. I'm
disappointed to migrate off it, now that the permissions issues have
been sorted out.
2021-06-17 13:32:56 -07:00
Sergii Tkachenko 814655cdde buildscripts: add option to use xds-k8s test driver from a fork 2021-06-16 19:09:04 -04:00
Eric Anderson 5642e01243
Replace failOnVersionConflict() with custom requireUpperBoundDeps
failOnVersionConflict has never been good for us. It is equivalent to
Maven dependencyConvergence which we discourage our users to use because
it is too tempermental and _creates_ version skew issues over time.
However, we had no real alternative for determining if our deps would be
misinterpeted by Maven.

failOnVersionConflict has been a constant drain and makes it really hard
to do seemingly-trivial upgrades. As evidenced by protobuf/build.gradle
in this change, it also caused _us_ to introduce a version downgrade.

This introduces our own custom requireUpperBoundDeps implementation so
that we can get back to simple dependency upgrades _and_ increase our
confidence in a consistent dependency tree.
2021-06-11 14:01:18 -07:00
Eric Anderson 29618a6bb6 Bump Protobuf to 3.17.2
Removed some references to the protobuf version from COMPILING.md and
compiler/README.md to make updates a tiny bit easier.
2021-06-07 11:20:48 -07:00
Menghan Li d4e90a78fd
buildscript: fold header/path matching tests to all (#8054) 2021-05-27 09:47:50 -07:00
Sergii Tkachenko 976daf2dd0 buildscripts: switch xds-k8s cluster to 1.20.x 2021-05-05 20:01:15 -04:00
Lidi Zheng d2160ea703
Extend the xDS interop tests timeout to 360 mins (#8133) 2021-05-05 10:18:44 -07:00
Jan Tattermusch 72527708f5
skip flaky :grpc-xds:test in linux aarch64 tests (#8119)
Due to #8118 the :grpc-xds:test currently doesn't provide useful signal.
2021-04-27 10:18:39 -07:00
Sergii Tkachenko fcbc1abc44 buildscripts: xds-k8s pin pip to 21.0.1
pip 21.1 released on Apr 24 introduced a regression for python 3.6.1.
The regression was identified on Apr 24, the fix merged on Apr 25.
The fix is expected to be delivered in the 21.1.1 patch.

There's no clear date, when 21.1.1 will be released.
Until then, pin is temporarily pinned to the previous release, 21.0.1.
2021-04-26 18:15:18 -04:00
Jan Tattermusch b436d0dfb7
Improve emulated linux aarch64 tests, without protoc artifact build
Based on my testing, the tests are currently flaky, so it would be better to first merge it and start running it continuously to asses the situation.
2021-04-26 10:54:28 -07:00
Lidi Zheng 134e9cbc42
buildscripts: enable CSDS test 2021-04-21 20:48:34 -04:00
Sergii Tkachenko 278a336d1f buildscript: xds-k8s increase build timeout 2021-04-08 22:05:29 -04:00
Sergii Tkachenko c113ba1030 buildscript: add xds-k8s cluster endpoint override
Missed this in 1b86618ce9
2021-04-08 21:14:08 -04:00
Sergii Tkachenko 1b86618ce9 buildscript: use different xds-k8s cluster
In preparation to the Public Preview.
2021-04-08 19:00:34 -04:00
ZHANG Dapeng e4ccf0eeac
buildscript: enable fault injection interop test 2021-03-17 17:41:58 -07:00
Doug Fawley 3752b9e365
add timeout test case to xds interop tests (#7888) 2021-02-12 14:25:20 -08:00
Eric Gribkoff 0ffd942fd2
buildscripts: increase xds job timeouts (#7831) 2021-01-25 04:33:01 -08:00
ZHANG Dapeng 150f994315
buildscript: fix execution path of xds.sh in xds_v3.sh 2021-01-20 09:21:36 -08:00
ZHANG Dapeng c1f9658a72
buildscripts: fix buildscript unbound variable error 2021-01-19 18:33:55 -08:00
ZHANG Dapeng 56fc8b8626
buildscripts: add xds_v3 buildscripts 2021-01-19 15:01:21 -08:00
Jan Tattermusch 76c5a651a1
Add a simple way to run grpc-java test inside emulated ARM64 docker container 2021-01-19 09:05:22 -08:00
Sergii Tkachenko e5f57adb7f
buildscripts: xDS Kubernetes Interop tests buildscript 2020-12-23 09:52:01 -05:00
Chengyuan Zhang 1e49ac665a
buildscript: ensure dependency convergence (#7745)
Run grpc-all:dependency task in Kokoro. For some unknown reason, building grpc-all doesn't reliably catch dependency conflicts.
2020-12-22 09:09:11 -08:00
Eric Anderson 8ce6355e89 buildscripts: Combine android builds together
Previously the android projects were separate from the main build and
each other. For quite a while now they have been integrated in the main
project. There's no longer any need to build each separately.
2020-12-07 15:01:51 -08:00
Chengyuan Zhang 71a3c55177
buildscripts: add missing CI coverage for examples (#7708)
Adds CI coverage for building example/android/strictmode and examples/example-jwt-auth. Also cleans up existing Gradle and Maven build command in the CIs.
2020-12-07 14:43:32 -08:00
Eric Anderson f13471d68d buildscripts: Check for out-of-date codegen after android builds
Issues like that fixed in 05048cf3 should have failed the CI, like it
does for the rest of the build (on both Travis and linux_artifacts).
2020-12-07 11:56:06 -08:00
Eric Anderson ee9286a0f2 buildscripts: Fix grpc-java-artifacts indentation
Re-sort packages, now that the tabs aren't skewing the sorting.

I'm quite confident I had fixed this already, but I had multiple copies
of this file on multiple machines and must have fixed the wrong copy.
2020-12-04 16:42:26 -08:00
Eric Anderson 486d82a7e1 Build Linux artifacts using CentOS 7
CentOS 6 is dead and no longer has update servers. CentOS 7 is older
than Debian 9 (oldstable), so binaries hopefully work on both. More
testing is necessary, but everything's broken now, so this is better
than nothing.

We stop using protoc-artifacts because now the container is
straight-forward enough that we can just use our own. Previously the
"devtoolset" stuff made us want to share the container.
2020-12-03 22:56:31 -08:00