Commit Graph

99 Commits

Author SHA1 Message Date
Eric Anderson 186409f868
buildscripts: Add script to sync .protos from grpc-proto
Currently only protos in alts, grpclb, and services are synced. Once
some java_package options are "upstreamed", benchmarks and
interop-testing could also be synced.
2018-12-20 16:47:16 -07:00
ZHANG Dapeng 94e7339748
all: remove copies of gradle wrapper 2018-12-11 16:23:28 -08:00
ZHANG Dapeng dd1a7e19b9
buildscripts: remove jenkins-pre.bat 2018-12-11 10:36:58 -08:00
ZHANG Dapeng 3202fcc7d9
examples: move TLS example to a separate project 2018-12-06 12:03:25 -08:00
ZHANG Dapeng 00b4b8870b
examples: move alts bazel and rm alts from pom.xml 2018-12-03 17:12:30 -08:00
Eric Anderson 80883f0983 buildscripts: Reduce verbosity of protobuf builds
A substantial portion of our build logs are just compiling protobuf.
This reduces the noise while still keeping most of the useful
information.
2018-09-21 10:58:07 -07:00
Eric Gribkoff 03dab29e3a
buildscripts: run Android interop tests on Firebase (#4314) 2018-08-28 14:56:06 -07:00
zpencer b8f4c7a57a
buildscripts: disable gae jdk7 test (#4787)
The test has become very flakey recently. With GAE jdk7 going away
completely in early 2019, let's disable running the test in CI.
2018-08-23 17:14:49 -07:00
Eric Anderson 71e9841c80 travis.yml: Fix protobuf compilation caching
Wildcard apparently doesn't work... silently. Move versions into their
own directory so we can use a hard-coded string in the travis
configuration.
2018-08-17 17:27:12 -07:00
Eric Anderson 142851dcbb buildscripts: Use full SHA for protobuf download
Github used to produce tars with the full SHA in the folder name.
However, now they are providing tars with just as much SHA as was in the
initial request. This coincides with a change in its services:

Before:
Resolving github.com... 192.30.253.113, 192.30.253.112
Connecting to github.com|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/23357588/88f90e58-e598-11e7-9263-ef8b1fb345b8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180817%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180817T190158Z&X-Amz-Expires=300&X-Amz-Signature=b684e1ead24a96764fd1f29ac5eb1df693d47c5fe86962088aee2ccfb8e4e72a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprotobuf-all-3.5.1.tar.gz&response-content-type=application%2Foctet-stream [following]

As of 2:00 PM PT today:
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/google/protobuf/tar.gz/61476b8e [following]

Note that repositories.bzl isn't impacted because it already specified
the full SHA.
2018-08-17 16:59:25 -07:00
Spencer Fang c3c810f5c3 buildscripts: Make kokoro script runnable from workstation
- Only cat /VERSION if it exists
- The host name is used as the version name, and if running locally
  the versions no longer necessarily start with 'kokoro'. Update the
  clean up command.
2018-08-16 15:03:03 -07:00
Spencer Fang 19f4491ce9 buildscripts: Clean up dummy default version check cmd
These statements can be simplified to be more readable.
2018-08-16 15:03:03 -07:00
Spencer Fang 9aa53b589d buildscripts,gae-interop-testing: do not promote versions by default
Only the dummy-default version should ever be promoted. Test versions
should have no traffic routed to it, so that deletions are
simpler. Versions receiving traffic can not be deleted in GAE.
2018-08-16 15:03:03 -07:00
Spencer Fang b821cf13b0 buildscripts,gae-interop-testing: Hardcode service name
This simplifies the kokoro script and makes things consistent between
the gradle script behavior and kokoro behavior.
2018-08-16 15:03:03 -07:00
Eric Anderson 8188a3eb92 gradle: Use config_loc in checkstyle
We previously passed a custom variable to the checkstyle configuration.
In Gradle 4.0 config_loc was added for the same purpose. The default
configDir is $projectDir/config/checkstyle which is different for each
project; we need to override it to always point to the root project.
2018-07-30 10:51:54 -07:00
Eric Anderson 4e276a52f3 buildscripts: Remove set -x from run_in_docker.sh
Printing the commands is pretty ugly output and not particularly
helpful, unless debugging the logic. If needing to debug, just add the
-x locally.
2018-07-20 14:08:07 -07:00
Eric Anderson 7bb88de264 buildscripts: remove grpc-java-releasing docker container
The buildscript was updated to pull in a fix to protoc-artifacts to
support TLS 1.2 (google/protobuf#4879), which was the only remaining
reason to have our own container.

Docker container building was split out into a separate build_docker.sh
so that people can call it blindly and get the container necessary for
run_in_docker.sh.
2018-07-11 13:04:30 -07:00
Eric Anderson 1887b93afd buildscripts: Hard-code PROTOBUF_VERSION in make_dependencies.sh
We always want to use a consistent version of protobuf; avoid the need
for the caller (which may be a person running the script) to specify the
version.
2018-07-11 10:13:09 -07:00
Eric Anderson 4879d6f6a8 buildscripts: Use "all" protobuf download to avoid ./autogen.sh
This cuts ~20 seconds from the build time and avoids the need to have
autoconf and libtool installed. This is the "normal" way to build
protobuf.
2018-07-11 10:12:33 -07:00
Eric Anderson 3a58a9999e Fix signatures in release process
We must not include signatures for maven-metadata.xml*, as the file is
a server-maintained index. Avoiding signing the .md5 and .sha1 avoids
signing unnecessary files (which double-excludes some
maven-metadata.xml* files).

We also swap to using ASCII armored output instead of the binary output,
as that's what is appropriate for "asc" files.

Fixes #4574
2018-06-20 11:29:32 -07:00
Eric Gribkoff 9a502ecf0b
buildscripts: correct Android apk size and dex diff output (#4522) 2018-06-05 12:50:39 -07:00
Eric Gribkoff 8fd762fb8e
cronet: use Cronet from Google's Maven repository (#4531) 2018-06-04 20:16:02 -07:00
Eric Gribkoff f3ef2850a9
buildscripts: fix Android apk/dex comparison (#4511) 2018-05-25 15:07:17 -07:00
zpencer b9df27b86f
buildscripts: add basic sanity check for android aar (#4448)
Just a very basic sanity check that the android artifact is not missing.
2018-05-07 15:43:08 -07:00
Eric Gribkoff b83312fb7b buildscripts: generate android artifact 2018-05-07 11:31:56 -07:00
zpencer c51a7749b1
buildscripts: assume artifacts are coalesced in upload_artifacts (#4439)
The `find` was unnecessary. Update the sanity check to reflect this
assumption.
2018-05-07 11:18:42 -07:00
zpencer 2404b25101
buildscripts: automate the release process (#4406)
These scripts set up the jobs needed to do one click releases.

List of changes:

Do not put files in mvn-artifacts/$ARCH. This makes the view uniform
across unix and windows and is easier to sign and upload.

A working curl is needed to build protobuf from source, so run yum
update.

run_in_docker.sh: use chmod to fix permisisons, do not set up user

linux_artifact builds 32 and 64 artifacts using unix.sh

add upload_artifacts cfg and script
2018-05-03 19:48:14 -07:00
Carl Mastrangelo 60a0b0c471
all: normalize copyright header 2018-05-03 14:55:21 -07:00
zpencer f8424f7b7b
examples: add kotlin and kotlin android examples (#4037)
Add kotlin version of HelloWorldClient and HelloWorldServer.
Add kotlin version of android example.
2018-05-01 14:54:34 -07:00
zpencer 1d80febbc0
buildscripts: make unix.sh and make_dependencies.sh arch aware (#4384)
ARCH can be '32' or '64'. If it is not set then default to '64'.

make_dependencies.sh should do the symlinking
2018-04-27 15:59:16 -07:00
zpencer c6c20e9491
buildscripts: add linux job that release_artifacts can dep on (#4392)
* buildscripts: add linux job that release_artifacts can dep on

* rename linux -> linux_artifacts

* fix script name
2018-04-25 15:54:15 -07:00
Carl Mastrangelo 720d4fab69
all: print out diff of methods in android APK 2018-04-25 15:53:56 -07:00
zpencer 218e944e16
buildscripts: initial kokoro config for auto releasing artifacts (#4391)
The script does nothing at the moment other than set up the kokoro
job.
2018-04-25 14:12:06 -07:00
zpencer 7cd2f5c3c0
buildscripts: sonatype uploader ussing CONF before it is set (#4388)
Set $CONF before we use it.
Also enable some defensive bash flags to catch errors.
Default value for USER and PASS
2018-04-25 12:54:49 -07:00
Eric Gribkoff ba86a86c77
android: add AndroidChannelBuilder (#4172) 2018-04-23 11:49:36 -07:00
Eric Anderson a47266ea4a kokoro: Avoid --include-build for Android
Since 4369e8cd the --include-build just opens us up to trouble with
accidentally building the protoc plugin. Since we're going to do a
./gradlew install anyway, let's just wait until after that point for
building cronet.

This sort of problem was experienced while developing #4369.
2018-04-21 00:03:44 -07:00
zpencer 48b7c62b43
buildscripts: always keep mvn artifacts (#4372)
After searching for "artifacts retention policy" in the
kokoro-users group, I learned that there's a default 90 day
retention policy already in place.
2018-04-20 15:19:48 -07:00
zpencer 9f92104b94
buildscripts: fix GAE dummy app deploy races (#4308)
This fixes a race where we can race with another deploy of the
'dummy' version.

Also, when we delete old versions, only delete versions beginning with
'kokoro'. This would make races even less likely because the
'dummy-default' should normally stick around forever.

This fixes this failure:
https://source.cloud.google.com/results/invocations/6b24aed4-7dc3-490a-87ea-c870d1435184/targets/grpc%2Fjava%2Fpresubmit%2Fgae-interop/log
2018-04-05 13:19:36 -07:00
Eric Anderson ef030abe6b java_grpc_library: re-add grpc-java repo to targets
They were removed in 137c74d1 since it was believed they were unnecessary.
However, since they are in a macro and not a rule, they are relative to the
caller, not their definition.

Added building the examples to the kokoro CI. Note that this means the examples
are built twice: once in grpc-java's build and once in their own (because it
has a WORKSPACE). Given that the Bazel build is our fastest build, this
slowdown won't probably be an issue.
2018-04-03 08:42:40 -07:00
zpencer 19a64cc336
buildscripts: build windows artifacts in kokoro (#4253)
We always build the artifacts, but only store them in placer via
kokoro if MVN_ARTIFACTS is set to a non empty value.

The 32 bit script runs tests as well as builds artifacts.
The 64 bit script only builds artifacts.
2018-03-22 17:56:59 -07:00
Eric Anderson 9c5e96e376
buildscripts: Cleaner Linux release Docker
This is a greatly simplified Docker container compared to that in
compiler/. We really want the docker image to just be a build
environment, and build the specific versions of dependencies as part of
our normal build (not docker build).

It also includes a helper script that lets you easily do build actions
with the docker environment, but into a checkout on the host. This can
dramatically reduce the pain in building from the docker container as
the source doesn't need to be checked out and only caches are lost
between invocations.
2018-03-13 09:38:21 -07:00
Eric Anderson ab92ff0819 buildscripts: Use ErrorActionPreference=stop with powershell
Otherwise when an exception is thrown powershell will still exit with 0.
This is similar to set -e, except exceptions wouldn't have a status code
of their own.

Another option would have been -ErrorAction Stop, but this need to be
passed to each command inside the powershell script, so would be
functionally similar to our "|| exit" style in the .bat.
2018-03-13 09:16:59 -07:00
Eric Anderson 4bd4cf4ec1 buildscripts: check errorlevel of error-prone commands in .bat files
There's no 'set -e' in BAT, so we get to do it manually. Wee!

Note that we need to use 'exit /b' instead of 'exit' in the scripts, as
otherwise it exits more than just the current script.
2018-03-13 09:16:59 -07:00
Eric Gribkoff 7fd22080e9
buildscripts: avoid unbound variable error (#4197) 2018-03-09 09:18:58 -08:00
Eric Gribkoff 4369e8cd9a
buildscripts: add Kokoro-based CI for Android APK stats (#3984) 2018-03-08 18:41:39 -08:00
Eric Anderson 9f677a991a buildscripts: Hard-code use of TLSv1.2 for github download
GitHub is now requiring TLS v1.2:
https://githubengineering.com/crypto-removal-notice/

Without this configuration, we see:
iwr : The request was aborted: Could not create SSL/TLS secure channel.
At line:1 char:5
+ & { iwr https://github.com/google/protobuf/archive/v3.5.1.zip -OutFile
protobuf. ...
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:Htt
  pWebRequest) [Invoke-WebRequest], WebException
   + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe
  ll.Commands.InvokeWebRequestCommand

Fixes #4120
2018-02-23 13:34:12 -08:00
zpencer a45d07bcb5
build: run unit tests on osx, ensure proto output included (#4081)
This adds back functionality that was accidentally dropped when
porting from travis:
- make sure generating protos will not lead to any uncomitted changes
- actually run the unit tests
2018-02-14 16:44:02 -08:00
zpencer fbc079d631
buildscripts: build macos artifacts in kokoro (#4066) 2018-02-14 12:53:11 -08:00
Eric Anderson f48a5097ef kokoro: Avoid getting cmake if it's installed
Kokoro already has cmake installed, so it's not necessary in that
environment. We still keep the old code around because it's helpful for
setting up new Windows environments in general.
2018-02-12 07:22:36 -08:00
Eric Anderson 850143f4a3 kokoro: Include kokoro image version in build logs
Otherwise there's no way to determine which version of the image was
used for the build.
2018-02-05 11:11:03 -08:00