diff --git a/docs/testing.md b/docs/testing.md index 49ddf174dd..3aa8ad00ca 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -44,12 +44,87 @@ spec: ### Running the kubernetes e2e test suite -The [e2e](../e2e/README.md) directory has a docker image and some scripts which make it easy to run -the kubernetes e2e tests, using kops. +The simple way: + +``` +# cd wherever you tend to put git repos +git clone https://github.com/kubernetes/test-infra.git +export KOPS_E2E_STATE_STORE=s3://your-kops-state-store # Change to your state store path +export KOPS_E2E_CLUSTER_NAME=e2e.cluster.name # Change to an FQDN for your e2e cluster name +test-infra/jobs/ci-kubernetes-e2e-kops-aws.sh |& tee /tmp/testlog +``` + +This: +* Brings up a cluster using the latest `kops` build from `master` (see below for how to use your current build) +* Runs the default series of tests (which the Kubernetes team is [also +running here](https://k8s-testgrid.appspot.com/google-aws#kops-aws)) (see below for how to override the test list) +* Tears down the cluster +* Pipes all output to `/tmp/testlog` + +(**Note**: By default this script assumes that your AWS credentials are in +`~/.aws/credentials`, and the SSH keypair you want to use is +`~/.ssh/kube_aws_rsa`. You can override `JENKINS_AWS_CREDENTIALS_FILE`, +`JENKINS_AWS_SSH_PRIVATE_KEY_FILE` and `JENKINS_AWS_SSH_PUBLIC_KEY_FILE` if you +want to change this.) + +This isn't yet terribly useful, though - it just shows how to replicate the +existing job, but not with your custom code. To test a custom `kops` build, you +can do the following: + +``` +# cd to your kops repo +export BUILD_BUCKET=your-kops-builds # Change to a bucket for kops builds +make upload S3_BUCKET=s3://${BUILD_BUCKET} + +# That will output s3 paths a it's uploading. Copy the relevant +# kops/git- path and then: +export KOPS_URL=http://${BUILD_BUCKET}.s3.amazonaws.com/kops/git- +``` + +Then follow the test directions above. + +To override the test list for the job, you need to familiar with the +`ginkgo.focus` and `ginkgo.skip` +flags. Using these flags, you can do: + +``` +export GINKGO_TEST_ARGS="--ginkgo.focus=\[Feature:Performance\]" +``` + +and follow the instructions above. [Here are some other examples from the `e2e.go` documentation.](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md#building-and-running-the-tests). + +If you want to test against an existing cluster, you can do: + +``` +export E2E_UP=false; export E2E_DOWN=false +``` + +and follow the instructions above. This is particularly useful for testing the +myriad of `kops` configuration/topology options without having to modify the +testing infrastructure. *Note:* This is also the only way currently to test a +custom Kubernetes build +(see +[kubernetes/test-infra#1454](https://github.com/kubernetes/test-infra/issues/1454)). + ### Uploading a custom build -If you want to upload a custom build, here is one way to do so: +If you want to upload a custom Kubernetes build, here is a simple way (note: +this assumes you've run `make quick-release` in the Kubernetes repo first): + + +``` +# cd wherever you tend to put git repos +git clone https://github.com/kubernetes/release.git + +# cd back to your kubernetes repo +/path/to/release/push-build.sh # Fix /path/to/release with wherever you cloned the release repo +``` + +That will upload the release to a GCS bucket and make it public. You can then +use the outputted URL in `kops` with `--kubernetes-version`. + +If you need it private in S3, here's a manual way: ``` make quick-release diff --git a/e2e/Dockerfile b/e2e/Dockerfile deleted file mode 100644 index 558c2ceacd..0000000000 --- a/e2e/Dockerfile +++ /dev/null @@ -1,28 +0,0 @@ -# Copyright 2016 The Kubernetes Authors. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM debian:jessie - -RUN apt-get update && apt-get install --yes curl python-pip openssh-client -RUN pip install awscli - -RUN curl https://sdk.cloud.google.com | bash - -RUN curl https://storage.googleapis.com/golang/go1.6.3.linux-amd64.tar.gz | tar -C /usr/local -xzf - - -ENV PATH /root/google-cloud-sdk/bin:/usr/local/go/bin:$PATH - -ADD runtests.sh / - -ADD conf /conf diff --git a/e2e/Makefile b/e2e/Makefile deleted file mode 100644 index c8890544b5..0000000000 --- a/e2e/Makefile +++ /dev/null @@ -1,35 +0,0 @@ -# Copyright 2016 The Kubernetes Authors. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#JOB_NAME=kubernetes-e2e-kops-aws -#KUBERNETES_VERSION=v1.3.5 -#DNS_DOMAIN="mydomain.com" -#JENKINS_GCS_LOGS_PATH=gs://kopeio-kubernetes-e2e/logs -#KOPS_STATE_STORE=s3://mys3bucket - -test: image - docker run -v ${HOME}/.ssh/id_rsa.pub:/root/.ssh/id_rsa.pub \ - -v ${HOME}/.aws:/root/.aws \ - -v ${HOME}/.gsutil:/root/.gsutil \ - -v ${HOME}/.config/gcloud:/root/.config/gcloud \ - -e "DNS_DOMAIN=${DNS_DOMAIN}" \ - -e "JENKINS_GCS_LOGS_PATH=${JENKINS_GCS_LOGS_PATH}" \ - -e "KOPS_STATE_STORE=${KOPS_STATE_STORE}" \ - -e "JOB_NAME=${JOB_NAME}" \ - -e "KUBERNETES_VERSION=${KUBERNETES_VERSION}" \ - kubernetes-e2e-runner-kops \ - /runtests.sh - -image: - docker build -t kubernetes-e2e-runner-kops . diff --git a/e2e/README.md b/e2e/README.md deleted file mode 100644 index 6028cf646f..0000000000 --- a/e2e/README.md +++ /dev/null @@ -1,18 +0,0 @@ -## Run kubernetes e2e tests - -This docker image lets you run the kubernetes e2e tests very easily, using kops to create the cluster. - -You simply call make, specifying some variables that controls the build. - -An example: - -`make JOB_NAME=kubernetes-e2e-kops-aws KUBERNETES_VERSION=v1.3.5 DNS_DOMAIN=e2e.mydomain.com JENKINS_GCS_LOGS_PATH=gs://kopeio-kubernetes-e2e/logs KOPS_STATE_STORE=s3://clusters.mydomain.com` - -Variables: - -* `JOB_NAME` the e2e job to run. Corresponds to a conf file in the conf directory. -* `KUBERNETES_VERSION` the version of kubernetes to run. Either a version like `v1.3.5`, or a URL prefix like `https://storage.googleapis.com/kubernetes-release-dev/ci/v1.4.0-alpha.2.677+ea69570f61af8e/`. See [testing docs](../docs/testing.md) -* `DNS_DOMAIN` the dns domain name to use for the cluster. Must be a real domain name, with a zone registered in DNS (route53) -* `JENKINS_GCS_LOGS_PATH` the gs bucket where we should upload the results of the build. Note these will be publicly readable. -* `KOPS_STATE_STORE` the url where the kops registry (store of cluster information) lives. - diff --git a/e2e/conf/cloud/aws b/e2e/conf/cloud/aws deleted file mode 100644 index 6d221141da..0000000000 --- a/e2e/conf/cloud/aws +++ /dev/null @@ -1,12 +0,0 @@ -##============================================================= -# AWS Settings -export AWS_DEFAULT_REGION=${AWS_REGION} -#export KUBE_AWS_ZONE="us-west-2a" -#export PROJECT="k8s-jkns-e2e-aws" -export AWS_CONFIG_FILE=~/.aws/credentials -export AWS_SSH_KEY=~/.ssh/kube_aws_rsa -export KUBE_SSH_USER=admin -# This is needed to be able to create PD from the e2e test -export AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials - -export AWS_PROFILE=kubernetes-e2e diff --git a/e2e/conf/cloud/gce b/e2e/conf/cloud/gce deleted file mode 100644 index f793b710a2..0000000000 --- a/e2e/conf/cloud/gce +++ /dev/null @@ -1,5 +0,0 @@ -# GCE variables -#export INSTANCE_PREFIX="${E2E_NAME:-jenkins-e2e}" -#export KUBE_GCE_NETWORK="${E2E_NAME:-jenkins-e2e}" -#export KUBE_GCE_INSTANCE_PREFIX="${E2E_NAME:-jenkins-e2e}" -#export GCE_SERVICE_ACCOUNT=$(gcloud auth list 2> /dev/null | grep active | cut -f3 -d' ') diff --git a/e2e/conf/kubernetes-e2e-kops-aws b/e2e/conf/kubernetes-e2e-kops-aws deleted file mode 100644 index 90f1e384b2..0000000000 --- a/e2e/conf/kubernetes-e2e-kops-aws +++ /dev/null @@ -1,12 +0,0 @@ -export KUBERNETES_PROVIDER="aws" - -export E2E_MIN_STARTUP_PODS="1" -export MASTER_SIZE="m3.large" -export NODE_SIZE="m3.large" -export NUM_NODES="3" -export GINKGO_TEST_ARGS="--ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]" -export GINKGO_PARALLEL="y" - -# Central only has 2 AZs, so we use it for the non-HA test -export NODE_ZONES="eu-central-1a" -export AWS_REGION="eu-central-1" diff --git a/e2e/conf/kubernetes-e2e-kops-aws-ha b/e2e/conf/kubernetes-e2e-kops-aws-ha deleted file mode 100644 index 22a2c3f16d..0000000000 --- a/e2e/conf/kubernetes-e2e-kops-aws-ha +++ /dev/null @@ -1,4 +0,0 @@ -. conf/kubernetes-e2e-upup-aws - -export NODE_ZONES="eu-west-1a,eu-west-1b,eu-west-1c" -export AWS_REGION="eu-west-1" diff --git a/e2e/conf/site b/e2e/conf/site deleted file mode 100644 index 4c8e6af5d3..0000000000 --- a/e2e/conf/site +++ /dev/null @@ -1,2 +0,0 @@ -# Used to set site-specific settings -# Most of these probably are better in the Makefile \ No newline at end of file diff --git a/e2e/runtests.sh b/e2e/runtests.sh deleted file mode 100755 index 96758a6733..0000000000 --- a/e2e/runtests.sh +++ /dev/null @@ -1,226 +0,0 @@ -#!/bin/bash -ex - -# Copyright 2014 The Kubernetes Authors. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -if [[ -z "${JOB_NAME}" ]]; then - echo "Must specify JOB_NAME env var" - exit 1 -fi -if [[ -z "${KUBERNETES_VERSION}" ]]; then - echo "Must specify KUBERNETES_VERSION env var" - exit 1 -fi -if [[ -z "${DNS_DOMAIN}" ]]; then - echo "Must specify DNS_DOMAIN env var" - exit 1 -fi -if [[ -z "${KOPS_STATE_STORE}" ]]; then - echo "Must specify KOPS_STATE_STORE env var" - exit 1 -fi -# TODO: Maybe skip if we don't want to upload logs? -if [[ -z "${JENKINS_GCS_LOGS_PATH}" ]]; then - echo "Must specify JENKINS_GCS_LOGS_PATH env var" - exit 1 -fi - -echo "JOB_NAME=${JOB_NAME}" -echo "Loading conf/${JOB_NAME}" - -. conf/${JOB_NAME} - -echo "Loading conf/cloud/${KUBERNETES_PROVIDER}" -. conf/cloud/${KUBERNETES_PROVIDER} - -echo "Loading conf/site" -. conf/site - -##============================================================= -# Global settings -export KUBE_GCS_RELEASE_BUCKET=kubernetes-release - -# We download the binaries ourselves -# TODO: No way to tell e2e to use a particular release? -# TODO: Maybe download and then bring up the cluster? -export JENKINS_USE_EXISTING_BINARIES=y - -# This actually just skips kube-up master detection -export KUBERNETES_CONFORMANCE_TEST=y - -##============================================================= -# System settings (emulate jenkins) -export USER=root -export WORKSPACE=$HOME -# Nothing should want Jenkins $HOME -export HOME=${WORKSPACE} -export BUILD_NUMBER=`date -u +%Y%m%d%H%M%S` -export JENKINS_HOME=${HOME} - -# We'll directly up & down the cluster -export E2E_UP="${E2E_UP:-false}" -export E2E_TEST="${E2E_TEST:-true}" -export E2E_DOWN="${E2E_DOWN:-false}" - -# Skip gcloud update checking -export CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK=true - - -##============================================================= - -branch=master - -build_dir=${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/ -rm -rf ${build_dir} -mkdir -p ${build_dir}/workspace - -cd ${build_dir}/workspace - -# Sanity check -#gsutil ls ${JENKINS_GCS_LOGS_PATH} - -exit_code=0 -SECONDS=0 # magic bash timer variable -curl -fsS --retry 3 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/e2e-runner.sh" > /tmp/e2e.sh -chmod +x /tmp/e2e.sh - -# We need kubectl to write kubecfg from kops -curl -fsS --retry 3 "https://storage.googleapis.com/kubernetes-release/release/v1.3.5/bin/linux/amd64/kubectl" > /usr/local/bin/kubectl -chmod +x /usr/local/bin/kubectl - -curl -fsS --retry 3 "https://kubeupv2.s3.amazonaws.com/kops/1.4.0/linux/amd64/kops" > /tmp/kops -cp /tmp/kops /usr/local/bin/kops -chmod +x /usr/local/bin/kops - -if [[ ! -e ${AWS_SSH_KEY} ]]; then - echo "Creating ssh key ${AWS_SSH_KEY}" - ssh-keygen -N "" -t rsa -f ${AWS_SSH_KEY} -fi - -function fetch_tars_from_gcs() { - local -r bucket="${1}" - local -r build_version="${2}" - echo "Pulling binaries from GCS; using server version ${bucket}/${build_version}." - gsutil -mq cp \ - "gs://${KUBE_GCS_RELEASE_BUCKET}/${bucket}/${build_version}/kubernetes.tar.gz" \ - "gs://${KUBE_GCS_RELEASE_BUCKET}/${bucket}/${build_version}/kubernetes-test.tar.gz" \ - . -} - -function unpack_binaries() { - md5sum kubernetes*.tar.gz - tar -xzf kubernetes.tar.gz - tar -xzf kubernetes-test.tar.gz -} - - -fetch_tars_from_gcs release ${KUBERNETES_VERSION} -unpack_binaries - -# Clean up everything when we're done -function finish { - kops delete cluster \ - --name ${JOB_NAME}.${DNS_DOMAIN} \ - --yes 2>&1 | tee -a ${build_dir}/build-log.txt -} -trap finish EXIT - -set -e - -# Create the cluster spec -kops create cluster \ - --name ${JOB_NAME}.${DNS_DOMAIN} \ - --cloud ${KUBERNETES_PROVIDER} \ - --zones ${NODE_ZONES} \ - --node-size ${NODE_SIZE} \ - --master-size ${MASTER_SIZE} \ - --ssh-public-key ${AWS_SSH_KEY}.pub \ - --kubernetes-version ${KUBERNETES_VERSION} \ - --v=4 2>&1 | tee -a ${build_dir}/build-log.txt -exit_code=${PIPESTATUS[0]} - -# Apply the cluster spec -if [[ ${exit_code} == 0 ]]; then - kops update cluster \ - --name ${JOB_NAME}.${DNS_DOMAIN} \ - --yes --v=4 2>&1 | tee -a ${build_dir}/build-log.txt - exit_code=${PIPESTATUS[0]} -fi - -# Wait for kubectl to begin responding (at least master up) -if [[ ${exit_code} == 0 ]]; then - attempt=0 - while true; do - kubectl get nodes --show-labels 2>&1 | tee -a ${build_dir}/build-log.txt - exit_code=${PIPESTATUS[0]} - - if [[ ${exit_code} == 0 ]]; then - break - fi - if (( attempt > 60 )); then - echo "Unable to connect to API in 15 minutes (master did not launch?)" - break - fi - attempt=$(($attempt+1)) - sleep 15 - done -fi - -# TODO: can we get rid of this? -echo "API responded; waiting 450 seconds for DNS to settle" -for ((i=1;i<=15;i++)); do - kubectl get nodes --show-labels 2>&1 | tee -a ${build_dir}/build-log.txt - sleep 30 -done - - -# Run e2e tests -if [[ ${exit_code} == 0 ]]; then - /tmp/e2e.sh 2>&1 | tee -a ${build_dir}/build-log.txt - exit_code=${PIPESTATUS[0]} -fi - -# Try to clean up normally so it goes into the logs -# (we have an exit hook for abnormal termination, but that does not get logged) -finish - -duration=$SECONDS -set +e - -if [[ ${exit_code} == 0 ]]; then - success="true" -else - success="false" -fi - -version=`cat kubernetes/version` - -gcs_acl="public-read" -gcs_job_path="${JENKINS_GCS_LOGS_PATH}/${JOB_NAME}" -gcs_build_path="${gcs_job_path}/${BUILD_NUMBER}" - -gsutil -q cp -a "${gcs_acl}" -z txt "${build_dir}/build-log.txt" "${gcs_build_path}/" - -curl -fsS --retry 3 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh" | bash - - - -curl -fsS --retry 3 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-finished.sh" > upload-finished.sh -chmod +x upload-finished.sh - -if [[ ${exit_code} == 0 ]]; then - ./upload-finished.sh SUCCESS -else - ./upload-finished.sh UNSTABLE -fi