From 1ecb12378f42a04fb4572c2070313454565fa6a5 Mon Sep 17 00:00:00 2001 From: "Erik L. Arneson" Date: Mon, 17 Aug 2020 15:14:57 -0700 Subject: [PATCH] Developer Guide: Refresh e2e testing documents for kubetest2 This change creates a new file with e2e testing documentation for the new kubetest2 test runner. Once kubetest is retired, this can drop right into the existing e2e developer guide docs. --- .../devel/sig-testing/e2e-tests-kubetest2.md | 228 ++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 contributors/devel/sig-testing/e2e-tests-kubetest2.md diff --git a/contributors/devel/sig-testing/e2e-tests-kubetest2.md b/contributors/devel/sig-testing/e2e-tests-kubetest2.md new file mode 100644 index 000000000..22cf90448 --- /dev/null +++ b/contributors/devel/sig-testing/e2e-tests-kubetest2.md @@ -0,0 +1,228 @@ +# End-to-End Testing in Kubernetes + +**Table of Contents** + +- [End-to-End Testing in Kubernetes](#end-to-end-testing-in-kubernetes) + - [Overview](#overview) + - [Setting up kubetest2](#setting-up-kubetest2) + - [Running the Tests](#running-the-tests) + - [Cleaning Up](#cleaning-up) + - [Advanced Testing](#advanced-testing) + - [Extracting a specific version of Kubernetes](#extracting-a-specific-version-of-kubernetes) + - [Testing against an existing cluster](#testing-against-an-existing-cluster) + - [Debugging clusters](#debugging-clusters) + +## Overview + +End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end +behavior of the system. It is the last signal to ensure end user operations +match developer specifications. Although unit and integration tests provide a +good signal, in a distributed system like Kubernetes it is not uncommon that a +minor change may pass all unit and integration tests, but cause unforeseen +changes at the system level. + +The primary objectives of the e2e tests are to ensure a consistent and reliable +behavior of the Kubernetes code base, and to catch hard-to-test bugs that were not caught by unit and integration tests. + +The e2e tests in Kubernetes are built on top of +[Ginkgo](http://onsi.github.io/ginkgo/) and +[Gomega](http://onsi.github.io/gomega/). There are a host of features that this +Behavior-Driven Development (BDD) testing framework provides, and it is +recommended that the developer read the documentation prior to diving into the +tests. + +The purpose of this document is to serve as a primer for developers who are +looking to execute or add tests using a local development environment. + +Before writing new tests or making substantive changes to existing tests, you +should also read [Writing Good e2e Tests](writing-good-e2e-tests.md) + +## Setting up kubetest2 + +The e2e tests in Kubernetes are managed and run with a helper +application. On July 14, 2020, it was announced that the venerable +[kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest#installation) +would be deprecated in favor of +[kubetest2](https://github.com/kubernetes-sigs/kubetest2). These +instructions walk you through the installation and configuration of +`kubetest2` and then describe some usage. + +Before proceeding to these instructions, you should have a working +Kubernetes development environment. Please follow the directions in +the [Developer Guide](../development.md) to get started. You should +also have a working and configured `gcloud` CLI tool for use with +Google Cloud +Platform. [Check out the `gcloud` cheatsheet for help getting started.](https://cloud.google.com/sdk/docs/cheatsheet) + +The first step is to install the Ginkgo testing framework and +Gomega. Run these commands: + +```sh +go get github.com/onsi/ginkgo/ginkgo +go get github.com/onsi/gomega/... +``` + +In order to successfully build Kubernetes tests, you will need to +install Bazel. +[Follow the instructions for your development environment.](https://docs.bazel.build/versions/3.4.0/install.html) + +Next, you need to download, build, and install the `kubetest2` +binary. Run these commands to do that: + +```sh +cd $GOPATH/src/k8s.io/ +git clone git@github.com:kubernetes-sigs/kubetest2.git +cd kubetest2 +make install +``` + +After building and installing `kubetest2`, you need to install +and configure the plugin for your cloud provider. Current available +plugins are: + +- Google Cloud Compute Engine (kubetest2-gce) +- Google Cloud Kubernetes Engine (kubetest2-gke) +- [KIND](https://kind.sigs.k8s.io/) (kubetest2-kind) + +You can install these via `go get`, but since you are already in the +`kubetest2` directory, you can also just use a command like this: + +```sh +make install-deployer-gce INSTALL_DIR=$GOPATH/bin +``` + +Next, install the interface between `kubetest2` and Ginkgo. While +remaining in the `kubetest2` directory, run this command: + +```sh +make install-tester-ginkgo INSTALL_DIR=$GOPATH/bin +``` + +## Running the Tests + +The examples below all use the Google Cloud Compute Engine plugin +(kubetest2-gce). Different cloud provider plugins use different +command-line switches. + +In these examples, replace `` with the name of the +Google Cloud project in which you are running e2e tests. + +To provision and start a Kubernetes test cluster, run this command: + +```sh +kubetest2 gce --gcp-project --up +``` + +You can also shut down and destroy a testing cluster with this +command: + +```sh +kubetest2 gce --gcp-project --down +``` + +You do not need the `--gcp-project` argument to run tests. While your +test cluster is up and running, you can run all tests with this +command: + +```sh +kubetest2 gce --test ginkgo +``` + +You can also pass options to Ginkgo to select which tests you'd like +to run. If you add a `--` at the end of the command line, everything +after that is passed as arguments to Ginkgo. + +For example, to run all tests that match the regular expression +`\[Feature:Performance\]`, use this command: + +```sh +kubetest2 gce --test ginkgo -- --focus-regex "\[Feature:Performance\]" +``` + +Conversely, to exclude tests that match the regular expression +`Pods.*env`, use this command: + +```sh +kubetest2 gce --test ginkgo -- --skip-regex "Pods.*env" +``` + +In this example, we instruct `kubetest2` to run two tests in parallel +at once, while skipping any that must be run serially: + +```sh +kubetest2 gce --test ginkgo -- --skip-regex "\[Serial\]" --parallel 2 +``` + +See `kubetest2 gce --help` for more options. Note that the different +deployer and tester plugins have their own additional options, which +can be seen in their help listings. + +### Cleaning Up + +During a run, pressing **Control-C** should result in an orderly +shutdown. However, if something goes wrong and you still have +containers or VMs provisioned, you can run a cleanup with a command +like this: + +```sh +kubetest2 gce --gcp-project --down +``` + +## Advanced Testing + +### Extracting a specific version of Kubernetes + +It is possible for `kubetest2` to download and extract a specific +version of Kubernetes for testing. This can be accomplished by passing +the `--test-package-version` flag to the tester plugin. For example: + +```sh +kubetest2 gce --test ginkgo -- --test-package-version v1.18.0 +``` + +The argument to `--test-package-version` can be changed to specify +some other test package version. To see available release names for +this option, use this command: + +```sh +gsutil ls gs://kubernetes-release/release/ +``` + +### Testing against an existing cluster + +You can run tests against an existing Kubernetes cluster by setting +your `KUBECONFIG` environment variable to point to that cluster's +configuration file. + +```sh +KUBECONFIG= kubetest2 gce --test +``` + +It doesn't matter which deployment plugin you specify on the command +line. + +### Debugging clusters + +The configuration file for your `kubetest2` testing cluster is kept in +`_artifacts/kubetest2-kubeconfig` off of your Kubernetes working +directory. If you would like to get a better understanding of your +cluster's state in the event of failed tests, you can use the +following command to collect information. + +```sh +KUBECONFIG=./_artifacts/kubetest2-kubeconfig kubectl cluster-info dump +``` + +If you are using the `gce` deployment plugin, you can also use the +`cluster/log-dump/log-dump.sh` script to copy the logs and +configuration files from your test cluster to your local machine. + +Use the following command to dump all cluster logs: + +```sh +KUBE_GCE_INSTANCE_PREFIX=kubetest2 KUBECONFIG=$PWD/_artifacts/kubetest2-kubeconfig ./cluster/log-dump/log-dump.sh +``` + +Note that `kubetest2` also automatically dumps all logs to +`_artifacts/cluster-logs/` when you bring up your test cluster with +the `--up` command.