Developer Guide: Refresh e2e testing documents for kubetest2
This change creates a new file with e2e testing documentation for the new kubetest2 test runner. Once kubetest is retired, this can drop right into the existing e2e developer guide docs.
This commit is contained in:
Erik L. Arneson
parent
d27cb26831
commit
1ecb12378f
|
|
@ -0,0 +1,228 @@
|
|||
# End-to-End Testing in Kubernetes
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
- [End-to-End Testing in Kubernetes](#end-to-end-testing-in-kubernetes)
|
||||
- [Overview](#overview)
|
||||
- [Setting up kubetest2](#setting-up-kubetest2)
|
||||
- [Running the Tests](#running-the-tests)
|
||||
- [Cleaning Up](#cleaning-up)
|
||||
- [Advanced Testing](#advanced-testing)
|
||||
- [Extracting a specific version of Kubernetes](#extracting-a-specific-version-of-kubernetes)
|
||||
- [Testing against an existing cluster](#testing-against-an-existing-cluster)
|
||||
- [Debugging clusters](#debugging-clusters)
|
||||
|
||||
## Overview
|
||||
|
||||
End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end
|
||||
behavior of the system. It is the last signal to ensure end user operations
|
||||
match developer specifications. Although unit and integration tests provide a
|
||||
good signal, in a distributed system like Kubernetes it is not uncommon that a
|
||||
minor change may pass all unit and integration tests, but cause unforeseen
|
||||
changes at the system level.
|
||||
|
||||
The primary objectives of the e2e tests are to ensure a consistent and reliable
|
||||
behavior of the Kubernetes code base, and to catch hard-to-test bugs that were not caught by unit and integration tests.
|
||||
|
||||
The e2e tests in Kubernetes are built on top of
|
||||
[Ginkgo](http://onsi.github.io/ginkgo/) and
|
||||
[Gomega](http://onsi.github.io/gomega/). There are a host of features that this
|
||||
Behavior-Driven Development (BDD) testing framework provides, and it is
|
||||
recommended that the developer read the documentation prior to diving into the
|
||||
tests.
|
||||
|
||||
The purpose of this document is to serve as a primer for developers who are
|
||||
looking to execute or add tests using a local development environment.
|
||||
|
||||
Before writing new tests or making substantive changes to existing tests, you
|
||||
should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)
|
||||
|
||||
## Setting up kubetest2
|
||||
|
||||
The e2e tests in Kubernetes are managed and run with a helper
|
||||
application. On July 14, 2020, it was announced that the venerable
|
||||
[kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest#installation)
|
||||
would be deprecated in favor of
|
||||
[kubetest2](https://github.com/kubernetes-sigs/kubetest2). These
|
||||
instructions walk you through the installation and configuration of
|
||||
`kubetest2` and then describe some usage.
|
||||
|
||||
Before proceeding to these instructions, you should have a working
|
||||
Kubernetes development environment. Please follow the directions in
|
||||
the [Developer Guide](../development.md) to get started. You should
|
||||
also have a working and configured `gcloud` CLI tool for use with
|
||||
Google Cloud
|
||||
Platform. [Check out the `gcloud` cheatsheet for help getting started.](https://cloud.google.com/sdk/docs/cheatsheet)
|
||||
|
||||
The first step is to install the Ginkgo testing framework and
|
||||
Gomega. Run these commands:
|
||||
|
||||
```sh
|
||||
go get github.com/onsi/ginkgo/ginkgo
|
||||
go get github.com/onsi/gomega/...
|
||||
```
|
||||
|
||||
In order to successfully build Kubernetes tests, you will need to
|
||||
install Bazel.
|
||||
[Follow the instructions for your development environment.](https://docs.bazel.build/versions/3.4.0/install.html)
|
||||
|
||||
Next, you need to download, build, and install the `kubetest2`
|
||||
binary. Run these commands to do that:
|
||||
|
||||
```sh
|
||||
cd $GOPATH/src/k8s.io/
|
||||
git clone git@github.com:kubernetes-sigs/kubetest2.git
|
||||
cd kubetest2
|
||||
make install
|
||||
```
|
||||
|
||||
After building and installing `kubetest2`, you need to install
|
||||
and configure the plugin for your cloud provider. Current available
|
||||
plugins are:
|
||||
|
||||
- Google Cloud Compute Engine (kubetest2-gce)
|
||||
- Google Cloud Kubernetes Engine (kubetest2-gke)
|
||||
- [KIND](https://kind.sigs.k8s.io/) (kubetest2-kind)
|
||||
|
||||
You can install these via `go get`, but since you are already in the
|
||||
`kubetest2` directory, you can also just use a command like this:
|
||||
|
||||
```sh
|
||||
make install-deployer-gce INSTALL_DIR=$GOPATH/bin
|
||||
```
|
||||
|
||||
Next, install the interface between `kubetest2` and Ginkgo. While
|
||||
remaining in the `kubetest2` directory, run this command:
|
||||
|
||||
```sh
|
||||
make install-tester-ginkgo INSTALL_DIR=$GOPATH/bin
|
||||
```
|
||||
|
||||
## Running the Tests
|
||||
|
||||
The examples below all use the Google Cloud Compute Engine plugin
|
||||
(kubetest2-gce). Different cloud provider plugins use different
|
||||
command-line switches.
|
||||
|
||||
In these examples, replace `<project>` with the name of the
|
||||
Google Cloud project in which you are running e2e tests.
|
||||
|
||||
To provision and start a Kubernetes test cluster, run this command:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --gcp-project <project> --up
|
||||
```
|
||||
|
||||
You can also shut down and destroy a testing cluster with this
|
||||
command:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --gcp-project <project> --down
|
||||
```
|
||||
|
||||
You do not need the `--gcp-project` argument to run tests. While your
|
||||
test cluster is up and running, you can run all tests with this
|
||||
command:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --test ginkgo
|
||||
```
|
||||
|
||||
You can also pass options to Ginkgo to select which tests you'd like
|
||||
to run. If you add a `--` at the end of the command line, everything
|
||||
after that is passed as arguments to Ginkgo.
|
||||
|
||||
For example, to run all tests that match the regular expression
|
||||
`\[Feature:Performance\]`, use this command:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --test ginkgo -- --focus-regex "\[Feature:Performance\]"
|
||||
```
|
||||
|
||||
Conversely, to exclude tests that match the regular expression
|
||||
`Pods.*env`, use this command:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --test ginkgo -- --skip-regex "Pods.*env"
|
||||
```
|
||||
|
||||
In this example, we instruct `kubetest2` to run two tests in parallel
|
||||
at once, while skipping any that must be run serially:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --test ginkgo -- --skip-regex "\[Serial\]" --parallel 2
|
||||
```
|
||||
|
||||
See `kubetest2 gce --help` for more options. Note that the different
|
||||
deployer and tester plugins have their own additional options, which
|
||||
can be seen in their help listings.
|
||||
|
||||
### Cleaning Up
|
||||
|
||||
During a run, pressing **Control-C** should result in an orderly
|
||||
shutdown. However, if something goes wrong and you still have
|
||||
containers or VMs provisioned, you can run a cleanup with a command
|
||||
like this:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --gcp-project <project> --down
|
||||
```
|
||||
|
||||
## Advanced Testing
|
||||
|
||||
### Extracting a specific version of Kubernetes
|
||||
|
||||
It is possible for `kubetest2` to download and extract a specific
|
||||
version of Kubernetes for testing. This can be accomplished by passing
|
||||
the `--test-package-version` flag to the tester plugin. For example:
|
||||
|
||||
```sh
|
||||
kubetest2 gce --test ginkgo -- --test-package-version v1.18.0
|
||||
```
|
||||
|
||||
The argument to `--test-package-version` can be changed to specify
|
||||
some other test package version. To see available release names for
|
||||
this option, use this command:
|
||||
|
||||
```sh
|
||||
gsutil ls gs://kubernetes-release/release/
|
||||
```
|
||||
|
||||
### Testing against an existing cluster
|
||||
|
||||
You can run tests against an existing Kubernetes cluster by setting
|
||||
your `KUBECONFIG` environment variable to point to that cluster's
|
||||
configuration file.
|
||||
|
||||
```sh
|
||||
KUBECONFIG=<path to config file> kubetest2 gce --test
|
||||
```
|
||||
|
||||
It doesn't matter which deployment plugin you specify on the command
|
||||
line.
|
||||
|
||||
### Debugging clusters
|
||||
|
||||
The configuration file for your `kubetest2` testing cluster is kept in
|
||||
`_artifacts/kubetest2-kubeconfig` off of your Kubernetes working
|
||||
directory. If you would like to get a better understanding of your
|
||||
cluster's state in the event of failed tests, you can use the
|
||||
following command to collect information.
|
||||
|
||||
```sh
|
||||
KUBECONFIG=./_artifacts/kubetest2-kubeconfig kubectl cluster-info dump
|
||||
```
|
||||
|
||||
If you are using the `gce` deployment plugin, you can also use the
|
||||
`cluster/log-dump/log-dump.sh` script to copy the logs and
|
||||
configuration files from your test cluster to your local machine.
|
||||
|
||||
Use the following command to dump all cluster logs:
|
||||
|
||||
```sh
|
||||
KUBE_GCE_INSTANCE_PREFIX=kubetest2 KUBECONFIG=$PWD/_artifacts/kubetest2-kubeconfig ./cluster/log-dump/log-dump.sh
|
||||
```
|
||||
|
||||
Note that `kubetest2` also automatically dumps all logs to
|
||||
`_artifacts/cluster-logs/` when you bring up your test cluster with
|
||||
the `--up` command.
|
||||
Loading…
Reference in New Issue