# End-to-End Testing in Kubernetes

**Table of Contents**

- [End-to-End Testing in Kubernetes](#end-to-end-testing-in-kubernetes)
  - [Overview](#overview)
  - [Setting up kubetest2](#setting-up-kubetest2)
  - [Running the Tests](#running-the-tests)
    - [Cleaning Up](#cleaning-up)
  - [Advanced Testing](#advanced-testing)
    - [Extracting a specific version of Kubernetes](#extracting-a-specific-version-of-kubernetes)
    - [Testing against an existing cluster](#testing-against-an-existing-cluster)
    - [Debugging clusters](#debugging-clusters)

## Overview

End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end
behavior of the system. It is the last signal to ensure end user operations
match developer specifications. Although unit and integration tests provide a
good signal, in a distributed system like Kubernetes it is not uncommon that a
minor change may pass all unit and integration tests, but cause unforeseen
changes at the system level.

The primary objectives of the e2e tests are to ensure a consistent and reliable
behavior of the Kubernetes code base, and to catch hard-to-test bugs that were not caught by unit and integration tests.

The e2e tests in Kubernetes are built on top of
[Ginkgo](http://onsi.github.io/ginkgo/) and
[Gomega](http://onsi.github.io/gomega/). There are a host of features that this
Behavior-Driven Development (BDD) testing framework provides, and it is
recommended that the developer read the documentation prior to diving into the
tests.

The purpose of this document is to serve as a primer for developers who are
looking to execute or add tests using a local development environment.

Before writing new tests or making substantive changes to existing tests, you
should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)

## Setting up kubetest2

The e2e tests in Kubernetes are managed and run with a helper
application. On July 14, 2020, it was announced that the venerable
[kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest#installation)
would be deprecated in favor of
[kubetest2](https://github.com/kubernetes-sigs/kubetest2). These
instructions walk you through the installation and configuration of
`kubetest2` and then describe some usage.

Before proceeding to these instructions, you should have a working
Kubernetes development environment. Please follow the directions in
the [Developer Guide](../development.md) to get started. You should
also have a working and configured `gcloud` CLI tool for use with
Google Cloud 
Platform. [Check out the `gcloud` cheatsheet for help getting started.](https://cloud.google.com/sdk/docs/cheatsheet)

The first step is to install the Ginkgo testing framework and
Gomega. Run these commands:

```sh
go get github.com/onsi/ginkgo/ginkgo
go get github.com/onsi/gomega/...
```

In order to successfully build Kubernetes tests, you will need to
install Bazel.
[Follow the instructions for your development environment.](https://docs.bazel.build/versions/3.4.0/install.html)

Next, you need to install the `kubetest2` binary and plugins. Run
these commands to do that:

```sh
cd
GO111MODULE=on go get sigs.k8s.io/kubetest2/...@latest
```

The currently available plugins for cloud providers are:

- Google Cloud Compute Engine (kubetest2-gce)
- Google Cloud Kubernetes Engine (kubetest2-gke)
- [KIND](https://kind.sigs.k8s.io/) (kubetest2-kind)

## Building Kubernetes

To reliably and repeatedly test changes locally, it is important to
minimize the number of steps the developer needs to take. To
facilitate this, you can use `kubetest2` to build Kubernetes using the
following command: 

```sh
kubetest2 gce --build --legacy-mode
```

To build without the `--legacy-mode` option, you will need to clone
the
[`gcp-cloud-provider` repository](https://github.com/kubernetes/cloud-provider-gcp). 
This repository is part of the
[cloud provider extraction effort](https://github.com/kubernetes/community/tree/master/sig-cloud-provider#cloud-provider-extraction-migration)
and the eventual home of some of the build scripts in the Kubernetes
source tree. Use this command to clone:

```sh
git clone https://github.com/kubernetes/cloud-provider-gcp.git
```

You can now use the following command to build using the
`gcp-cloud-provider` repository. Replace `<path to
cloud-provider-gcp>` with the path to your cloned repository:

```sh
kubetest2 gce --build --repo-root <path to cloud-provider-gcp>
```

## Running the Tests

The examples below all use the Google Cloud Compute Engine plugin
(kubetest2-gce). Different cloud provider plugins use different
command-line switches.

In these examples, replace `<project>` with the name of the
Google Cloud project in which you are running e2e tests.

To provision and start a Kubernetes test cluster, run this command:

```sh
kubetest2 gce --gcp-project <project> --up
```

You can also shut down and destroy a testing cluster with this
command:

```sh
kubetest2 gce --gcp-project <project> --down
```

You do not need the `--gcp-project` argument to run tests. While your
test cluster is up and running, you can run all tests with this
command:

```sh
kubetest2 gce --test ginkgo
```

You can also pass options to Ginkgo to select which tests you'd like
to run. If you add a `--` at the end of the command line, everything
after that is passed as arguments to Ginkgo.

For example, to run all tests that match the regular expression
`\[Feature:Performance\]`, use this command:

```sh
kubetest2 gce --test ginkgo -- --focus-regex "\[Feature:Performance\]"
```

Conversely, to exclude tests that match the regular expression
`Pods.*env`, use this command:

```sh
kubetest2 gce --test ginkgo -- --skip-regex "Pods.*env"
```

In this example, we instruct `kubetest2` to run two tests in parallel
at once, while skipping any that must be run serially:

```sh
kubetest2 gce --test ginkgo -- --skip-regex "\[Serial\]" --parallel 2
```

See `kubetest2 gce --help` for more options. Note that the different
deployer and tester plugins have their own additional options, which
can be seen in their help listings.

### Cleaning Up

During a run, pressing **Control-C** should result in an orderly
shutdown. However, if something goes wrong and you still have
containers or VMs provisioned, you can run a cleanup with a command
like this:

```sh
kubetest2 gce --gcp-project <project> --down
```

## Advanced Testing

### Extracting a specific version of Kubernetes

It is possible for `kubetest2` to download and extract a specific
version of Kubernetes for testing. This can be accomplished by passing
the `--test-package-version` flag to the tester plugin. For example:

```sh
kubetest2 gce --test ginkgo -- --test-package-version v1.18.0
```

The argument to `--test-package-version` can be changed to specify
some other test package version. To see available release names for
this option, use this command:

```sh
gsutil ls gs://kubernetes-release/release/
```

### Testing against an existing cluster

You can run tests against an existing Kubernetes cluster by setting
your `KUBECONFIG` environment variable to point to that cluster's
configuration file.

```sh
KUBECONFIG=<path to config file> kubetest2 gce --test
```

It doesn't matter which deployment plugin you specify on the command
line.

### Debugging clusters

The configuration file for your `kubetest2` testing cluster is kept in
`_artifacts/kubetest2-kubeconfig` off of your Kubernetes working
directory. If you would like to get a better understanding of your
cluster's state in the event of failed tests, you can use the
following command to collect information.

```sh
KUBECONFIG=./_artifacts/kubetest2-kubeconfig kubectl cluster-info dump
```

If you are using the `gce` deployment plugin, you can also use the
`cluster/log-dump/log-dump.sh` script to copy the logs and
configuration files from your test cluster to your local machine. 

Use the following command to dump all cluster logs:

```sh
KUBE_GCE_INSTANCE_PREFIX=kubetest2 KUBECONFIG=$PWD/_artifacts/kubetest2-kubeconfig ./cluster/log-dump/log-dump.sh
```

Note that `kubetest2` also automatically dumps all logs to
`_artifacts/cluster-logs/` when you bring up your test cluster with
the `--up` command.
