kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

file testing.md moved to the new folder /devel/sig-testing - URLs updated - tombstone file created

This commit is contained in:
eduartua 2019-01-30 13:26:54 -06:00
parent 15f4ab7469
commit 06a2a2f686
9 changed files with 236 additions and 233 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
contributors/devel/README.md
View File

@ -24,7 +24,7 @@ Guide](http://kubernetes.io/docs/admin/).
* **Development Guide** ([development.md](development.md)): Setting up your development environment. * **Development Guide** ([development.md](development.md)): Setting up your development environment.
* **Testing** ([testing.md](testing.md)): How to run unit, integration, and end-to-end tests in your development sandbox. * **Testing** ([testing.md](sig-testing/testing.md)): How to run unit, integration, and end-to-end tests in your development sandbox.
* **Conformance Testing** ([conformance-tests.md](conformance-tests.md)) * **Conformance Testing** ([conformance-tests.md](conformance-tests.md))
What is conformance testing and how to create/manage them. What is conformance testing and how to create/manage them.

2
contributors/devel/development.md
View File

@ -186,7 +186,7 @@ To check out code to work on, please refer to [this guide](/contributors/guide/g
[build/common.sh]: https://git.k8s.io/kubernetes/build/common.sh [build/common.sh]: https://git.k8s.io/kubernetes/build/common.sh
[e2e-image]: https://git.k8s.io/test-infra/jenkins/e2e-image [e2e-image]: https://git.k8s.io/test-infra/jenkins/e2e-image
[etcd-latest]: https://coreos.com/etcd/docs/latest [etcd-latest]: https://coreos.com/etcd/docs/latest
[etcd-install]: testing.md#install-etcd-dependency [etcd-install]: sig-testing/testing.md#install-etcd-dependency
<!-- https://github.com/coreos/etcd/releases --> <!-- https://github.com/coreos/etcd/releases -->
[go-workspace]: https://golang.org/doc/code.html#Workspaces [go-workspace]: https://golang.org/doc/code.html#Workspaces
[issue]: https://github.com/kubernetes/kubernetes/issues [issue]: https://github.com/kubernetes/kubernetes/issues

227
contributors/devel/sig-testing/testing.md Normal file
View File

@ -0,0 +1,227 @@
# Testing guide
**Table of Contents**
- [Testing guide](#testing-guide)
- [Unit tests](#unit-tests)
- [Run all unit tests](#run-all-unit-tests)
- [Set go flags during unit tests](#set-go-flags-during-unit-tests)
- [Run unit tests from certain packages](#run-unit-tests-from-certain-packages)
- [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package)
- [Stress running unit tests](#stress-running-unit-tests)
- [Unit test coverage](#unit-test-coverage)
- [Benchmark unit tests](#benchmark-unit-tests)
- [Integration tests](#integration-tests)
- [Install etcd dependency](#install-etcd-dependency)
- [Etcd test data](#etcd-test-data)
- [Run integration tests](#run-integration-tests)
- [Run a specific integration test](#run-a-specific-integration-test)
- [End-to-End tests](#end-to-end-tests)
This assumes you already read the [development guide](development.md) to
install go, godeps, and configure your git client. All command examples are
relative to the `kubernetes` root directory.
Before sending pull requests you should at least make sure your changes have
passed both unit and integration tests.
Kubernetes only merges pull requests when unit, integration, and e2e tests are
passing, so it is often a good idea to make sure the e2e tests work as well.
## Unit tests
* Unit tests should be fully hermetic
- Only access resources in the test binary.
* All packages and any significant files require unit tests.
* The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](https://git.k8s.io/kubernetes/test/integration/auth/auth_test.go)
* Unit tests must pass on macOS and Windows platforms.
- Tests using linux-specific features must be skipped or compiled out.
- Skipped is better, compiled out is required when it won't compile.
* Concurrent unit test runs must pass.
* See [coding conventions](../guide/coding-conventions.md).
### Run all unit tests
`make test` is the entrypoint for running the unit tests that ensures that
`GOPATH` is set up correctly. If you have `GOPATH` set up correctly, you can
also just use `go test` directly.
```sh
cd kubernetes
make test # Run all unit tests.
```
If any unit test fails with a timeout panic (see [#1594](https://github.com/kubernetes/community/issues/1594)) on the testing package, you can increase the `KUBE_TIMEOUT` value as shown below.
```sh
make test KUBE_TIMEOUT="-timeout 300s"
```
### Set go flags during unit tests
You can set [go flags](https://golang.org/cmd/go/) by setting the
`GOFLAGS` environment variable.
### Run unit tests from certain packages
`make test` accepts packages as arguments; the `k8s.io/kubernetes` prefix is
added automatically to these:
```sh
make test WHAT=./pkg/api # run tests for pkg/api
```
To run multiple targets you need quotes:
```sh
make test WHAT="./pkg/api ./pkg/kubelet" # run tests for pkg/api and pkg/kubelet
```
In a shell, it's often handy to use brace expansion:
```sh
make test WHAT=./pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet
```
### Run specific unit test cases in a package
You can set the test args using the `KUBE_TEST_ARGS` environment variable.
You can use this to pass the `-run` argument to `go test`, which accepts a
regular expression for the name of the test that should be run.
```sh
# Runs TestValidatePod in pkg/api/validation with the verbose flag set
make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"
```
For other supported test flags, see the [golang
documentation](https://golang.org/cmd/go/#hdr-Testing_flags).
### Stress running unit tests
Running the same tests repeatedly is one way to root out flakes.
You can do this efficiently.
```sh
# Have 2 workers run all tests 5 times each (10 total iterations).
make test PARALLEL=2 ITERATION=5
```
For more advanced ideas please see [flaky-tests.md](flaky-tests.md).
### Unit test coverage
Currently, collecting coverage is only supported for the Go unit tests.
To run all unit tests and generate an HTML coverage report, run the following:
```sh
make test KUBE_COVER=y
```
At the end of the run, an HTML report will be generated with the path
printed to stdout.
To run tests and collect coverage in only one package, pass its relative path
under the `kubernetes` directory as an argument, for example:
```sh
make test WHAT=./pkg/kubectl KUBE_COVER=y
```
Multiple arguments can be passed, in which case the coverage results will be
combined for all tests run.
### Benchmark unit tests
To run benchmark tests, you'll typically use something like:
```sh
go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch
```
This will do the following:
1. `-run=XXX` is a regular expression filter on the name of test cases to run
2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name
* See `grep -nr BenchmarkWatch .` for examples
3. `-benchmem` enables memory allocation stats
See `go help test` and `go help testflag` for additional info.
## Integration tests
* Integration tests should only access other resources on the local machine
- Most commonly etcd or a service listening on localhost.
* All significant features require integration tests.
- This includes kubectl commands
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](https://git.k8s.io/kubernetes/test/integration/auth/auth_test.go)
* Each test should create its own master, httpserver and config.
- Example: [TestPodUpdateActiveDeadlineSeconds](https://git.k8s.io/kubernetes/test/integration/pods/pods_test.go)
* See [coding conventions](coding-conventions.md).
### Install etcd dependency
Kubernetes integration tests require your `PATH` to include an
[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes
includes a script to help install etcd on your machine.
```sh
# Install etcd and add to PATH
# Option a) install inside kubernetes root
hack/install-etcd.sh # Installs in ./third_party/etcd
echo export PATH="\$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH
# Option b) install manually
grep -E "image.*etcd" cluster/gce/manifests/etcd.manifest # Find version
# Install that version using yum/apt-get/etc
echo export PATH="\$PATH:<LOCATION>" >> ~/.profile # Add to PATH
```
### Etcd test data
Many tests start an etcd server internally, storing test data in the operating system's temporary directory.
If you see test failures because the temporary directory does not have sufficient space,
or is on a volume with unpredictable write latency, you can override the test data directory
for those internal etcd instances with the `TEST_ETCD_DIR` environment variable.
### Run integration tests
The integration tests are run using `make test-integration`.
The Kubernetes integration tests are written using the normal golang testing
package but expect to have a running etcd instance to connect to. The `test-integration.sh`
script wraps `make test` and sets up an etcd instance for the integration tests to use.
```sh
make test-integration # Run all integration tests.
```
This script runs the golang tests in package
[`test/integration`](https://git.k8s.io/kubernetes/test/integration).
### Run a specific integration test
You can also use the `KUBE_TEST_ARGS` environment variable with the `make test-integration`
to run a specific integration test case:
```sh
# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
make test-integration WHAT=./test/integration/pods GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$"
```
If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API
version and the watch cache test is skipped.
## End-to-End tests
Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md).

228
contributors/devel/testing.md
View File

@ -1,227 +1,3 @@
# Testing guide This file has moved to https://git.k8s.io/community/contributors/devel/sig-testing/testing.md.
**Table of Contents** This file is a placeholder to preserve links. Please remove by April 30, 2019 or the release of kubernetes 1.13, whichever comes first.
- [Testing guide](#testing-guide)
- [Unit tests](#unit-tests)
- [Run all unit tests](#run-all-unit-tests)
- [Set go flags during unit tests](#set-go-flags-during-unit-tests)
- [Run unit tests from certain packages](#run-unit-tests-from-certain-packages)
- [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package)
- [Stress running unit tests](#stress-running-unit-tests)
- [Unit test coverage](#unit-test-coverage)
- [Benchmark unit tests](#benchmark-unit-tests)
- [Integration tests](#integration-tests)
- [Install etcd dependency](#install-etcd-dependency)
- [Etcd test data](#etcd-test-data)
- [Run integration tests](#run-integration-tests)
- [Run a specific integration test](#run-a-specific-integration-test)
- [End-to-End tests](#end-to-end-tests)
This assumes you already read the [development guide](development.md) to
install go, godeps, and configure your git client. All command examples are
relative to the `kubernetes` root directory.
Before sending pull requests you should at least make sure your changes have
passed both unit and integration tests.
Kubernetes only merges pull requests when unit, integration, and e2e tests are
passing, so it is often a good idea to make sure the e2e tests work as well.
## Unit tests
* Unit tests should be fully hermetic
- Only access resources in the test binary.
* All packages and any significant files require unit tests.
* The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](https://git.k8s.io/kubernetes/test/integration/auth/auth_test.go)
* Unit tests must pass on macOS and Windows platforms.
- Tests using linux-specific features must be skipped or compiled out.
- Skipped is better, compiled out is required when it won't compile.
* Concurrent unit test runs must pass.
* See [coding conventions](../guide/coding-conventions.md).
### Run all unit tests
`make test` is the entrypoint for running the unit tests that ensures that
`GOPATH` is set up correctly. If you have `GOPATH` set up correctly, you can
also just use `go test` directly.
```sh
cd kubernetes
make test # Run all unit tests.
```
If any unit test fails with a timeout panic (see [#1594](https://github.com/kubernetes/community/issues/1594)) on the testing package, you can increase the `KUBE_TIMEOUT` value as shown below.
```sh
make test KUBE_TIMEOUT="-timeout 300s"
```
### Set go flags during unit tests
You can set [go flags](https://golang.org/cmd/go/) by setting the
`GOFLAGS` environment variable.
### Run unit tests from certain packages
`make test` accepts packages as arguments; the `k8s.io/kubernetes` prefix is
added automatically to these:
```sh
make test WHAT=./pkg/api # run tests for pkg/api
```
To run multiple targets you need quotes:
```sh
make test WHAT="./pkg/api ./pkg/kubelet" # run tests for pkg/api and pkg/kubelet
```
In a shell, it's often handy to use brace expansion:
```sh
make test WHAT=./pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet
```
### Run specific unit test cases in a package
You can set the test args using the `KUBE_TEST_ARGS` environment variable.
You can use this to pass the `-run` argument to `go test`, which accepts a
regular expression for the name of the test that should be run.
```sh
# Runs TestValidatePod in pkg/api/validation with the verbose flag set
make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"
```
For other supported test flags, see the [golang
documentation](https://golang.org/cmd/go/#hdr-Testing_flags).
### Stress running unit tests
Running the same tests repeatedly is one way to root out flakes.
You can do this efficiently.
```sh
# Have 2 workers run all tests 5 times each (10 total iterations).
make test PARALLEL=2 ITERATION=5
```
For more advanced ideas please see [flaky-tests.md](flaky-tests.md).
### Unit test coverage
Currently, collecting coverage is only supported for the Go unit tests.
To run all unit tests and generate an HTML coverage report, run the following:
```sh
make test KUBE_COVER=y
```
At the end of the run, an HTML report will be generated with the path
printed to stdout.
To run tests and collect coverage in only one package, pass its relative path
under the `kubernetes` directory as an argument, for example:
```sh
make test WHAT=./pkg/kubectl KUBE_COVER=y
```
Multiple arguments can be passed, in which case the coverage results will be
combined for all tests run.
### Benchmark unit tests
To run benchmark tests, you'll typically use something like:
```sh
go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch
```
This will do the following:
1. `-run=XXX` is a regular expression filter on the name of test cases to run
2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name
* See `grep -nr BenchmarkWatch .` for examples
3. `-benchmem` enables memory allocation stats
See `go help test` and `go help testflag` for additional info.
## Integration tests
* Integration tests should only access other resources on the local machine
- Most commonly etcd or a service listening on localhost.
* All significant features require integration tests.
- This includes kubectl commands
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](https://git.k8s.io/kubernetes/test/integration/auth/auth_test.go)
* Each test should create its own master, httpserver and config.
- Example: [TestPodUpdateActiveDeadlineSeconds](https://git.k8s.io/kubernetes/test/integration/pods/pods_test.go)
* See [coding conventions](coding-conventions.md).
### Install etcd dependency
Kubernetes integration tests require your `PATH` to include an
[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes
includes a script to help install etcd on your machine.
```sh
# Install etcd and add to PATH
# Option a) install inside kubernetes root
hack/install-etcd.sh # Installs in ./third_party/etcd
echo export PATH="\$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH
# Option b) install manually
grep -E "image.*etcd" cluster/gce/manifests/etcd.manifest # Find version
# Install that version using yum/apt-get/etc
echo export PATH="\$PATH:<LOCATION>" >> ~/.profile # Add to PATH
```
### Etcd test data
Many tests start an etcd server internally, storing test data in the operating system's temporary directory.
If you see test failures because the temporary directory does not have sufficient space,
or is on a volume with unpredictable write latency, you can override the test data directory
for those internal etcd instances with the `TEST_ETCD_DIR` environment variable.
### Run integration tests
The integration tests are run using `make test-integration`.
The Kubernetes integration tests are written using the normal golang testing
package but expect to have a running etcd instance to connect to. The `test-integration.sh`
script wraps `make test` and sets up an etcd instance for the integration tests to use.
```sh
make test-integration # Run all integration tests.
```
This script runs the golang tests in package
[`test/integration`](https://git.k8s.io/kubernetes/test/integration).
### Run a specific integration test
You can also use the `KUBE_TEST_ARGS` environment variable with the `make test-integration`
to run a specific integration test case:
```sh
# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
make test-integration WHAT=./test/integration/pods GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$"
```
If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API
version and the watch cache test is skipped.
## End-to-End tests
Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md).

2
contributors/guide/README.md
View File

@ -217,7 +217,7 @@ When reviewing PRs from others [The Gentle Art of Patch Review](http://sage.thes
## Testing ## Testing
Testing is the responsibility of all contributors and is in part owned by all SIGss, but is also coordinated by [sig-testing](/sig-testing). Testing is the responsibility of all contributors and is in part owned by all SIGss, but is also coordinated by [sig-testing](/sig-testing).
Refer to the [Testing Guide](/contributors/devel/testing.md) for more information. Refer to the [Testing Guide](/contributors/devel/sig-testing/testing.md) for more information.
There are multiple types of tests. There are multiple types of tests.
The location of the test code varies with type, as do the specifics of the environment needed to successfully run the test: The location of the test code varies with type, as do the specifics of the environment needed to successfully run the test:

2
contributors/guide/coding-conventions.md
View File

@ -86,7 +86,7 @@ required when your code does not compile on Windows).
asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be
running). Wait and retry instead. running). Wait and retry instead.
- See the [testing guide](/contributors/devel/testing.md) for additional testing advice. - See the [testing guide](/contributors/devel/sig-testing/testing.md) for additional testing advice.
## Directory and file conventions ## Directory and file conventions

2
contributors/guide/github-workflow.md
View File

@ -149,7 +149,7 @@ make test
make test WHAT=./pkg/api/helper GOFLAGS=-v make test WHAT=./pkg/api/helper GOFLAGS=-v
# Run integration tests, requires etcd # Run integration tests, requires etcd
# For more info, visit https://git.k8s.io/community/contributors/devel/testing.md#integration-tests # For more info, visit https://git.k8s.io/community/contributors/devel/sig-testing/testing.md#integration-tests
make test-integration make test-integration
# Run e2e tests by building test binaries, turn up a test cluster, run all tests, and tear the cluster down # Run e2e tests by building test binaries, turn up a test cluster, run all tests, and tear the cluster down

2
sig-cli/CONTRIBUTING.md
View File

@ -421,6 +421,6 @@ See the sig-cli [community page] for points of contact and meeting times:
[release]: #release [release]: #release
[slack-messages]: https://kubernetes.slack.com/messages/sig-cli [slack-messages]: https://kubernetes.slack.com/messages/sig-cli
[slack-signup]: http://slack.k8s.io/ [slack-signup]: http://slack.k8s.io/
[tests]: /contributors/devel/testing.md [tests]: /contributors/devel/sig-testing/testing.md
[cli mentors]: https://groups.google.com/a/google.com/forum/#!forum/kubernetes-sig-cli-mentors [cli mentors]: https://groups.google.com/a/google.com/forum/#!forum/kubernetes-sig-cli-mentors
[about me form]: https://docs.google.com/forms/d/1ID6DX1abiDr9Z9_sXXC0DsMwuyHb_NeFdB3xeRa4Vf0 [about me form]: https://docs.google.com/forms/d/1ID6DX1abiDr9Z9_sXXC0DsMwuyHb_NeFdB3xeRa4Vf0

2
sig-multicluster/CONTRIBUTING.md
View File

@ -295,4 +295,4 @@ See the sig-multicluster [community page] for points of contact and meeting time
[release]: #release [release]: #release
[slack-messages]: https://kubernetes.slack.com/messages/sig-multicluster [slack-messages]: https://kubernetes.slack.com/messages/sig-multicluster
[slack-signup]: http://slack.k8s.io/ [slack-signup]: http://slack.k8s.io/
[tests]: /contributors/devel/testing.md [tests]: /contributors/devel/sig-testing/testing.md