56 lines
2.9 KiB
Markdown
56 lines
2.9 KiB
Markdown
# Writing Good Conformance Tests for Kubernetes #
|
|
|
|
The overarching goal of conformance tests is to exercise core
|
|
Kubernetes functionality in (as much as possible) a provider-agnostic
|
|
way.
|
|
|
|
Conformance tests are promoted from existing e2e tests which have
|
|
been proven to be stable and "non-flaky".
|
|
|
|
Please read the the ["Writing good e2e tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md) guide as a supplement to this document.
|
|
|
|
### Requirements ###
|
|
|
|
In addition to the "Writing good e2e tests" guide, there are specific
|
|
requirements that all Conformance tests must adhere to. You can find
|
|
the definitive list of requirements in the [SIG-Arch community
|
|
docs](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements).
|
|
|
|
Generally speaking, tests should not target internal behaviors, but
|
|
instead target exposed system behaviors. Care must also be taken to ensure that
|
|
the tests do not implicitly test features that are not already subject to
|
|
conformance. If a feature is needed for a test, make sure that an existing test
|
|
already covers that feature directly.
|
|
|
|
### Existing Tests ###
|
|
|
|
When looking for an existing e2e test which exercises a behaviour
|
|
which is useful for Conformance, you may find that the test does not
|
|
meet all of the [requirements](#Requirements). In these cases, you
|
|
might either modify the test to meet requirements, but sometimes doing
|
|
so will mean losing some of the intent of the test.
|
|
|
|
Instead, a better approach may be to duplicate the test, and make your
|
|
modifications, thus preserving the original test as-is.
|
|
|
|
Any existing e2e test should have gone through a standard review
|
|
process by a respective [SIG](https://github.com/kubernetes/community/blob/master/sig-list.md)
|
|
in order to have been merged in, which means that the test was most likely
|
|
reviewed by domain-specific experts. This can be useful to note when
|
|
modifying tests to better suit conformance concerns, or meet
|
|
[requirements](#Requirements). That said, existings tests are of varying levels
|
|
of quality. It is important to verify that the test actually does validate the
|
|
expected behavior, as described in any API documentation, and as specified in
|
|
the test description you add for promotion.
|
|
|
|
### New Tests ###
|
|
|
|
New tests must go through the standard process for e2e tests even when
|
|
intended for Conformance from the onset. This means working with
|
|
SIG-specific reviewers in order to get your PR accepted. After which,
|
|
the test must be in the test cycle for 2 weeks, to prove its
|
|
stability. If the test is not slow or flaky, then you may begin the
|
|
promotion process by following the steps detailed in ["Promoting
|
|
Tests to
|
|
Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance).
|