diff --git a/contributors/design-proposals/release-test-signal.md b/contributors/design-proposals/release-test-signal.md new file mode 100644 index 000000000..6975629d3 --- /dev/null +++ b/contributors/design-proposals/release-test-signal.md @@ -0,0 +1,130 @@ +# Overview + +Describes the process and tooling (`find_green_build`) used to find a +binary signal from the Kubernetes testing framework for the purposes of +selecting a release candidate. Currently this process is used to gate +all Kubernetes releases. + +## Motivation + +Previously, the guidance in the [(now deprecated) release document](https://github.com/kubernetes/kubernetes/blob/fc3ef9320eb9d8211d85fbc404e4bbdd751f90af/docs/devel/releasing.md) +was to "look for green tests". That is, of course, decidedly insufficient. + +Software releases should have the goal of being primarily automated and +having a gating binary test signal is a key component to that ultimate goal. + +## Design + +### General + +The idea is to capture and automate the existing manual methods of +finding a green signal for testing. + +* Identify a green run from the primary job `ci-kubernetes-e2e-gce` +* Identify matching green runs from the secondary jobs + +The tooling should also have a simple and common interface whether using it +for a dashboard, to gate a release within anago or for an individual to use it +to check the state of testing at any time. + +Output looks like this: + +``` +$ find_green_build +find_green_build: BEGIN main on djmm Mon Dec 19 16:28:15 PST 2016 + +Checking for a valid github API token: OK +Checking required system packages: OK +Checking/setting cloud tools: OK + +Getting ci-kubernetes-e2e-gce build results from Jenkins... +Getting ci-kubernetes-e2e-gce-serial build results from Jenkins... +Getting ci-kubernetes-e2e-gce-slow build results from Jenkins... +Getting ci-kubernetes-kubemark-5-gce build results from Jenkins... +Getting ci-kubernetes-e2e-gce-reboot build results from Jenkins... +Getting ci-kubernetes-e2e-gce-scalability build results from Jenkins... +Getting ci-kubernetes-test-go build results from Jenkins... +Getting ci-kubernetes-cross-build build results from Jenkins... +Getting ci-kubernetes-e2e-gke-serial build results from Jenkins... +Getting ci-kubernetes-e2e-gke build results from Jenkins... +Getting ci-kubernetes-e2e-gke-slow build results from Jenkins... + +(*) Primary job (-) Secondary jobs + +Jenkins Job Run # Build # Time/Status += ================================= ====== ======= =========== +* ci-kubernetes-e2e-gce #1668 #2347 [14:46 12/19] +* (--buildversion=v1.6.0-alpha.0.2347+9925b68038eacc) +- ci-kubernetes-e2e-gce-serial -- -- GIVE UP + +* ci-kubernetes-e2e-gce #1666 #2345 [13:23 12/19] +* (--buildversion=v1.6.0-alpha.0.2345+523ff93471b052) +- ci-kubernetes-e2e-gce-serial -- -- GIVE UP + +* ci-kubernetes-e2e-gce #1664 #2341 [09:38 12/19] +* (--buildversion=v1.6.0-alpha.0.2341+def802272904c0) +- ci-kubernetes-e2e-gce-serial -- -- GIVE UP + +* ci-kubernetes-e2e-gce #1662 #2339 [08:45 12/19] +* (--buildversion=v1.6.0-alpha.0.2339+ce67a03b81dee5) +- ci-kubernetes-e2e-gce-serial -- -- GIVE UP + +* ci-kubernetes-e2e-gce #1653 #2335 [07:42 12/19] +* (--buildversion=v1.6.0-alpha.0.2335+d6046aab0e0678) +- ci-kubernetes-e2e-gce-serial #192 #2335 PASSED +- ci-kubernetes-e2e-gce-slow #989 #2335 PASSED +- ci-kubernetes-kubemark-5-gce #2602 #2335 PASSED +- ci-kubernetes-e2e-gce-reboot #1523 #2335 PASSED +- ci-kubernetes-e2e-gce-scalability #460 #2335 PASSED +- ci-kubernetes-test-go #1266 #2335 PASSED +- ci-kubernetes-cross-build -- -- GIVE UP + +* ci-kubernetes-e2e-gce #1651 #2330 [06:43 12/19] +* (--buildversion=v1.6.0-alpha.0.2330+75dfb21018a7c3) +- ci-kubernetes-e2e-gce-serial #191 #2319 PASSED +- ci-kubernetes-e2e-gce-slow #988 #2330 PASSED +- ci-kubernetes-kubemark-5-gce #2599 #2330 PASSED +- ci-kubernetes-e2e-gce-reboot #1521 #2330 PASSED +- ci-kubernetes-e2e-gce-scalability #459 #2321 PASSED +- ci-kubernetes-test-go #1264 #2330 PASSED +- ci-kubernetes-cross-build #320 #2330 PASSED +- ci-kubernetes-e2e-gke-serial #233 #2319 PASSED +- ci-kubernetes-e2e-gke #1834 #2330 PASSED +- ci-kubernetes-e2e-gke-slow #1041 #2330 PASSED + +JENKINS_BUILD_VERSION=v1.6.0-alpha.0.2330+75dfb21018a7c3 +RELEASE_VERSION[alpha]=v1.6.0-alpha.1 +RELEASE_VERSION_PRIME=v1.6.0-alpha.1 +``` + +### v1 + +The initial release of this analyzer did everything on the client side. +This was slow to grab 100s of individual test results from GCS. +This was mitigated somewhat by building a local cache, but for those that +weren't using it regularly, the cache building step was a significant +(~1 minute) hit when just trying to check the test status. + +### v2 + +Building and storing that local cache on the jenkins server at build time +was the way to speed things up. Getting the cache from GCS is now consistent +for all users at ~10 seconds. After that the analyzer is running. + + +## Uses + +`find_green_build` and its functions are used in 3 ways: + +1. During the release process itself via `anago`. +1. When creating a pending release notes report via `relnotes --preview`, + used in creating dashboards +1. By an individual to get a quick check on the binary signal status of jobs + +## Future work + +1. There may be other ways to improve the performance of this check by + doing more work server side. +1. Using the `relnotes --preview` output to generate an external dashboard + will give more real-time visibility to both candidate release notes and + testing state.