From 3a13fb1487badd1c746978024fec964442d6e3b3 Mon Sep 17 00:00:00 2001 From: Tim Pepper Date: Wed, 19 Sep 2018 17:08:46 -0700 Subject: [PATCH] devel guide: note where to track flaky tests We have a good document on investigating flaky tests, but it can be hard to correlate personal experience (my PR wont pass presubmits!?!) with broader CI health (did AWS go down?). For that portion of triage it's useful to look at some aggregated statistics and we have two nice pages for that. I'm always forgetting these two links though and while some kind folks have reminded me repeatedly, I take that as a sign it's time to put them somewhere more visible. Signed-off-by: Tim Pepper --- contributors/devel/flaky-tests.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/contributors/devel/flaky-tests.md b/contributors/devel/flaky-tests.md index f06bb078a..14f8da9e7 100644 --- a/contributors/devel/flaky-tests.md +++ b/contributors/devel/flaky-tests.md @@ -15,6 +15,19 @@ what caused the failure. Note that flakes can occur in unit tests, integration tests, or end-to-end tests, but probably occur most commonly in end-to-end tests. +## Hunting Flakes + +You may notice lots of your PRs or ones you watch are having a common +pre-submit failure, but less frequent issues that are still of concern take +more analysis over time. There are metrics recorded and viewable in: +- [TestGrid](https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking#Summary) +- [Velodrome](http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1) + +It is worth noting tests are going to fail in presubmit a lot due +to unbuildable code, but that wont happen as much on the same commit unless +there's a true issue in the code or a broader problem like a dep failed to +pull in. + ## Filing issues for flaky tests Because flakes may be rare, it's very important that all relevant logs be