History

hasheddan 1016d17ce3 [flake finders] fix episode 000 readme formatting Fixes code block formatting in guide for flake finders episode 000. Signed-off-by: hasheddan <georgedanielmangum@gmail.com>		2021-02-28 15:19:25 -06:00
..
README.md	[flake finders] fix episode 000 readme formatting	2021-02-28 15:19:25 -06:00

Flake Finder Fridays #0

February 5th 2021 (Recording)

Introduction

This is the first episode of Flake Finder Fridays with Dan Mangum and Rob Kielty.

On the first friday of every month we will go through an issue that was logged for a failing or flaking test on the Kubernetes project.

We will review the triage, root cause analysis, and problem resolution for a test related issue logged in the past four weeks.

We intend to demo how CI works on the Kubernetes project and also how we collaborate across teams to resolve test maintenance issues.

Desire to move from Google-owned infrastructure to Kubernetes community infrastructure. Thus the introduction of a canary build job to test pushing building and pushing artifacts with new infrastructure.
Desire to move off of bootstrap.py job (currently being used for canary job) to krel tooling.
Separate job existed (ci-kubernetes-build-no-bootstrap) that was doing the same thing as the canary job, but with krel tooling.
The no-bootstrap job was running smoothly, so updated to use it for the canary job.
Right before the update, we switched to using buildx for multi-arch images.
Job started failing, which showed up in some interesting ways.
Triage begins! Issue opened and release management team is pinged in Slack.
The build-master job was still passing though... interesting.
Both are eventually calling make release, so environment must be different.

Let's look inside!

docker run -it --entrypoint /bin/bash gcr.io/k8s-testimages/bootstrap:v20210130-12516b2

docker run -it gcr.io/k8s-staging-releng/k8s-ci-builder:v20201128-v0.6.0-6-g6313f696-default /bin/bash

A few directions we could go here:
1. Update the k8s-ci-builder image to you use newer version of Docker
2. Update the k8s-ci-builder image to ensure that DOCKER_CLI_EXPERIMENTAL=enabled is set
3. Update the release.sh script to set DOCKER_CLI_EXPERIMENTAL=enabled
Making the release.sh script more flexible serves the community better because it allows for building with more environments. Would also be good to update the k8s-ci-builder image for this specific case as well.
And we get a new failure!
Let's see what is going on in those images again...
Why would this cause an error in one but not the other if we have DOCKER_CLI_EXPERIMENTAL=enabled? (this is why)
In the mean time we went ahead and re-enabled the bootstrap job (consumers of those images need them!)
Decided to increase logging verbosity on failures to see if that would give us a clue into what was going wrong (and to remove those annoying quiet currently not implemented warnings).
Job turns green! But how?
Buildx is versioned separately than Docker itself. Turns out that the --quiet flag warning was actually an error until v0.5.1 of Buildx.
The build-master job was running with buildx v0.5.1 while the krel job was running with v0.4.2. This meant the quiet flag was causing an error in the krel job, and removing it alleviated the error.
Finished up by once again removing the bootstrap job.

Brand new to the project?

Setup already and interested in maintaining tests?

Check out this video from Jordan Liggit who describes strategies and tactics to deflake flaking tests (Jordan's show notes for that talk)

Here's how the CI Signal Team actively monitors CI during a release cycle: