Commit Graph

2 Commits

Author SHA1 Message Date
Alejandro Pedraza f62a2e6ee4
Refactor integration tests to use annotations functions (#4341)
* Refactor integration tests to use annotations functions

First part of #4176

Replaced all the `t.Error`/`t.Fatal` calls in the integration with the
new functions defined in `testutil/annotations.go` as described in #4292,
in order for the errors to produce Github annotations.

Most of these calls have now two strings: one containing a generic error
message and another with a more specific message. The former is what
will be aggregated and seen in the CI reports at
[linkerd2-ci-metrics](https://github.com/linkerd/linkerd2-ci-metrics).

Other changes:

- Improved the annotation generator in `annotations.go` so that the
  message includes the name of the test.
- When a failure from `RetryFor` occurs, log the original timeout so
  we can consider incrementing it when the failure is persistent.
2020-05-08 08:41:42 -05:00
Alejandro Pedraza 2cd48bc488
Go test failure message wrappers to create GH Annotations (#4292)
* Go test failure message wrappers to create GH Annotations

First part of #4176

## Problem

Failures in go tests need to be properly formatted as Github annotations
so that we can fetch them through Github's API for aggregation and
analysis.

## Solution

A wrapper for error messages has been created in `testutil/annotations.go`.
The idea is that instead of throwing test failures like this:

```go
t.Failf("error retrieving data;\nExpected: %#v\nActual: %#v", expected,
actual)
```

We'd throw them like this:
```go
testutil.AnnotationFatalf("error retrieving data", "error retrieving data;\nExpected: %#v\nActual: %#v", expected,
actual)
```

That will continue reporting the error as before (when using `go test`
or another test runner), but as a side-effect it will also send to
stdout something like:

```
::error file=pkg/inject_test.go,line=133::error retrieving data
```
Which becomes a GH annotation, visible in the CI run summary screen.

The fist string art is used to have the GH annotation be a generic error message
that can be aggregated and counted across multiple test runs. If `testutil.Fatalf(str, args...)`
is called instead, the original error message will be used.

Note that that the output will be produced only when the env var
`GH_ANNOTATION` is set (which will when tests are triggered from a
Github Actions workflow).

Besides `testutil/annotation.go` and its accompanying unit test file,
other changes were made in other tests as examples, the plan being that
in a further PR _all_ the tests will use these wrappers.
2020-05-01 16:16:06 -05:00