e2e test failures are rife with messages like:
Expected 1 to equal 0
These make me cry. They're anti-helpful, requiring the reader
to dive into the source code to figure out what those numbers
mean.
Solution: Go tests have a '.Should(Exit(NNN))' mechanism. I
don't know if it spits out a better diagnostic (I have no way
to run e2e tests on my laptop), but I have to fantasize that
it will, and given the state of our flakes I assume that at
least one test will fail and give me the opportunity to see
what the error message looks like.
THIS IS NOT REVIEWABLE CODE. There is no way for a human
to review it. Don't bother. Maybe look at a few random
ones for sanity. If you want to really review, here is
a reproducer of what I did:
cd test/e2e
! positive assertions. The second is the same as the first,
! with the addition of (unnecessary) parentheses because
! some invocations were written that way. The third is BeZero().
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Equal\((\d+)\)\)/Expect($1).Should(Exit($2))/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(\(Equal\((\d+)\)\)\)/Expect($1).Should(Exit($2))/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(BeZero\(\)\)/Expect($1).Should(Exit(0))/' *_test.go
! Same as above, but handles three non-numeric exit codes
! in run_exit_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Equal\((\S+)\)\)/Expect($1).Should(Exit($2))/' *_test.go
! negative assertions. Difference is the spelling of 'To(Not)',
! 'ToNot', and 'NotTo'. I assume those are all the same.
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Not\(Equal\((0)\)\)\)/Expect($1).To(ExitWithError())/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.ToNot\(Equal\((0)\)\)/Expect($1).To(ExitWithError())/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.NotTo\(Equal\((0)\)\)/Expect($1).To(ExitWithError())/' *_test.go
! negative, old use of BeZero()
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.ToNot\(BeZero\(\)\)/Expect($1).Should(ExitWithError())/' *_test.go
Run those on a clean copy of main branch (at the same branch
point as my PR, of course), then diff against a checked-out
copy of my PR. There should be no differences. Then all you
have to review is that my replacements above are sane.
UPDATE: nope, that's not enough, you also need to add gomega/gexec
to the files that don't have it:
perl -pi -e '$_ .= "$1/gexec\"\n" if m!^(.*/onsi/gomega)"!' $(grep -L gomega/gexec $(git log -1 --stat | awk '$1 ~ /test\/e2e\// { print $1}'))
UPDATE 2: hand-edit run_volume_test.go
UPDATE 3: sigh, add WaitWithDefaultTimeout() to a couple of places
UPDATE 4: skip a test due to bug #10935 (race condition)
Signed-off-by: Ed Santiago <santiago@redhat.com>
We missed bumping the go module, so let's do it now :)
* Automated go code with github.com/sirkon/go-imports-rename
* Manually via `vgrep podman/v2` the rest
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Change API Handlers to use the same functions that the
local podman uses.
At the same time:
implement remote API for --all and --ignore flags for podman stop
implement remote API for --all flags for podman stop
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
in an effort to speed up the remote testing, we should be using
lookaside storage to avoid pull images as well as importing multiple
images into the RW store.
one test was removed and added into system test by Ed in #8325
Signed-off-by: baude <bbaude@redhat.com>
Remove ones that are not needed.
Document those that should be there.
Document those that should be fixed.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
With the advent of Podman 2.0.0 we crossed the magical barrier of go
modules. While we were able to continue importing all packages inside
of the project, the project could not be vendored anymore from the
outside.
Move the go module to new major version and change all imports to
`github.com/containers/libpod/v2`. The renaming of the imports
was done via `gomove` [1].
[1] https://github.com/KSubedi/gomove
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Rather than checking for non-zero, we need to check for >0 to
distinguish between timeouts and error exit codes.
Signed-off-by: Jhon Honce <jhonce@redhat.com>
when doing localized tests (not varlink), we can use secondary image
stores as read-only image caches. this cuts down on test time
significantly because each test does not need to restore the images from
a tarball anymore.
Signed-off-by: baude <bbaude@redhat.com>
a series of improvements to our ginkgo test framework so we can
get better ideas of whats going on when run in CI
Signed-off-by: baude <bbaude@redhat.com>
We have little to no testing to make sure we don't break podman image and
podman container commands that wrap traditional commands.
This PR adds tests for each of the commands.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add the ability to run the integration (ginkgo) suite using
the remote client.
Only the images_test.go file is run right now; all the rest are
isolated with a // +build !remotelinux. As more content is
developed for the remote client, we can unblock the files and
just block single tests as needed.
Signed-off-by: baude <bbaude@redhat.com>
Because our tests are getting so long, we want to be able to audit which tests are taking
the longest to complete. This may indicate a bad test, bad CI, bad code, etc and therefore
should be auditable.
Also, make speed improvements to tests by making sure we only unpack caches images that
actually get used.
Signed-off-by: baude <bbaude@redhat.com>
Closes: #1178
Approved by: mheon
As Matt pointed out, when running sleep in a container, the clean up was taking a
full ten seconds to stop container because sleep does not catch SIGTERM which is
the default podman stop signal and it had to wait for SIGKILL. Changing sleep to
top should result in better test times.
Signed-off-by: baude <bbaude@redhat.com>
Closes: #492
Approved by: rhatdan
Valentin Rothberg
Ed Santiago
Daniel J Walsh
baude
Jhon Honce
Yiqiao Pu