Commit 2 of 2: individual special-case handling of tests
which did not pass under ExitCleanly(), one or more of:
- add "-q" to commit & push commands
- add ErrorToString() checks
- remove unnecessary ErrorToString() checks
Signed-off-by: Ed Santiago <santiago@redhat.com>
Final push on RUN-1907. Commit 1 of 2.
This is the final set of test/e2e/*_test.go files to be
converted from Exit(0) to ExitCleanly().
This commit is a mix of automated string-replace with
manual revert-back: tests that did not pass with ExitCleanly()
are reverted back to Exit(0), so they will not show up as
diffs in this commit. When possible, I address those in
my next commit.
My goal was to make this commit a don't-bother-reviewing one
that will also pass tests (so as not to break git-bisect).
The next commit is the important one to review.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Remove "HTTP if one registry" test. It is a NOP, has been skipped
for two months, and nobody knows what its original purpose was.
Closes: #18768
Signed-off-by: Ed Santiago <santiago@redhat.com>
- trust_test: adding 'Ordered' seems to resolve a very common
flake. I've tested this for dozens of CI runs, and haven't
seen the flake recur (normally it fails every few runs).
- exec and search tests: add FlakeAttempts(3). This is a NOP
under our current CI setup, in which we run ginkgo with
a global --flake-attempts=3. I am submitting this as an
optimistic step toward a no-flake-attempts world (#17967)
Fixes: #18358
Signed-off-by: Ed Santiago <santiago@redhat.com>
Intented to fix an obscure, unlikely race condition in which (I
think) two parallel jobs called GetPort() and were assigned the
same port.
Also, add actual proper testing to two HTTP-registry tests, and
Skip a third that's a waste of cycles (filed #18768)
Signed-off-by: Ed Santiago <santiago@redhat.com>
There is no reason to define the same code every time in each file, just
use global nodes. This diff should speak for itself.
CleanupSecrets()/Volume() no longer call Cleanup() directly, as the
global AfterEach node will always call Cleanup() this is no longer
necessary. If one AfterEach() node fails it will still run the others.
Also always unset the CONTAINERS_CONF env vars. This prevents people
from forgetting to unset it. And fix the special CONTAINERS_CONF logic
in the system connection tests, we do not want to preserve
CONTAINERS_CONF anyway so just remove this logic.
Ginkgo orders the BeforeEach and AfterEach nodes. They will be executed
from the outer-most defined to inner-most. This means our global
BeforeEach is always first. Only then the inner one (in the Describe()
function in each file). For AfterEach it is inverted, from the inner to
the outer.
Also see https://onsi.github.io/ginkgo/#organizing-specs-with-container-nodes
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
I found the ginkgolinter[1] by accident, this looks for not optimal
matching and suggest how to do it better.
Overall these fixes seem to be all correct and they will give much
better error messages when something fails.
Check out the repo to see what the linter reports.
[1] https://github.com/nunnatsa/ginkgolinter
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The current test uses registry.redhat.io which does not support unauthenticated access
Replace the registry with registry.access.redhat.com which does
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
There's no guarantee that the searched image will be returned, so only
make sure that "alpine" is mentioned somewhere.
Fixes: #16248
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Package `io/ioutil` was deprecated in golang 1.16, preventing podman from
building under Fedora 37. Fortunately, functionality identical
replacements are provided by the packages `io` and `os`. Replace all
usage of all `io/ioutil` symbols with appropriate substitutions
according to the golang docs.
Signed-off-by: Chris Evich <cevich@redhat.com>
A number of standard image names were lower-case, leading to
confusion in code such as:
registry := podman(... , "-n", "registry", registry, ...)
^--- variable ^---- constant
Fix a number of those to be capitalized and with _IMAGE suffix:
registry := podman(..., REGISTRY_IMAGE
Signed-off-by: Ed Santiago <santiago@redhat.com>
The errcheck linter makes sure that errors are always check and not
ignored by accident. It spotted a lot of unchecked errors, mostly in the
tests but also some real problem in the code.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The search endpoint on registry.redhat.io is broken.
Skip one test and update another to avoid hitting it.
Also leave some breadcrumbs to revert once it's back
in a working state.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Write a BeValidJSON() matcher, and replace IsJSONOutputValid():
sed -i -e 's/Expect(\(.*\)\.IsJSONOutputValid()).To(BeTrue())/Expect(\1.OutputToString())\.To(BeValidJSON())/' test/e2e/*_test.go
(Plus a few manual tweaks)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Many ginkgo tests have been written to use this evil form:
GrepString("foo")
Expect(that to BeTrue())
...which yields horrible useless messages on failure:
false is not true
Identify those (automatically, via script) and convert to:
Expect(output to ContainSubstring("foo"))
...which yields:
"this output" does not contain substring "foo"
There are still many BeTrue()s left. This is just a start.
This is commit 1 of 2. It includes the script I used, and
all changes to *.go are those computed by the script.
Commit 2 will apply some manual fixes.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Change the default format of `podman search` to only display the name
and the description of each image. The index is redundant to the name
and consumes a lot of space, and other descriptors (i.e., stars,
official, automated) are specific to Docker Hub and also consume a lot
space. Users can still use `--format` for displaying the descriptors
they want to.
Add a `--compatible` flag to offer an easy way to get them back.
Also update the man page to account for the behavior and get some fresh
data in the examples.
Motivated by a recent conversation in libimage:
https://github.com/containers/common/pull/802#issuecomment-937108734
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
* JSON and API description fields are no longer truncated. Formatting
moved to client, better support of MVP.
* --no-trunc now defaults to true
* Updated tests for changes
Closes#11894
Signed-off-by: Jhon Honce <jhonce@redhat.com>
Some search tests were looking for an explicit amount of images to
match. Since images are moving targets on these registries, make
sure to use lower bounds instead of exact matches.
Fixes CI which started to break when Red Hat images changed.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
e2e test failures are rife with messages like:
Expected 1 to equal 0
These make me cry. They're anti-helpful, requiring the reader
to dive into the source code to figure out what those numbers
mean.
Solution: Go tests have a '.Should(Exit(NNN))' mechanism. I
don't know if it spits out a better diagnostic (I have no way
to run e2e tests on my laptop), but I have to fantasize that
it will, and given the state of our flakes I assume that at
least one test will fail and give me the opportunity to see
what the error message looks like.
THIS IS NOT REVIEWABLE CODE. There is no way for a human
to review it. Don't bother. Maybe look at a few random
ones for sanity. If you want to really review, here is
a reproducer of what I did:
cd test/e2e
! positive assertions. The second is the same as the first,
! with the addition of (unnecessary) parentheses because
! some invocations were written that way. The third is BeZero().
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Equal\((\d+)\)\)/Expect($1).Should(Exit($2))/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(\(Equal\((\d+)\)\)\)/Expect($1).Should(Exit($2))/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(BeZero\(\)\)/Expect($1).Should(Exit(0))/' *_test.go
! Same as above, but handles three non-numeric exit codes
! in run_exit_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Equal\((\S+)\)\)/Expect($1).Should(Exit($2))/' *_test.go
! negative assertions. Difference is the spelling of 'To(Not)',
! 'ToNot', and 'NotTo'. I assume those are all the same.
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.To\(Not\(Equal\((0)\)\)\)/Expect($1).To(ExitWithError())/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.ToNot\(Equal\((0)\)\)/Expect($1).To(ExitWithError())/' *_test.go
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.NotTo\(Equal\((0)\)\)/Expect($1).To(ExitWithError())/' *_test.go
! negative, old use of BeZero()
perl -pi -e 's/Expect\((\S+)\.ExitCode\(\)\)\.ToNot\(BeZero\(\)\)/Expect($1).Should(ExitWithError())/' *_test.go
Run those on a clean copy of main branch (at the same branch
point as my PR, of course), then diff against a checked-out
copy of my PR. There should be no differences. Then all you
have to review is that my replacements above are sane.
UPDATE: nope, that's not enough, you also need to add gomega/gexec
to the files that don't have it:
perl -pi -e '$_ .= "$1/gexec\"\n" if m!^(.*/onsi/gomega)"!' $(grep -L gomega/gexec $(git log -1 --stat | awk '$1 ~ /test\/e2e\// { print $1}'))
UPDATE 2: hand-edit run_volume_test.go
UPDATE 3: sigh, add WaitWithDefaultTimeout() to a couple of places
UPDATE 4: skip a test due to bug #10935 (race condition)
Signed-off-by: Ed Santiago <santiago@redhat.com>
We missed bumping the go module, so let's do it now :)
* Automated go code with github.com/sirkon/go-imports-rename
* Manually via `vgrep podman/v2` the rest
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
podman-remote search had some FIXMEs in tests that were failing.
So I reworked the search handler to use the local abi. This
means the podman search and podman-remote search will use the
same functions.
While doing this, I noticed we were just outputing errors via
logrus.Error rather then returning them, which works ok for
podman but the messages get lost on podman-remote. Changed
the code to actually return the error messages to the caller.
This allows us to turn on the remaining podman-remote FIXME
tests.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Adding another check in the `podman search --list-tags --format json` test case.
Replacing an anonymous struct by \`listEntryTag\` struct.
Signed-off-by: Alexandre Fourcat <afourcat@gmail.com>
Remove references on contrib/gate. The gating image is currently not
maintained and was replaced in the CI rewrite. We can disable builds
in Quay once the change is merged.
Note that running `make validate` in a container is still desired.
A similar approach may be re-added in the future.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
in an effort to speed up the remote testing, we should be using
lookaside storage to avoid pull images as well as importing multiple
images into the RW store.
one test was removed and added into system test by Ed in #8325
Signed-off-by: baude <bbaude@redhat.com>
Remove the search limit check since the c/image v5.6.0 supports pagination and can give result over 100 entries.
Signed-off-by: Qi Wang <qiwan@redhat.com>
These options are now fully supported in the remote API and should no
longer be hidden and/or documented as non supported.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Now that Dan has added helpful comments to each SkipIfRemote,
let's take the next step and include those messages in the
Skip() output so someone viewing test results can easily
see if a remote test is skipped for a real reason or for
a FIXME.
This commit is the result of a simple:
perl -pi -e 's;(SkipIfRemote)\(\)(\s+//\s+(.*))?;$1("$3");' *.go
in the test/e2e directory, with a few minor (manual) changes
in wording.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Remove ones that are not needed.
Document those that should be there.
Document those that should be fixed.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Ed Santiago
Paul Holzinger
Black-Hole1
Ygal Blum
Daniel J Walsh
Valentin Rothberg
Chris Evich
Giuseppe Scrivano
Brent Baude
Valentin Rothberg
Jhon Honce
Alexandre Fourcat
Josh Soref
Paul Holzinger
Qi Wang
Sascha Grunert
Ralf Haferkamp