This test is currently disabled due to several issues, only some of which
are described in the existing comments. Add some more details to clarify
the situation.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
This name for the tests is misleading, since in the default configuration
podman will already configure a forwarding addres, which could forward
to either another local forwarder or an external nameserver on the host
side. What this test is really about is explicitly configuring the pasta
DNS forwarding address. Rename accordingly.
The IPv4 version of the test doesn't use the podman --dns option, only
the pasta --dns-forward option. This exercises the podman behaviour that
pasta --dns-forward options are added to /etc/resolv.conf automatically.
However there could also be other things in /etc/resolv.conf, so the
nslookup might not use the custom forwarding address for the lookup.
To fix that, split the test into two parts: one verifying that the custom
address is in /etc/resolv.conf and another performing the nslookup with an
explicit server address to make sure we exercise the pasta side as well.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
In both the "Basic nameserver lookup" and "Local forwarder, IPv4" pasta
tests, we check whether DNS resolution is working by running "nslookup
127.0.0.1" in the container and checking if 1.0.0.127.in-addr.arpa is in
the output.
1.0.0.127.in-addr.arpa isn't the expected result of the resolution though,
it's just the DNS name that nslookup will tranlated 127.0.0.1 into. The
test mostly works, because nslookup echoes that on successful lookups.
However, it could also echo it in certain sorts of failure, so it's not a
very reliable test.
Furthermore, resolving 127.0.0.1 from a nameserver is a rather strange
thing to do. It's done that way because RFC1912[0] suggests it should
always resolve, even for nameservers on a disconnected network. But, this
doesn't really appear to be true in practice: a number of resolvers return
NXDOMAIN. That works by accident because nslookup seems to echo the
name above as part of the error message.
Change to instead looking up one of the root servers by name. This does
now rely on access to the global DNS during tests, but other podman tests
attempt to resolve google.com, so that should be ok. One of the root
servers is about as close to universal resolvability as it's possible to
get
[0] https://datatracker.ietf.org/doc/html/rfc1912#section-4.1
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The idea behind the "External resolver" tests is simply to check that we
can contact a nameserver, regardless of this configuration. To this end
the "IPv4" version looks up 127.0.0.1 which RFC1912[0] suggests should
always be resolvable.
The IPv6 version instead looks up [::1]. While it makes sense for
that to be resolvable in a similar way, there appear to be quite a few
nameservers which do not resolve it, making this test flaky.
Furthermore the idea behind resolving [::1] is that it should make
nslookup prefer to resolve over IPv6. That appears to be very
unreliable at best. Since making a different query doesn't actually
exercise anything different in pasta, drop the test.
The remaining IPv4 test isn't really specific to an "external" resolver,
it's simply checking that we can contact some sort of resolver with the
default podman configuration. Rename accordingly, and run it regardless of
IPv4 connectivity on the host: we can still query a nameserver about an
IPv4 address, even if we only have IPv6 connectivity ourselves.
[0] https://datatracker.ietf.org/doc/html/rfc1912#section-4.1
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The "Local forwarder, IPv4" pasta test, amongst other things, checks that
podman's default DNS forwarding address - 169.254.0.1 - appears in the
container's /etc/resolv.conf. That's not really related to anything else
going on in that test (which is about _changing_ that default address).
So, move it into its own test case.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
...or at least as much as possible. Some tests cannot
be run in parallel due to #23750: "--events-backend=file"
does not actually work the way a naïve user would intuit.
Stop/die events are asynchronous, and can be gathered
by *ANY OTHER* podman process running after it, and if
that process has the default events-backend=journal,
that's where the event will be logged. See #23987 for
further discussion.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Need --layers=false in podman build, otherwise a buildah race
can trigger "layer not known" failures:
https://github.com/containers/buildah/issues/5674
Signed-off-by: Ed Santiago <santiago@redhat.com>
When running parallel, multiple tests could be trying to start
the registry at once. Make this parallel-safe.
Also, use a safer port range for the registry. Something
outside of /proc/sys/net/ipv4/ip_local_port_range
Sorry, I'm including a FIXME section that I haven't investigated
deeply enough.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add a few best-practices examples, and add a whole section
describing the dos and donts of writing parallel-safe tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
For tests run in parallel, show file number as |nnn| (vs [nnn])
Teach logformatter to distinguish the two, adding 'p' to anchors
in parallel tests. Necessary because in this scheme we run bats
twice, thus see 'ok 1' twice, and we want to differentiate them.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Workaround for #23292, where simultaneous 'pod create' commands
will all start a podman-build of the pause image, but only
one of them will be tagged, and the others will leak <none>
images.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The issue is closed and I recently fixed a number of races (bf74797c69)
in the remote attach API that sound like exactly like the same error
that was mentioned in issue #9597.
As such I think this works, if it start flaking again we can revert this
or better fix the actual bug.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
As it turns on things are not so simple after all...
In podman-py it was reported[1] that waiting might hang, per our docs wait
on multiple conditions should exit once the first one is hit and not all
of them. However because the new wait logic never checked if the context
was cancelled the goroutine kept running until conmon exited and because
we used a waitgroup to wait for all of them to finish it blocked until
that happened.
First we can remove the waitgroup as we only need to wait for one of
them anyway via the channel. While this alone fixes the hang it would
still leak the other goroutine. As there is no way to cancel a goroutine
all the code must check for a cancelled context in the wait loop to no
leak.
Fixes 8a943311db ("libpod: simplify WaitForExit()")
[1] https://github.com/containers/podman-py/issues/425
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
We were only splitting on tabs, not spaces, so we returned just a
single line most of the time, not an array of the fields in the
output of `ps`. Unfortunately, some of these fields are allowed
to contain spaces themselves, which makes things complicated, but
we got lucky in that Docker took the simplest possible solution
and just assumed that only one field would contain spaces and it
would always be the last one, which is easy enough to duplicate
on our end.
Fixes#23981
Signed-off-by: Matt Heon <mheon@redhat.com>
For the past two months we've been splitting system tests
into two categories: those that CAN be run in parallel,
and those that CANNOT. Much work has been done to replace
hardcoded names (mycontainer, mypod) with safename().
Hundreds of test runs, in CI and on Ed's laptop, have
proven this approach viable.
make {local,remote}system now runs in two steps: first
the serial ones, then the parallel ones. hack/bats will
now recognize the 'ci:parallel' tag and add --jobs (nprocs).
This requires some tweaking of leak_check, because there
can be umpteen tests running (affecting image/container/pod/etc
state) when any given test completes.
Rules for enabling parallelization in tests:
* use unique container/pod/volume/network names (safename)
* do not run 'podman rm -a' or 'rmi -a'
* never use the -l (--latest) option
* do not run 'podman ps/images' and expect precise output
Signed-off-by: Ed Santiago <santiago@redhat.com>
...and remove one old skip() for older debian, but leave
two others in place and mark that they're still a problem.
Signed-off-by: Ed Santiago <santiago@redhat.com>
podman-remote events are not flushed, so order is not guaranteed.
This results in CI flakes. Only on Debian, for reasons unknown.
Make the network-connection events test more lenient when remote.
Closes: #23634 (but does not actually fix it)
Signed-off-by: Ed Santiago <santiago@redhat.com>
convert the owner UID and GID into the user namespace only when
":idmap" mount is used.
This changes the behaviour of :idmap with an empty volume. Now the
existing directory ownership is copied up as in the other case.
Closes: https://github.com/containers/podman/issues/23347
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Use safename. Add ci:parallel tags. Use a random port, not
hardcoded 9999. Do not remove pause image. And especially
do not "rm -a" anything.
Signed-off-by: Ed Santiago <santiago@redhat.com>
...because it requires 100% control and knowledge of the
state of all images, containers, and volumes.
Use safename anyway, just in case we ever have a leak from here.
I'm finding safename sooooooo helpful when reading journal.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add ci:parallel tags; move one non-parallel-safe test to
another networking-test file; and a few drive-by fixes
Signed-off-by: Ed Santiago <santiago@redhat.com>
Use safename for containers and pods. Add ci:parallel tags.
And reenable distro-integration tests that had been skipped
due to a container-selinux bug that is now fixed.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Minor bump. Fedora VMs now include ShellCheck, so we can
remove the 'dnf install' at CI run time.
Also, FWIW, Debian *vark are now at 1.12 (from 1.9)
VMs built in https://github.com/containers/automation_images/pull/385
Signed-off-by: Ed Santiago <santiago@redhat.com>
pasta added a new --map-guest-addr to option that maps a to the actual
host ip. This is exactly what we need for host.containers.internal
entry. So we now make use of this option by default but still have to
keep the exclude fallback because the option is very new and some
users/distros will not have it yet.
This also fixes an issue where the --dns-forward ip were not used when
using the bridge network mode, only useful when not using aardvark-dns
as this used the proper ips there already from the rootless netns
resolv.conf file.
Fixes#19213
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When you sort by repository a user most likely also want the tags to be
sorted as well. At the very least to get a stable output as the order
could be changed pull podman tag/pull even if they keep using the same
tag name.
Fixes#23803
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Creating networks in a different dir is not parallel safe when running
containers on them as the network configs may end up using the same
bridge names which then causes conflicts on the host.
Fixes#23876
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The kube generate command can now generate a yaml for
the Job kind and the kube play command can create a pod
and containers with podman when passed in a Job yaml.
Add relevant tests and docs for this.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Use safename, and add ci:parallel tags to all tests. (One
test was running "podman wait -l", which cannot work in
parallel. I choose to change it to "wait $cname", and
lose the -l testing)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Where possible, use safename and add ci:parallel tags.
One test runs "podman kill -a", which would be unwise to run
in parallel with other tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add 'ci:parallel' tags to a few easy places. And, two
small easily-reviewed safename or random-port additions.
These have been working fine in #23275. I want to stop
carrying them there so I can work on simplifying my PR.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Many dependencies started using go 1.22 which means we have to follow in
order to update.
Disable the now depracted exportloopref linter as it was replaced by
copyloopvar as go fixed the loop copy problem in 1.22[1]
Another new chnage in go 1.22 is the for loop syntax over ints, the
intrange linter chacks for this but there a lot of loops that have to be
converted so I didn't do it here and disable th elinter for now, th eold
syntax is still fine.
[1] https://go.dev/blog/loopvar-preview
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
- replace random_string with safename in container/network names
- add ci:parallel tags where possible.
- where not possible, add explanations
- fix a userns leak
Signed-off-by: Ed Santiago <santiago@redhat.com>
Workaround (NOT A FIX) for pasta issue #23482, wherein
podman logs includes a waitpid: ESRCH warning. Consensus
seems to be that this is a bug in socat.
Signed-off-by: Ed Santiago <santiago@redhat.com>
- fix a few missing safenames
- eliminate 'container rm -a'
- when running ps, do substring match, not exact
- where possible, add ci:parallel tags
- when not possible, explain
Also, fix a completely broken inspect test
Signed-off-by: Ed Santiago <santiago@redhat.com>
This started off as an attempt to make `podman stop` on a
container started with `--rm` actually remove the container,
instead of just cleaning it up and waiting for the cleanup
process to finish the removal.
In the process, I realized that `podman run --rmi` was rather
broken. It was only done as part of the Podman CLI, not the
cleanup process (meaning it only worked with attached containers)
and the way it was wired meant that I was fairly confident that
it wouldn't work if I did a `podman stop` on an attached
container run with `--rmi`. I rewired it to use the same
mechanism that `podman run --rm` uses, so it should be a lot more
durable now, and I also wired it into `podman inspect` so you can
tell that a container will remove its image.
Tests have been added for the changes to `podman run --rmi`. No
tests for `stop` on a `run --rm` container as that would be racy.
Fixes#22852
Fixes RHEL-39513
Signed-off-by: Matt Heon <mheon@redhat.com>
e.g.: if the pod name is systemd-awd, the name of its infra container will be systemd-awd-infra
Signed-off-by: Misaki Kasumi <misakikasumi@outlook.com>
As discussed in Aug 13 Cabal, we are almost at a point where
e2e tests are reliably passing on the first try. Let's try to
keep things that way, and not hide future flakes.
Closes: #17967
Signed-off-by: Ed Santiago <santiago@redhat.com>
It qemu cannot be compiled anyway so make sure we do not try to compile
parts where the typechecker complains about on windows.
Also all the e2e test files are only used on linux as well.
pkg/machine/wsl also reports some error but to many for me to fix them
now. One minor problem was fixed in pkg/machine/machine_windows.go.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
We need something newer than 4.14 anyway now for most Podman functions.
This is breaking liniting on windows as the function doesn't work there.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Now that we have propert !remote tags set everywhere we can just rely on
that and do not need to skip any dirs.
Also on linux do not lint three times, one remote run is enough.
We still have to skip the test dir for windows/macos though or we need
to add linux build tags there everywhere as well. This seems simpler.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
We have a lot of systemd and quadlet based tests in the system tests.
This test doesn't seem very useful and it seems to flake so just remove
it.
Fixes#23480
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
By default wait only waits for the exit of a container, there is really
no way to make it wait for the removal too when the container was
created with --rm. I though I found a clever way in 8a943311db but this
is not working race free. While it works most of the time any other
parallel process might call syncContainer() before the cleanup process
holds the lock until it removes it. As such the wait hack to only update
the state and not sync the exit file did not work so we can drop that.
However the test wants to wait for the removal to happen by the cleanup
process and we can already say --condition=removing to do this but this
will throw an error if the ctr was removed instead of counting this as
success so fix that as well.
Fixes#23640
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The usual, safename instead of hardcoded names or random_string.
And remove some rmi statements: we no longer clean up pause_image.
Been working great in #23275 all week.
Signed-off-by: Ed Santiago <santiago@redhat.com>
...by using a crude port lock-and-reserve mechanism. This is
a small cherrypick from code that has been working in #23275
over dozens of CI runs. Am separating out into a small PR
because it's stable, harmless to serial runs, and will
simplify the eventual review of #23275.
Closes: #23488
Signed-off-by: Ed Santiago <santiago@redhat.com>
Use safename instead of hardcoded object names. Requires moving
a test table down, into the function itself instead of global,
because the table needs to know object names.
Also: sneak in a workaround for dealing with quay flakes (in
image search). The local registry is allowing almost all tests
to pass even when quay is down, but this one test still needs
to hit quay.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Now that on-failure exits right away the test is racy as the
RestartCount is not at the value we expect as the container is still
restarting in the background. As such add a timer based approach.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The current code did several complicated state checks that simply do not
work properly on a fast restarting container. It uses a special case for
--restart=always but forgot to take care of --restart=on-failure which
always hang for 20s until it run into the timeout.
The old logic also used to call CheckConmonRunning() but synced the
state before which means it may check a new conmon every time and thus
misses exits.
To fix the new the code is much simpler. Check the conmon pid, if it is
no longer running then get then check exit file and get exit code.
This is related to #23473 but I am not sure if this fixes it because we
cannot reproduce.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When will I learn not to dismiss something as "easy"?
Anyhow, this doesn't actually change anything parallel-wise
but it does reduce a race condition seen on heavily-loaded
slow systems, wherein a container goes into unhealthy before
we want it to. This version isn't perfect; I don't think
there's an ideal fix for this.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Only one test can be parallelized. Do so, and add a comment
to the other one explaining why it can't be.
Also, add some missing error-message checks.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Very few changes needed, all of them simple.
It is impossible to parallelize this entire file, because "stop -a".
Add tags to tests that can be parallelized, and comments to those
that can't.
Signed-off-by: Ed Santiago <santiago@redhat.com>
When the cidfile does not exists and ignore is set the cli parser skips
the file without error and we call into the backend code without any
names at all. This should logically be a NOP but on remote it caused all
containers to be returned which caused podman stop to stop everything in
this case.
Fixes#23554
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Do not rely on an arbitrary delay in order to ensure the port was bound
in the container. Instead this approach checks if the port is bound in
the netns and only then starts the client. This speeds up the entire
test file by 50% but more importantly in parallel testing it solves
hangs as the timeout there was unreliable.
Fixes#23471
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
c/image now throws a warning when using encryption and zstd:chunked as
they do not work together[1]. As CI uses default configs from fedora it
means rawhide now defaults to zstd:chunked which trigger the warning
there. To work around that force zstd compression.
[1] https://github.com/containers/image/issues/2485
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
If we try to stop a contianer that is not running or paused we get an
ErrCtrStateInvalid or ErrCtrStopped error. As podman stop is idempotent
this is not a user visable error at all so we should also never log it
in the container state.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Previously there were two CI tasks that ended up both testing docker-py
compatibility. Remove the duplicate from the `localapiv2-python` make
target, and symlink the identical requirements file.
Signed-off-by: Chris Evich <cevich@redhat.com>
Previously these tests pulled some test images from quay, opening them up
to networking-flake induced failures. As has already been done for
other tests, update to utilize the locally running registry server.
Also: Add `test/python/**` into the apiv2 task conditions as referenced
by the `Makefile` `localapiv2-python` target.
Signed-off-by: Chris Evich <cevich@redhat.com>
Basically commit ada4e1a8c1 for e2e tests. The timeout does not kill the
process so if it is stucked it hangs forever. So make sure we kill it
via SIGABRT.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The podman container cleanup command is not really intended for human
use. Instead each conmon will spawn this command after the container
exit to make sure we can cleanup resources asynchronously. However this
command will always race against other foreground process such as podman
rm -fa. Therefore it is possible that the ctr was already removed and we
should not log errors in this case.
While these errors are normally not seen as the command is int he
background you can see it if you enable syslog logging and then they
just spam the log with useless errors so just ignore them.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Add support for the ServiceName key for all unit types
Extend the PodInfo struct into UnitInfo to consolidate all prepopulated data into a single map
Use the NodesInfo map instead of the resourceName
Update the UnitInfo in the convert function instead of returning it
No need to replace extension anymore just remove it
All e2e tests with dependencies on other Quadlet files moved to a separate section
Add the capability of overriding the service name in the test
Add e2e tests for the new functionality
Adjust integration tests
Update the MAN page
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
Two tests, both check port 80 on host, one wants it live,
the other wants it blocked. Prevent them from running
concurrently.
Fixes: #23470
Signed-off-by: Ed Santiago <santiago@redhat.com>
David Gibson