As agreed in Planning meeting of 2024-03-20, Podman 5.x will
drop support for cgroups v1 and for runc. Make it so.
CI images built in https://github.com/containers/automation_images/pull/338
Signed-off-by: Ed Santiago <santiago@redhat.com>
This container did not react to sigterm thus we always waited 10s for it
to stop. Also do not wait 2s for the logs instead use a retry loop.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The test does a normal stop on a command that does not react to sigterm.
As I cannot fix the system stop logic use a command which does. This
safes us 10s as it no longer waits for the timeout.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Both tests take 10s longer than they need to because they run the sleep
command int he container which does not react to sigterm, as such podman
waits 10s before killing it with sigkill.
To fix it just stop them with podman rm -fa -t0 to avoid the wait and do
not use podman kube down as we cannot set a timeout there. podman kube
down is still covered in many other tests so this is not an issue.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
IMO it is not important to cover each case with each sdnotify policy, to
speed them up we run all the exit code cases only once just twice for
each policy while switching the sdnotify policy between each case. This
way we safe 50% of runs and should still have sufficient coverage.
Before it took around 24 seconds, with this it is around 12 seconds now.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
There is really no point in waiting 10s for the kill, let's use 2 this
should be good enough to observe the timing.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This test waits 15 seconds to send sigterm for no good reason, we can
just make the timeout shorter. Also make sure the podman command quit on
sigterm by looking for the output message.
While at it fix the tests to use $PODMAN_TMPDIR not /tmp and define the
yaml in the test instead of using the podman create && podman kube
generate && podman rm way to create the yaml as it is a bit slower as we
have to call three podman commands for it.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Merge two podman event tests into one to speed them up as they did
mostly the same anyway. This way we only have to do the setup/teardown
once and only run one container.
Second, add the --since option because reading the journal can be slow
if you have thousands of event entries. This is not so critical in CI as
we run on fresh systems but on local dev machines I have almost 100k
events in the journal so parsing all of them makes this test slow (like
30s), with this change I can get it under 1s.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
.build files allow to build an image via Quadlet. The keys from a .build
file are translated to arguments of a `podman build` command by Quadlet.
Minimal keys for .build files are `ImageTag=` and a context directory,
see `SetWorkingDirectory=`, or a `File=` pointing to a Containerfile.
After sorting .build files into the Quadlet dependency order, there
remains a possible dependency cycle issue between .volume and .build
files: A .volume can have `Image=some.build`, and a .build can have
`Volume=some.volume:/some/volume`.
We solve this dependency cycle by prefilling resourceNames with all
image names from .build files before converting all the unit files.
This results in an issue for the test suite though: For .volume's
depending on *.image or *.build, we need to copy these additional
dependencies to the test's quadletDir, otherwise the test will fail.
This is necessary, because `handleImageSource()` actually needs to know
the image name defined in the referenced *.{build,image} file. It cannot
fall back on the default names, as it is done for networks or volumes,
for example.
Signed-off-by: Johannes Maibaum <jmaibaum@gmail.com>
Defining a timer with a fixed interval is not a good idea as we first
have to wait until the timer triggers, while the interval was every two
seconds it means that we have to wait at least 2s for it to start.
However much worse it means it triggers the unit over and over, this
seems to cause some soft of race with the output check. I have seen
this test run 10-60s which does not make much sense.
Switching the timer to trgger once on start seem to make the test run
consistently in 7s locally for me so this is much better.
There still is the question if we really have to test this at all on
each upstream PR but I left it for now.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
It takes over 10 seconds for this test as it uses --wait 5 twice which
runs into the timeout. IMO this tests is just redundant as it is already
covered in the e2e tests much better. Thus remove it here.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The new c/image version is returning a slightly new error message[1] so
make tests use the new one.
[1] https://github.com/containers/image/pull/2408
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When an empty volume is mounted into a container, Docker will
chown that volume appropriately for use in the container. Podman
does this as well, but there are differences in the details. In
Podman, a chown is presently a one-and-done deal; in Docker, it
will continue so long as the volume remains empty. Mount into a
dozen containers, but never add content, the chown occurs every
time. The chown is also linked to copy-up; it will always occur
when a copy-up occurred, despite the volume now not being empty.
This PR changes our logic to (mostly) match Docker's.
For some reason, the chowning also stops if the volume is chowned
to root at any point. This feels like a Docker bug, but as they
say, bug for bug compatible.
In retrospect, using bools for NeedsChown and NeedsCopyUp was a
mistake. Docker isn't actually tracking this stuff; they're just
doing a copy-up and permissions change unconditionally as long as
the volume is empty. They also have the two linked as one
operation, seemingly, despite happening at very different times
during container init. Replicating that in our stateful system is
nontrivial, hence the need for the new CopiedUp field. Basically,
we never want to chown a volume with contents in it, except if
that data is a result of a copy-up that resulted from mounting
into the current container. Tracking who did the copy-up is the
easiest way to do this.
Fixes#22571
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Now WaitForExit returns the exit code as stored in the db instead of
returning an error when the container was removed.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
If a container unit starts on boot with a dependency on `default.target`
the image unit may start too soon, before network is ready. This cause
the unit to fail to pull the image.
- Add a dependency on `network-online.target` to make sure image pulls
don't fail.
See https://github.com/containers/podman/issues/21873
- Document the hardcoded dependency on `network-online.target` for images unit
and explain how it can be overriden if necessary.
- tests/e2e/quadlet: Add `assert-last-key-regex`
Required to test the `After=` override in [Unit] section
See https://github.com/containers/podman/pull/22057#issuecomment-2008959993
- quadlet/unitfile: add a prepenUnitLine method
Requirements on networks should be inserted at the top of the
section so the user can override them.
Signed-off-by: jbtrystram <jbtrystram@redhat.com>
The default_addr shell function in test/system/helpers.network is used to
get the host's default address, which is used in a number of pasta
networking tests. However, in certain circumstances it can incorrectly
pick a deprecated address as the primary address. Correct it to exclude
those.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
We have to exclude the ips in the rootless netns as they are not the
host. Now that fix only works if there are more than one ip one the
host available, if there is only one we do not set the entry at all
which I consider better as failing to resolve this name is a much better
error for users than connecting to a wrong ip. It also matches what
--network pasta already does.
The test is bit more compilcated as I would like, however it must deal
with both cases one ip, more than one so there is no way around it I
think.
Fixes#22653
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
They are not run in CI and to my knowledge are not used by anyone, we
have much more/better tests in test/e2e and test/system that should
cover everything done in these scripts so just delete them to not
confuse contributors.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
First, point users to hack/bats for running them locally. Second, remove
TODO.md as it doesn't contain any helpful information. Basically all the
missing tests there have been added so this does not serve any purpose
and is missleading.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
wait for the healthy status on the thread where the container lock is
held. Otherwise, if it is performed from a go routine, a different
thread is used (since the runtime.LockOSThread() call doesn't have any
effect), causing pthread_mutex_unlock() to fail with EPERM.
Closes: https://github.com/containers/podman/issues/22651
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The function that's handing us events will return an error after closing
the channel over which it's sending events, and its caller (in its own
goroutine) will then send that error over another channel.
The logic that started the goroutine is likely to notice that the events
channel is closed before noticing that the error channel has a result
for it to read, so any error that would have been communicated would be
lost.
When we finish reading events, check if the reader returned an error
before telling our caller that there was no error.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The v5 API made a breaking change for podman inspect, this means that
an old client could not longer parse the result from the new 5.X server.
The other way around new client and old server already worked.
As it turned out there were several users that run into this, one case
to hit this is using an old 4.X podman machine wich now pulls a newer
coreos with podman 5.0. But there are also other users running into it.
In order to keep the API working we now have a version check and return
the old v4 compatible payload so the old remote client can still work
against a newer server thus removing any major breaking change for an
old client.
Fixes#22657
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Under some circumstances podman might be executed with a different argv0
than the actual path to the podman binary. This breaks the reexec logic
as it tried to exec argv0 which failed.
This is visible when using podmansh as login shell which get's the
special -podmansh on argv0 to signal the shell it is a login shell.
To fix this we can simply use /proc/self/exe as command path which is
much more robust and the argv array is still passed correctly.
Fixes#22672
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Final followup to #22270. That PR added a temporary convention
allowing a new form of ExitWithError(), one with an exit code
and stderr substring. In order to allow bite-size progress,
the old no-args form was still allowed. This PR removes
support for no-args ExitWithError().
This PR also adds one piece of new functionality: passing ""
(empty string) as the stderr arg means "expect exit code
but fail if there's anything at all in stderr".
Signed-off-by: Ed Santiago <santiago@redhat.com>
Follow up to commit eaf60c7fe7, with the toolbox image removal it is
possible to run all tests from tmpfs.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles only one file, test/e2e/rmi_test.go , because
my changes are significant enough to merit individual review.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles all remaining test/e2e/r*_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles test/e2e/s*_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles a subset of test/e2e/run_xxx_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
The scenario for inducing this is as follows:
1. Start a container with a long stop timeout and a PID1 that
ignores SIGTERM
2. Use `podman stop` to stop that container
3. Simultaneously, in another terminal, kill -9 `pidof podman`
(the container is now in ContainerStateStopping)
4. Now kill that container's Conmon with SIGKILL.
5. No commands are able to move the container from Stopping to
Stopped now.
The cause is a logic bug in our exit-file handling logic. Conmon
being dead without an exit file causes no change to the state.
Add handling for this case that tries to clean up, including
stopping the container if it still seems to be running.
Fixes#19629
Signed-off-by: Matt Heon <mheon@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles all remaining test/e2e/p*_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
This never tested what it said it did, the command line was wrong so
`,ro=false` was taken as image causing a error. What this actually
should care about is that a glob is taken as is and not evaluated.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Followup to #22270 : wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
A small number of tests were broken, as in, not actually testing
what they claimed to be testing. I've done my best to fix those.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles test/e2e/play_kube_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles test/e2e/v*_test.go
Signed-off-by: Ed Santiago <santiago@redhat.com>
The image is way to big (over 800MB) that slows tests down as we always
have to pull this, the tests itself are also super slow due the
entrypoint logic that we don't care about. We should be testing for
features needed and not specific tools.
I think the current changes should have a similar coverage in terms of
podman features, it no longer tests toolbox but IMO this never was a
task for podman CI tests.
The main driver for this is to make the tests run entirely based on
tmpfs and this image is just to much[1].
[1] https://github.com/containers/podman/pull/22533
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Some programs have their configuration files relative to the user's
home. It would be convenient being able to mount these into the container, but
that requires expansion of `~` or `$HOME` in a label. This commit adds support
for that for the `runlabel` command.
Signed-off-by: Dan Čermák <dcermak@suse.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
This commit handles a subset of test/e2e/pod_xxxx_test.go
(I stopped before this grew too huge for review)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
Signed-off-by: Ed Santiago <santiago@redhat.com>
..to match the version in root dir, to get rid of the mismatch
warning on every ginkgo run.
I still don't understand why renovatebot isn't doing this.
(Also, touch a file under e2e, to force tests to run)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Because the test left the image mounted the cleanup failed to remove the
tmpdir as it contained an active mount point. Thus ensure we unmount the
image again to prevent this leak.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Using /tmp means this file will be leaked and no deleted, switch to
using the per test tempdir which is removed after the test.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
TMPDIR is typically /tmp which is typically(*) a tmpfs.
This PR ignores $TMPDIR when $CI is defined, forcing all
e2e tests to set up one central working directory in /var/tmp
instead.
Also, lots of cleanup.
(*) For many years, up to and still including the time of
this PR, /tmp on Fedora CI VMs is actually NOT tmpfs,
it is just / (root). This is nonstandard and undesirable.
Efforts are underway to remove this special case.
Signed-off-by: Ed Santiago <santiago@redhat.com>
It is not clear why rootless was forced to the cgroupfs manager when
systemd is the default. In any case it causes local test failures as
described in the issue[1]. Using systemd manager makes them pass as
expected, I don't know enough aout cgroups to know the difference and
why certain tests have bad asumptions but this fixes it.
[1] https://github.com/containers/podman/issues/22474
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When the source dir is already mounted noexec, nodev or nosuid then a
rootless user cannot mount the dir into the container without these
options for obvious reasons.
So in order to run the test we must ensure the dir is mounted with these
options first, if they are simply skip as the test will fail otherwise.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When you run locally with a higher oom_score_adj then the one used in
the test podman will print a warning and not set the oom lower then the
current value. Thus use 999 as value which should only cause problems
for users with oom_score_adj value of 1000 (max value) which seems
unlikely.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Effectively, this is an ability to take an image already pulled
to the system, and automatically mount it into one or more
containers defined in Kubernetes YAML accepted by `podman play`.
Requirements:
- The image must already exist in storage.
- The image must have at least 1 volume directive.
- The path given by the volume directive will be mounted from the
image into the container. For example, an image with a volume
at `/test/test_dir` will have `/test/test_dir` in the image
mounted to `/test/test_dir` in the container.
- Multiple images can be specified. If multiple images have a
volume at a specific path, the last image specified trumps.
- The images are always mounted read-only.
- Images to mount are defined in the annotation
"io.podman.annotations.kube.image.automount/$ctrname" as a
semicolon-separated list. They are mounted into a single
container in the pod, not the whole pod.
As we're using a nonstandard annotation, this is Podman only, any
Kubernetes install will just ignore this.
Underneath, this compiles down to an image volume
(`podman run --mount type=image,...`) with subpaths to specify
what bits we want to mount into the container.
Signed-off-by: Matt Heon <mheon@redhat.com>
Image volumes (the `--mount type=image,...` kind, not the
`podman volume create --driver image ...` kind - it's strange
that we have two) are needed for our automount scheme, but the
request is that we mount only specific subpaths from the image
into the container. To do that, we need image volume subpath
support. Not that difficult code-wise, mostly just plumbing.
Also, add support to the CLI; not strictly necessary, but it
doesn't hurt anything and will make testing easier.
Signed-off-by: Matt Heon <mheon@redhat.com>
Followup to [1]#22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
Just trying to shrink down #22346 to a manageable, reviewable size.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
More low-hanging fruit: small reviewable chunks
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #22270: wherever possible/practical, extend command
error checks to include explicit exit status codes and error strings.
Because #22346 is stalled, these are some trivial easy-to-review
changes that get us closer to the goal.
Signed-off-by: Ed Santiago <santiago@redhat.com>
When the field is set to false we should never log healthcheck events.
Fixes https://issues.redhat.com/browse/RHEL-18987
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The change in healthcheck_run_test.go, depends on the
containers/image change:
commit b6afa8ca7b324aca8fd5a7b5b206fc05c0c04874
Author: Mikhail Sokolov <msokolov@evolution.com>
Date: Fri Mar 15 13:37:44 2024 +0200
Add support for Docker HealthConfig.StartInterval (v25.0.0+)
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This is something Docker does, and we did not do until now. Most
difficult/annoying part was the REST API, where I did not really
want to modify the struct being sent, so I made the new restart
policy parameters query parameters instead.
Testing was also a bit annoying, because testing restart policy
always is.
Signed-off-by: Matt Heon <mheon@redhat.com>
The Docker endpoint here is kind of a nightmare - accepts a full
Resources block, including a large number of scary things like
devices. But it only documents (and seems to use) a small subset
of those. This implements support for that subset. We can always
extend things to implement more later if we have a need.
Signed-off-by: Matt Heon <mheon@redhat.com>
The logic here is more complex than I would like, largely due to
the behavior of `podman inspect` for running containers. When a
container is running, `podman inspect` will source as much as
possible from the OCI spec used to run that container, to grab
up-to-date information on things like devices. We don't want to
change this, it's definitely the right behavior, but it does make
updating a running container inconvenient: we have to rewrite the
OCI spec as part of the update to make sure that `podman inspect`
will read the correct resource limits.
Also, make update emit events. Docker does it, we should as well.
Signed-off-by: Matt Heon <mheon@redhat.com>
nixery registry has been down all day. Disable test.
Someone will need to fix this on the buildah end.
Signed-off-by: Ed Santiago <santiago@redhat.com>
First of all this removes the need for a network connection, second
renovate can update the version as it is tracked in go.mod.
However the real important part is that the binary downloads are
broken[1]. For some reason the swagger created with them does not
include all the type information for the examples. However when building
from source the same thing works fine.
[1] https://github.com/go-swagger/go-swagger/issues/2842
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Two system tests were relying on $SYSTEMD_IMAGE but were not
running _prefetch. This led to baffling flakes that wasted
my time. (Quay flakes, of course. New manifestation.)
Signed-off-by: Ed Santiago <santiago@redhat.com>
when listing images through the restful service, consumers want to know
if the image they are listing is a manifest or not because the libpod
endpoint returns both images and manifest lists.
in addition, we now add `arch` and `os` as fields in the libpod endpoint
for image listing as well.
Fixes: #22184Fixes: #22185
Signed-off-by: Brent Baude <bbaude@redhat.com>
...to match the version in root dir, to get rid of the mismatch
warning on every ginkgo run.
The last bump was done by renovatebot; I don't know why bot didn't
do it this time.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Not sure why this only triggers now but this code was broken for a
while. It is racy as reported on the issue but because it changes the
actual map part of the network backend it means it can also alter the
behavior of the network which is very bad.
Fixes#22330
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
...and an optional error-message string, to be checked
against stderr.
This is a starting point and baby-steps progress toward #18188.
There are 249 ExitWithError() checks in test/e2e. It will take
weeks to fix them all. This commit enables new functionality:
Expect(ExitWithError(125, "expected substring"))
...while also allowing the current empty-args form. Once
all 249 empty-args uses are modernized, the matcher code
will be cleaned up.
I expect it will take several months of light effort to get
all e2e tests transitioned to the new form. I am choosing to
do so in pieces, for (relative) ease of review. This PR:
1) makes the initial changes described above; and
2) updates a small subset of e2e _test.go files such that:
a) ExitWithError() is given an exit code and error string; and
b) Exit(Nonzero) is changed to ExitWithError(Nonzero, "string")
(when possible)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Docker shows exposed ports as just PORT/PROTO so match that behavior. It
is not clear to me why someone needs that information in ps as "expose"
doesn't effect anything networking related.
Fixes https://issues.redhat.com/browse/RHEL-32154
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
As of April 2024, it's no longer included in rawhide by default.
We could force-install it, but it's 2024 and it seems likely
that all systems on which Podman 5 will run will have kernels
that support native overlay.
I also added two debugging printfs to the 'podman info' test
that initially failed on an (unpublished) rawhide VM. Without
these printfs it was impossible to diagnose the failure.
Updating docs is left as a future exercise.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This was added ages ago in commit c65b3599cc, however in the meantime
both podman and conmon can support longer socket paths as they use a
workaround to open the path via /proc/self/fd, see openUnixSocket() in
libpod/oci_conmon_attach_linux.go
Thus this restriction is not needed anymore and we can drop a workaround
in the tests.
Fixes#22272
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When we remove with --force we do not return a error if the input does
not exists, however if we get more than on input we must try to remove
all and not just NOP out and not remove anything just because one arg
did not exists.
Also make the code simpler for commands that do have the --ignore option
and just make --force imply --ignore which reduces the ugly error
handling.
Fixes#21529
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
if the 'U' option is provided, do not chown the destination target to
the existing target in the image.
Closes: https://github.com/containers/podman/issues/22224
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
As of podman 5.0, slirp4netns is a soft dependency. It might
not be installed on a host (and, in gating tests, is not).
Deal with it.
Use podman itself, not 'which', to tell us if slirp4netns
is available. We don't want to duplicate podman's path-check
logic. Since this check is expensive, cache the result.
(Change the has_pasta check similarly)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Three infrequent flakes. Add debug code to help track
down if/when they happen again.
And, one of them, fix a logic bug that will save us 8-10s
on system tests runs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
there are no overlay mounts in the "podman run with --volume and U
flag" tests so no need to skip them.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
if the volume is mounted with "idmap", there should not be any mapping
using the user namespace mappings since this is done at runtime using
the "idmap" kernel feature.
Closes: https://github.com/containers/podman/issues/22228
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Emergency update to get pasta 03-26. Also gives us crun 1.14.4.
One unexplained difference: fc39 and rawhide now create:
/run/log/journal/SOMETHING/system.journal
...and the SOMETHING is o-rwx. This triggers journalctl to spit out a warning:
Hint: You are currently not seeing messages from the system.
Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
Pass -q to turn off this notice.
...which in turn causes ExitCleanly() to fail.
It is not clear who/what is creating this journal directory, or
why it allofasudden started just now. Workaround is to add -q
to journalctl in one test.
One more difference, another test now requires SYSLOG capability.
VM package info:
https://github.com/containers/automation_images/pull/342
Signed-off-by: Ed Santiago <santiago@redhat.com>
Useful to tell whether containers are being made with pasta or
slirp4netns by default. Info is bloated enough already that I
don't really have concerns about shoving more into it.
Fixes#22172
Signed-off-by: Matt Heon <mheon@redhat.com>
Fix following issues:
- create container API handler ignores Annotations from HostConfig
- inspect container API handler does not provide Annotations as
part of HostConfig
Signed-off-by: diplane <diplane3d@gmail.com>
New run_minikube() helper, modeled after run_podman(). Echoes
each command being run and its output. On failure, runs minikube logs.
Addresses (does not close) #21931 which is hitting us hard in CI.
Probably quay flakes, but it's impossible to tell without logs.
Also: bug fix: one "run podman" fixed to run_podman
Signed-off-by: Ed Santiago <santiago@redhat.com>
By default we just ignored any localhost reolvers, this is problematic
for anyone with more complicated dns setups, i.e. split dns with
systemd-reolved. To address this we now make use of the build in dns
proxy in pasta. As such we need to set the default nameserver ip now.
A second change is the option to exclude certain ips when generating the
host.containers.internal ip. With that we no longer set it to the same
ip as is used in the netns. The fix is not perfect as it could mean on a
system with a single ip we no longer add the entry, however given the
previous entry was incorrect anyway this seems like the better behavior.
Fixes#22044
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
In some environments, such as the one described in
https://github.com/containers/podman/issues/20927, the default route
is given as nexthop gateways. That is, it's a multipath routes with
multiple gateways.
That means that pasta(1), after commit 6c7623d07bbd ("netlink: Add
support to fetch default gateway from multipath routes"), can start
and use a default gateway from that route.
Just like in pasta(1), in these tests, the default route indicates
which upstream interface we should pick. If we ignore multipath
routes, IPv6 addresses and gateway addresses themselves won't be
available, so, while pasta is now able to configure the container,
IPv6 tests will expect to find no address and no gateway, hence fail
due to the mismatch.
Try to get routes, including gateway addresses and interface names,
from nexthop objects, in case the selection of a regular default
route yields no results.
Link: https://github.com/containers/podman/issues/20927Closes: #20927
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
The annotations should be maintained by CRI-O itself to decouple the
projects from a dependency perspective.
[NO NEW TESTS NEEDED]
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Most of them look like our usual "assume too much about run -d".
One of them is just an unexpected warning, a push retry. Remove
the ExitCleanly() from that test, just rely on Exit(0).
The other two have to do with podman logs, which we know can lag.
Add a short 1-second retry loop.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Two test flakes in the past week. Looks like the usual race
between "run -d" and "assume the container is ready". I don't
know if this will resolve them, but it's still a good idea.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Commit 03f6589f3 added basic support for pull-error event from libimage
but it contains several problems:
1. storing the error as error type prevents it from being unmarshalled,
thus change it to a string
2. the error was never propagated from the libimage event to the podman
event struct
3. the error message was not wired into the cli and API
This commit fixes these problems.
Fixes#21458
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
There are many code paths which only do logrus but still exit 0 so this
should catch more bugs. Unfortunately runc logs way to much random stuff
so we ignore this check for runc right now.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
when performing a system reset with containers that run somewhere where
a soft kill wont work (like sleep), containers will wait 10 seconds
before terminating with a sigkill. But for a forceful action like
system reset, we should outright set no timeout so containers stop
quickly and are not waiting on a timeout
Fixes#21874
Signed-off-by: Brent Baude <bbaude@redhat.com>
This vendors the latest c/common version, including making Pasta
the default rootless network provider. That broke a number of
tests, which have been fixed as part of this PR.
Also includes a change to network stats logic, which simplifies
the code a bit and makes it actually work with Pasta.
Signed-off-by: Matt Heon <mheon@redhat.com>
fix an assumption in the test, that the --compression-format is
unchanged from the original image.
Instead validate that all the required architectures are part of the
manifest.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a --artifact flag to `podman manifest add` which can be used to
create an artifact manifest for one or more files and attach it to a
manifest list. Corresponding --artifact-type, --artifact-config-type,
--artifact-config, --artifact-layer-type, --artifact-subject, and
--artifact-exclude-titles options can be used to fine-tune the fields in
the artifact manifest that don't refer to the files themselves.
Add a --index option to `podman manifest annotate` that will cause
values passed to the --annotation flag to be applied to the manifest
list as a whole instead of to an entry in the list.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Checking for the mountdir is not relevent, a recent c/storage change[1] no
longer deletes the mount point directory so the check will cause a false
positive. findmnt exits 1 when the given path is not a mountpoint so
let's use that to check.
[1] 3f2e81abb3
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Upgrade-from-v4.1 was temporarily disabled in #21618: we brought
in new CI VMs, and v4.1 setup no longer works. Nobody quite
understands why, and we've hit diminishing returns in the
investigation (#21863). The current thinking is that it
is related to netavark, and versions < 1.3 do not work
in the current f39 VMs. (No clue why, because f39 kernel
did not change in #21618).
Anyhow, reenable the pre-4.8 upgrade test, using v4.3.1
which has netavark-1.4.0 which seems to work fine. For now.
Also, some upgrade test cleanup:
- Skip, not fail, if initial setup fails. Makes for less noisy logs.
- Remove duplicate --pid=host
- Ports: 808x -> 909x, because 8081 is used by restraint on 1mt
Signed-off-by: Ed Santiago <santiago@redhat.com>
Like docker podman network inspect should output the information of
running container with their ip/mac address on this network.
However the output format is not docker compatible as this cannot
include all the info we have and the previous output was already not
compatible so this is not new.
New example output:
```
[
{
...
"containers": {
"7c0d295779cee4a6db7adc07a99e635909413a390eeab9f951edbc4aac406bf1": {
"name": "c2",
"interfaces": {
"eth0": {
"subnets": [
{
"ipnet": "10.89.0.4/24",
"gateway": "10.89.0.1"
},
{
"ipnet": "fda3:b4da:da1e:7e9d::4/64",
"gateway": "fda3:b4da:da1e:7e9d::1"
}
],
"mac_address": "1a:bd:ca:ea:4b:3a"
}
}
},
"b17c6651ae6d9cc7d5825968e01d6b1e67f44460bb0c140bcc32bd9d436ac11d": {
"name": "c1",
"interfaces": {
"eth0": {
"subnets": [
{
"ipnet": "10.89.0.3/24",
"gateway": "10.89.0.1"
},
{
"ipnet": "fda3:b4da:da1e:7e9d::3/64",
"gateway": "fda3:b4da:da1e:7e9d::1"
}
],
"mac_address": "f6:50:e6:22:d9:55"
}
}
}
}
}
]
```
Fixes#14126
Fixes https://issues.redhat.com/browse/RHEL-3153
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
I'm tired of this flake, it's hitting us ~once/day. Root cause
still unknown.
Workaround: add a READY file to the http server, and run 'curl'
until we get it. Tested in #17831 for the last two weeks, flake
has not been seen even once since then.
Closes: #21649
Signed-off-by: Ed Santiago <santiago@redhat.com>
For a unix socket we should not trim this at all. The problem exists for
ssh only so make sure we only do this when a ssh URL is given.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Too many tests use port 5000. Although there's a putative GetPortLock()
it seems to be unreliable, and we often get what appear to be collisions
between tests.
A proper solution would be to pseudorandomly allocate ports, verify
that they're not being reused, Sprintf() these everywhere that
needs them, and sprinkle some powdered cinnamon on top.
This is not that proper solution.
Fixes: #20655
Signed-off-by: Ed Santiago <santiago@redhat.com>
Currently if a user specifies a negative time to stop a container the
code ends up specifying the negative time to time.Duration which treats
it as 0. By settine the default to max.Unint32 we end up with a positive
number which indicates > 68 years which is probably close enough to
infinity for our use case.
Fixes: https://github.com/containers/podman/issues/21811
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When we want the original image to be gzip, explicitly ask for that
instead of assuming the containers.conf defaults do that.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
When we want the original image to be gzip, explicitly ask for that
instead of assuming the containers.conf defaults do that.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
ToHumanReadable() exists twice now, there is no reason for this just
call the function on the backend event type is fine as this still has to
be used there.
It also fixes a bug where the wrong event type was passed to the
template which did not match the docs and json output.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Add new event type in cmd/podman to better match the docker format.
Signed-off-by: AhmedGrati <ahmedgrati1999@gmail.com>
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Seeing constant e2e test failures today, in search tests.
Reason: tests are searching for "alpine", which is common,
and we're hitting pollution.
Solution: search for "testdigest_v2s<x>", an image in quay
under the libpod namespace. And, in other tests that rely
on docker.io, switch to quay.
Signed-off-by: Ed Santiago <santiago@redhat.com>
- use PODMAN_TMPDIR, not BATS_TMPDIR, for temp file
- in teardown, do not assume that SNAME_FILE will exist
(test could fail before that file gets created)
- remove "?" ("ignore exit status") from rmi & prune.
Probably holdovers from the days before -f. If
these commands fail even with -f, we need to know.
Signed-off-by: Ed Santiago <santiago@redhat.com>
There's currently no way to inspect failures of the
parallel-remove test (#21742). Add debugging ability.
Also, clean up nasty red warnings
Signed-off-by: Ed Santiago <santiago@redhat.com>
This removes a lot of boilerplate, but also ensures that every
stop test that is not directly testing podman stop or podman pod
stop uses `-t0` for quick, error-free stopping.
Signed-off-by: Matt Heon <mheon@redhat.com>
Continuing to see CI failures of the form "StopSignal SIGTERM
failed to stop container in 10 seconds". Work around those,
either by adding "-t0" to podman stop, or by using Expect(Exit(0))
instead of ExitCleanly().
Addresses, but does not close, #20196
Signed-off-by: Ed Santiago <santiago@redhat.com>
The reserved annotation io.podman.annotations.volumes-from is made public to let user define volumes-from to have one container mount volumes of other containers.
The annotation format is: io.podman.annotations.volumes-from/tgtCtr: "srcCtr1:mntOpts1;srcCtr2:mntOpts;..."
Fixes: containers#16819
Signed-off-by: Vikas Goel <vikas.goel@gmail.com>
if the target mount path already exists and the container uses a user
namespace, correctly map the target UID/GID to the host values before
attempting a chown.
Closes: https://github.com/containers/podman/issues/21608
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
And, runc-1.12 broke our seccomp e2e tests (runc now calls getcwd(),
which is the dummy syscall blocked for testing seccomp). Switch
to blocking link() instead.
Also, disable v4.1.0 upgrade tests. They're hanging, and I have
no idea why, and have wasted most of a day debugging.
Fixes: #21546
Signed-off-by: Ed Santiago <santiago@redhat.com>
Conmon writes the exit file and oom file (if container
was oom killed) to the persist directory. This directory
is retained across reboots as well.
Update podman to create a persist-dir/ctr-id for the exit
and oom files for each container to be written to. The oom
state of container is set after reading the files
from the persist-dir/ctr-id directory.
The exit code still continues to read the exit file from
the exits directory.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Align the behaviour of `podman kube play file.yaml` to Kubernetes' by forcing
an image pull when `imagePullPolicy` is omitted and the container image does
not specify a tag.
Signed-off-by: Maurizio Porrato <mporrato@redhat.com>
For some reason this starting to flake f38. I don't think the issue in
podman rather the test start nc -l in the background so it may not yet
have bound the port in the container when we try to connect.
To fix this simply add some retry logic to nc.
While at it also add pasta to this test and make it use
defer-assertion-failures to run all loop iterations before reporting the
errors.
Fixes#21561 (hopefully)
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Simply because it's been a while since the last testimage
build, and I want to confirm that our image build process
still works.
Added /home/podman/healthcheck. This saves us having to
podman-build on each healthcheck test. Removed now-
unneeded _build_health_check_image helper.
testimage: bump alpine 3.16.2 to 3.19.0
systemd-image: f38 to f39
- tzdata now requires dnf **install**, not reinstall
(this is exactly the sort of thing I was looking for)
PROBLEMS DISCOVERED:
- in e2e, fedoraMinimal is now == SYSTEMD_IMAGE. This
screws up some of the image-count tests (CACHE_IMAGES).
- "alter tarball" system test now barfs with tar < 1.35.
TODO: completely replace fedoraMinimal with SYSTEMD_IMAGE
in all tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Moving from Go module v4 to v5 prepares us for public releases.
Move done using gomove [1] as with the v3 and v4 moves.
[1] https://github.com/KSubedi/gomove
Signed-off-by: Matt Heon <mheon@redhat.com>
No longer bother testing any 2.x or 3.x. Only 4.1 and above.
Remove all CNI-related code. CNI is gone.
Add DatabaseBackend tests, confirming that we can handle
both boltdb and sqlite.
Require BATS >= 1.8.0, and use "run -0" to do exit-status checks.
Update docs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
When interface_name attribute in containers.conf file is set to "device", then set interface names inside containers same as the network_interface names of the respective network.
The change applies to macvlan and ipvlan networks only. The interface_name attribute value has no impact on any other types of networks.
If the interface name is set in the user request, then that takes precedence.
Fixes: #21313
Signed-off-by: Vikas Goel <vikas.goel@gmail.com>
This mirrors how the Docker API handles things, allowing us to be
more compatible with Docker and more verbose on the Libpod API.
Stats are given as per network interface in the container, but
still aggregated for `podman stats` and `podman pod stats`
display (so the CLI does not change, only the Libpod and Compat
APIs).
Signed-off-by: Matt Heon <mheon@redhat.com>
These should all work with the latest netavark. The ipvlan case needs a
subnet because it does not support DHCP.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Currently we deadlock in the slirp4netns setup code as we try to
configure an non exissting netns. The problem happens because we tear
down the netns in the userns case correctly since commit bbd6281ecc but
that introduces this slirp4netns problem. The code does a proper new
network setup later so we should only use the short cut when not in a
userns.
Fixes#21477
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Podman v5 will not support cgroups-v1. This commit will print a warning
if it detects a cgroups-v1 system. The warning can be hidden by setting
envvar `PODMAN_CGROUPSV1_WARNING`.
This warning is patched out for RHEL 9 builds as cgroups-v1 will still
be supported on RHEL 9 systems.
Resolves: https://issues.redhat.com/browse/RUN-1957
[NO NEW TESTS NEEDED]
Co-authored-by: Ed Santiago <santiago@redhat.com>
Co-authored-by: Sascha Grunert <sgrunert@redhat.com>
Co-authored-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@redhat.com>
Just like all the other inspect commands that accept multiple args we
should just make podman pod inspect output a json array.
This makes the code more consistent and removes the extra workaround
which was needed before to support this.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
CNI is deprecated and is build tagged out for 5.0. Don't test it in our CI.
This commit also disables upgrade tests for now - those need more work since the old version of Podman only uses CNI. Upgrade tests will be re-vamped in a later commit.
Signed-off-by: Ashley Cui <acui@redhat.com>
The pasta network mode has been added in podman v4.4 and this causes a
conflict with named networks that could also be called "pasta". To not
break anything we had special logic to prefer the named network over the
network mode. Now with 5.0 we can break this and remove this awkward
special handling from the code.
Containers created with 4.X that use a named network pasta will also
continue to work fine, this chnage will only effect the creation of new
containers with a named network pasta and instead always used the
network mode pasta. We now also block the creation of networks with the
name "pasta".
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The current field separator comma of the inspect annotation conflicts with the mount options of --volumes-from as the mount options itself can be comma separated.
Signed-off-by: Vikas Goel <vikas.goel@gmail.com>
From https://github.com/containers/automation_images/pull/325
Major change: netavark and aardvark are now included in prior-fedora,
so CNI can be fully eliminated from CI (#21410)
FIXME FIXME FIXME: skip two e2e tests, waiting for new netavark
Signed-off-by: Ed Santiago <santiago@redhat.com>
Fix the way we set skipTLSVerify on the client side
to ensure that the push stage in farm build takes into
account the configuration in the farm node's registries.conf
when the user hasn't set it on the client side.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Podman Desktop [1] is looking into improving the user experience which
requires to know the source of an image. Consider the user triggers an
image pull and Podman Desktop wants to figure out whether the image name
refers to a Red Hat registry, for instance, to prompt installing the RH
auth extension.
Since the input values of images may be a short name [2], Podman Desktop
has no means to figure out the (potential) source of the image. Hence,
add a new `/resolve` endpoint to allow external callers to figure out
the (potential) fully-qualified image name of a given value.
With the new endpoint, Podman Desktop can ask Podman directly to resolve
the image name and then make an informed decision whether to prompt the
user to perform certain tasks or not. This for sure can also be used
for any other registry (e.g., Quay, Docker Hub).
[1] https://github.com/containers/podman-desktop/issues/5771
[2] https://www.redhat.com/sysadmin/container-image-short-names
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
We now no longer write containers.conf, instead system connections and
farms are written to a new file called podman-connections.conf.
This is a major rework and I had to change a lot of things to get this
to compile again with my c/common changes.
It is a breaking change for users as connections/farms added before this
commit can now no longer be removed or modified directly. However because
the logic keeps reading from containers.conf the old connections can
still be used to connect to a remote host.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When inspecting a container that does not define any health check, the health field should return nil. This matches docker behavior.
Signed-off-by: Ashley Cui <acui@redhat.com>
SpecGen is our primary container creation abstraction, and is
used to connect our CLI to the Libpod container creation backend.
Because container creation has a million options (I exaggerate
only slightly), the struct is composed of several other structs,
many of which are quite large.
The core problem is that SpecGen is also an API type - it's used
in remote Podman. There, we have a client and a server, and we
want to respect the server's containers.conf. But how do we tell
what parts of SpecGen were set by the client explicitly, and what
parts were not? If we're not using nullable values, an explicit
empty string and a value never being set are identical - and we
can't tell if it's safe to grab a default from the server's
containers.conf.
Fortunately, we only really need to do this for booleans. An
empty string is sufficient to tell us that a string was unset
(even if the user explicitly gave us an empty string for an
option, filling in a default from the config file is acceptable).
This makes things a lot simpler. My initial attempt at this
changed everything, including strings, and it was far larger and
more painful.
Also, begin the first steps of removing all uses of
containers.conf defaults from client-side. Two are gone entirely,
the rest are marked as remove-when-possible.
[NO NEW TESTS NEEDED] This is just a refactor.
Signed-off-by: Matt Heon <mheon@redhat.com>
Given that we can have multiple image digests,
fix the inspect test to check whether the digest
given matches one of the digests of the image.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Fix the image filter parsing in the common libraries
to follow an AND logic for all filters passed in ensuring
compatibility with Docker behavior.
Also fix the filter parsing on the tunnel side so that we grab
all the filters given by the user and not only the last filter
in the list.
Add tests for the fixes.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
This is one of the breaking changes in Podman 5.0: removing the
ability to create new instances of the old Bolt database. This
does not remove support for the database entirely, as existing
Bolt databases will still be usable, but all new installs will
use SQLite after this point - if Bolt is forced by config, we'll
just error.
We don't have plans to outright remove the Bolt code. If that
were to happen, it'd be Podman 6.0 at least, and a significant
enough change it'd warrant a lot of discussion and planning. We
do intend to start winding down support of BoltDB, though, and
new features may be added only to SQLite from here on.
I have added an escape hatch via an undocumented environment
variable that allows us to continue testing BoltDB in CI (and, if
necessary, locally) but I don't want this to be used for any
purpose except continued testing of the old DB to ensure we don't
break it.
Signed-off-by: Matt Heon <mheon@redhat.com>
Some OCI runtimes (cf. [1]) may tolerate container images that don't
specify an entrypoint even if no entrypoint is given on the command
line. In those cases, it's annoying for the user to have to pass a ""
argument to podman.
If no entrypoint is given, make the behavior the same as if an empty ""
entrypoint was given.
[1] https://github.com/containers/crun-vm
Signed-off-by: Alberto Faria <afaria@redhat.com>
Currently, if the container creation failed with
either run or create and you've used --pod with new:
the pod would be created nonetheless. This change ensures
the pod just created is also cleaned up in case
of container creation failure
Fixes#21228
Signed-off-by: danishprakash <danish.prakash@suse.com>
- #15074 ("subtree_control" flake). The flake is NOT FIXED, I
saw it six months ago on my (non-aarch64) laptop. However,
it looks like the frequent-flake-on-aarch64 bug is resolved.
I've been testing in #17831 and have not seen it. So,
tentatively remove the skip and see what happens.
- Closes: #19407 (broken tar, "duplicates of file paths")
All Fedoras now have a fixed tar. Debian DOES NOT, but
we're handling that in our build-ci-vm code. I.e., the
Debian VM we're using has a working tar even though there's
currently a broken tar out in the wild.
Added distro-integration tag so we can catch future problems
like this in OpenQA.
- Closes: #19471 (brq / blkio / loopbackfs in rawhide)
Bug appears to be fixed in rawhide, at least in the VMs we're
using now.
Added distro-integration tag because this test obviously
relies on other system stuff.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Update farm build to directly push images to a registry
after all the builds are complete on all the nodes.
A manifest list is then created locally and pushed to
the registry as well.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Ed Santiago
openshift-merge-bot[bot]
Paul Holzinger
jkwiatko
Johannes Maibaum
Giuseppe Scrivano
renovate[bot]
Matthew Heon
jbtrystram
David Gibson
Daniel J Walsh
Nalin Dahyabhai
Shion Tanaka
Matt Heon
Andre Marianiello
Jonas Berlin
Dan Čermák
Evgeni Golov
Brent Baude
diplane
Stefano Brivio
Denys Knertser
Sascha Grunert
Florian Bezannier
Romain Geissler
Miloslav Trmač
AhmedGrati
Vikas Goel
Urvashi Mohnani
Maurizio Porrato
Ygal Blum
Lokesh Mandvekar
Ashley Cui
Valentin Rothberg
Alexander Larsson
Alberto Faria
danishprakash