On cgroup v1 we need to mount only the systemd named hierarchy as
writeable, so we configure the OCI runtime to mount /sys/fs/cgroup as
read-only and on top of that bind mount /sys/fs/cgroup/systemd.
But when we use a private cgroupns, we cannot do that since we don't
know the final cgroup path.
Also, do not override the mount if there is already one for
/sys/fs/cgroup/systemd.
Closes: https://github.com/containers/podman/issues/17727
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
It makes little sense to create a log line string from the entry just to
parse it again into a LogLine. We have the typed fields so we can
assemble the logLine direclty, this makes things simpler and more
efficient.
Also entries from the passthrough driver do not use the CONTAINER_ID_FULL
field, instead we can just access c.ID() directly.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The passthrough driver is designed for use in systemd units. By default
we can expect systemd to log the output on journald unless the unit sets
differen StandardOutput/StandardError settings.
At the moment podman logs just errors out when the passthrough driver is
used. With this change we will read the journald for the unit messages.
The logic is actually very similar to the existing one, we just need to
change the filter. We now filter by SYSTEMD_UNIT wich equals to the
contianer cgroup, this allows us the actually filter on a per contianer
basis even when multiple contianers are started in the same unit, i.e.
via podman-kube@.service.
The only difference a user will see is that journald will merge
stdout/err into one stream so we loose the separation there.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Only enforce the passthrough log driver for Quadlet. Commit 68fbebf
introduced a regression on the `podman-kube@` template as `podman logs`
stopped working and settings from containers.conf were ignored.
Fixes: #17482
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Replace existing tab indentations with spaces, and add
a test to CI to prevent new ones from sneaking in.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Commit 4fa307f149 fixed a number of issues in the sdnotify proxies.
Whenever a container runs with a custom sdnotify policy, the proxies
need to keep running which in turn required Podman to run and wait for
the service container to stop. Improve on that behavior and set the
service container as the main PID (instead of Podman) when no container
needs sdnotify.
Fixes: #17345
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Increase the loop range from 5 to 20 to make sure we give the service
enough time to transition to inactive. Other tests have the same range
with 0.5 seconds sleeps, so I expect the new value to be sufficient and
consistent.
Fixes: #17093
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Reasoning
---------
When the log-driver is passthrough, the journal socket is passed to the containers as-is which has two advantages:
1. journald can see who the actual sender of the log event is,
rather than thinking everything comes from the conmon process
2. conmon will not have to copy all the log data
Code Changes
------------
If log-driver was not set by the user and service-container is set use
passthrough as the default log-driver
Update the system tests
- explicitly set logdriver in sdnotify and play tests
- podman-kube template test: Verify the default log driver for service-container
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
As outlined in #16076, a subsequent BARRIER *may* follow the READY
message sent by a container. To correctly imitate the behavior of
systemd's NOTIFY_SOCKET, the notify proxies span up by `kube play` must
hence process messages for the entirety of the workload.
We know that the workload is done and that all containers and pods have
exited when the service container exits. Hence, all proxies are closed
at that time.
The above changes imply that Podman runs for the entirety of the
workload and will henceforth act as the MAINPID when running inside of
systemd. Prior to this change, the service container acted as the
MAINPID which is now not possible anymore; Podman would be killed
immediately on exit of the service container and could not clean up.
The kube template now correctly transitions to in-active instead of
failed in systemd.
Fixes: #16076Fixes: #16515
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Basically, in the timeout loop where we checked for new CID
on the restarted container, we were running 'podman inspect'
(not 'inspect --format ID'), and comparing full hundred-line
output against single-line CID string.
While I'm in here, add 'c_' prefix to container to make it
easier for my old eyes to recognize "oh, that's a container name"
vs "is that a name? a SHA? a woozle?"
Signed-off-by: Ed Santiago <santiago@redhat.com>
Make sure that the on-failure actions only kick in once the health check
has passed its retries. Also fix race conditions on reading/writing the
log.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Emit a warning to the user when generating a unit with --new on a
container that was created with a custom --restart policy. As shown
in #15284, a custom --restart policy in that case can lead to issues
on system shutdown where systemd attempts to nuke the unit but Podman
keeps on restarting the container.
Fixes: #15284
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
- basic : add actual log-level tests
- events : clean up, add --format tests
- systemd : reorder proxy args for legibility
- auto-update : fix missing timeout that could lead to hang
Signed-off-by: Ed Santiago <santiago@redhat.com>
For systems that have extreme robustness requirements (edge devices,
particularly those in difficult to access environments), it is important
that applications continue running in all circumstances. When the
application fails, Podman must restart it automatically to provide this
robustness. Otherwise, these devices may require customer IT to
physically gain access to restart, which can be prohibitively difficult.
Add a new `--on-failure` flag that supports four actions:
- **none**: Take no action.
- **kill**: Kill the container.
- **restart**: Restart the container. Do not combine the `restart`
action with the `--restart` flag. When running inside of
a systemd unit, consider using the `kill` or `stop`
action instead to make use of systemd's restart policy.
- **stop**: Stop the container.
To remain backwards compatible, **none** is the default action.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Add auto-update support to `podman kube play`. Auto-update policies can
be configured for:
* the entire pod via the `io.containers.autoupdate` annotation
* a specific container via the `io.containers.autoupdate/$name` annotation
To make use of rollbacks, the `io.containers.sdnotify` policy should be
set to `container` such that the workload running _inside_ the container
can send the READY message via the NOTIFY_SOCKET once ready. For
further details on auto updates and rollbacks, please refer to the
specific article [1].
Since auto updates and rollbacks bases on Podman's systemd integration,
the k8s YAML must be executed in the `podman-kube@` systemd template.
For further details on how to run k8s YAML in systemd via Podman, please
refer to the specific article [2].
An examplary k8s YAML may look as follows:
```YAML
apiVersion: v1
kind: Pod
metadata:
annotations:
io.containers.autoupdate: "local"
io.containers.autoupdate/b: "registry"
labels:
app: test
name: test_pod
spec:
containers:
- command:
- top
image: alpine
name: a
- command:
- top
image: alpine
name: b
```
[1] https://www.redhat.com/sysadmin/podman-auto-updates-rollbacks
[2] https://www.redhat.com/sysadmin/kubernetes-workloads-podman-systemd
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
See https://github.com/containers/conmon/pull/352
As of a few days ago, Ubuntu still hadn't built a fixed conmon.
Just skip the test until we get a fixed Ubuntu or until we
figure out a better solution to the test-something-RHEL8ish
problem.
UPDATE: WEIRD: this 'skip' triggered a baffling failure
on Ubuntu: the "Kubernetes only allows 63 characters"
warning message stopped appearing, on Ubuntu only, which
then caused the kube-generate tests to fail because they
actually checked for that. The message doesn't appear
because generate-kube is no longer spitting out a line
for org.opencontainers.image.base.digest/CONTAINER.
(Why this line is gone, I don't know, and choose not
to investigate). Solution: stop checking for the kube-63
warning. It's just not that important.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #14957, which added a new test that doesn't
actually belong in the 250-systemd.bats file. It was
copy-pasted from another test that doesn't belong there.
Move both tests to a new .bats file, because (1) they
need a custom cleanup, and (2) one of the tests should
very definitely run under podman-remote, and the 250
bats file has a global skip_if_remote().
Signed-off-by: Ed Santiago <santiago@redhat.com>
Docker supports -H and --host for specify the listening socket. Podman
should support them also in order to match the CLI.
These will not be documented since Podman defaults to using the
--url option.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
With the upcoming plans of introducing a podman-kube command with
various subcommands, rename the podman-play-kube systemd template
to podman-kube before releasing it.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
I noticed 'rmi -a' in a test. I tried to fix it. Hilarity ensued.
'rmi -a' is evil: it forces a fresh pull of our test image,
which in turn almost guarantees a flake some day. We avoid
it, but once in a while it slips in.
While fixing it, I noticed a bevy of other problems that
needed cleanup.
Signed-off-by: Ed Santiago <santiago@redhat.com>
commit 1951ff168a introduced a check so
that conmon is not moved to a new cgroup when podman is running inside
of a systemd service. This is helpful to integrate podman in systemd
so that the spawned conmon lives in the same cgroup as the service
that created it.
Unfortunately this breaks when podman daemon is running in a systemd
service since the same check is in place thus all the conmon processes
end up in the same cgroup as the podman daemon. When the podman
daemon systemd service stops the conmon processes are also terminated
as well as the containers they monitor.
Improve the check to exclude podman running as a daemon.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2052697
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
`podman auto-update` is now properly exercised in the system tests, so
we can safely remove the outdated TODO.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Support running `podman play kube` in systemd by exploiting the
previously added "service containers". During `play kube`, a service
container is started before all the pods and containers, and is stopped
last. The service container communicates its conmon PID via sdnotify.
Add a new systemd template to dispatch such k8s workloads. The argument
of the template is the path to the k8s file. Note that the path must be
escaped for systemd not to bark:
Let's assume we have a `top.yaml` file in the home directory:
```
$ escaped=$(systemd-escape ~/top.yaml)
$ systemctl --user start podman-play-kube@$escaped.service
```
Closes: https://issues.redhat.com/browse/RUN-1287
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Problem: the system test 'is()' checker was poorly thought out.
For example, there is no way to check for inequality or for
absence of a substring.
Solution, step 1: introduce new assert(), copied almost verbatim
from buildah, where it has been successful in addressing the
gaps in is().
The logical next step is to search the tests for 'die' and
for 'run', looking for negative assertions which we can
replace with assert(). There were a lot, and in the process
I found a number of ugly bugs in the tests themselves. I've
taken the liberty of fixing these.
Important note: at this time we have both assert() and is().
Replacing all instances of is() would be impossible to review.
Signed-off-by: Ed Santiago <santiago@redhat.com>
systemd expects the container_uuid environment variable be set
when it is running in a container.
Fixes: https://github.com/containers/podman/issues/13187
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When running podman inside systemd user units, it is possible that
systemd kills the rootless netns slirp4netns process because it was
started in the default unit cgroup. When the unit is stopped all
processes in that cgroup are killed. Since the slirp4netns process is
run once for all containers it should not be killed. To make sure
systemd will not kill the process we move it to the user.slice.
Fixes#13153
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
move the check after the cgroup manager is set, so to correctly detect
--cgroup-manager=cgroupfs and do not raise a warning about dbus not
being present.
Closes: https://github.com/containers/podman/issues/12802
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Replace `multi-user.target` with `default.target` across the code base.
It seems like the multi-user one is not available for (rootless) users
on F35 anymore is causing issues in all kinds of ways, for instance,
enabling the podman.service or generated systemd units.
Fixes: #12438
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add a new flag to set the start timeout for a generated systemd unit.
To make naming consistent, add a new --stop-timeout flag as well and let
the previous --time map to it.
Fixes: #11618
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Podman adds a few environment variables by default, and
currently there is no way to get rid of them from your container.
This option will allow you to specify which defaults you don't
want.
--unsetenv-all will remove all default environment variables.
Default environment variables can come from podman builtin,
containers.conf or from the container image.
Fixes: https://github.com/containers/podman/issues/11836
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add --time flag to podman container rm
Add --time flag to podman pod rm
Add --time flag to podman volume rm
Add --time flag to podman network rm
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Handle custom restart policies of containers when generating the unit
files; those should be set on the unit level and removed from ExecStart
flags.
Fixes: #11438
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure that Podman passes the LISTEN_* environment into containers.
Similar to runc, LISTEN_PID is set to 1.
Also remove conditionally passing the LISTEN_FDS as extra files.
The condition was wrong (inverted) and introduced to fix#3572 which
related to running under varlink which has been dropped entirely
with Podman 3.0. Note that the NOTIFY_SOCKET and LISTEN_* variables
are cleared when running `system service`.
Fixes: #10443
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Commit 9ac5267 changed the type of the generated systemd units from
`forking` to `notify`. It further stopped using `--cidfile` and instead
intended systemd to take care of stopping the container, which turned
out to be a bad idea.
Systemd will send the stop/kill signals to conmon which in turn may exit
non-zero, depending on the signal, and ultimately breaking container
cleanup.
Hence, we need to use --cidfile again and let podman stop and remove the
container to make sure that everything's in order.
Fixes: #11304
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This reverts commit 70801b3d71.
It turns out that letting systemd handle stopping the container is not
working as I thought it will. Conmon is receiving the stop/kill signals
and may exit non-zero, which in turn lets the systemd service transition
into the `failed` state.
We need to get back to letting Podman stop the containers and do a
partial revert of commit 9ac5267 which removed using --cidfile.
Happening in a following commit.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Commit 9ac5267598 changed the type of the generated systemd units from
forking to notify. Parts of these changes was also removing the need to
pass any information via the file system (e.g., PIDFILE, container ID).
That in turn implies that systemd takes care of stopping the container.
By default, systemd first sends a SIGTERM and after a certain timeout,
it'll send a SIGKILL. That's pretty much what Podman is doing, unless
the container was created with a custom stop signal which is the case
when the --stop-signal flag was used or systemd is mounted.
Account for that by using systemd's KillSignal option which allows for
changing SIGTERM to another signal. Also make sure that we're using the
correct timeout for units generated with --new.
Fixes: #11304
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
First and foremost: use ephemeral (/run, $XDG) directories
for systemd unit files, so as not to vandalize a working system.
Second, refactor common systemd-related functionality into
a new helper file, loaded by the systemd-related tests.
Shared functionality includes:
* setting $XDG_RUNTIME_DIR if unset and rootless
* setting $UNIT_DIR for use by tests
* new systemctl() and journalctl() functions, which
include "--user" when rootless (why can't systemd
figure this out on its own?)
Signed-off-by: Ed Santiago <santiago@redhat.com>
The rather raw and scarce output of `podman auto-update` has been a
thorn in my eyes for a longer while. So far, Podman would only print
updated systemd units, one per line, without further formatting.
Motivated by issue #9949 which is asking for some more useful
information in combination with a dry-run feature, I sat down and
reflected which information may come in handy.
Running `podman auto-update` will now look as follows:
```
$ podman auto-update
Trying to pull [...]
UNIT CONTAINER IMAGE POLICY UPDATED
container-test.service 08fd34e533fd (test) localhost:5000/busybox registry false
```
Also refactor the spaghetti code in the backend a bit to make it easier
to digest and maintain.
For easier testing and for the sake of consistency with other commands
listing output, add a `--format` flag.
The man page will get an overhaul in a follow up commit.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The with --new generated systemd unit loses the environment variables
when the create command only contains the key without the value. Since
podman tries to lookup those values from the environment the unit can
fail.
This commits ensures that we will add the environment variables to the
unit file when this is the case. The container environment variables are
looked up in the container spec.
Fixes#10101
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
changed struct to policyMapper
change "image" to "registry" in multiple locations
Updated documentation with registry alias & autoupdate local
Added relevant test
Signed-off-by: Parker Van Roy <pvanroy@redhat.com>
Detached containers and detach keys are only created with the podman run, i
exec, and start commands. We do not store the detach key sequence or the
detach flags in the database, nor does Docker. The current code was ignoreing
these fields but documenting that they can be used.
Fix podman create man page and --help output to no longer indicate that
--detach and --detach-keys works.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Record the user-specified "raw" image name in the SpecGenerator, so we
can pass it along to the config when creating a container. We need a
separate field as the image name in the generator may be set to the
ID of the previously pulled image - ultimately the cause of #7404.
Reverting the image name from the ID to the user input would not work
since "alpine" for pulling iterates over the search registries in the
registries.conf but looking up "alpine" normalizes to
"localhost/alpine".
Recording the raw-image name directly in the generator was the best of
the options I considered as no hidden magic from search registries or
normalizations (that may or may not change in the future) can interfere.
The auto-update backend enforces that the raw-image name is a
fully-qualified reference, so we need to worry about that in the front
end.
Fixes: #7407
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
info, images, run, networking tests: remove some skip_if_remote()s
that were added in the varlink days. All of these tests now seem
to work with APIv2.
help test: check that first output line from 'podman --help'
is the program description (regression check for #7273).
load test: clean up stray images, rewrite test to make it conform
to existing convention. In the process, discover and file #7337
exec test (and networking): file #7360, and add FIXME comment
to skip()s suggesting evaluating those tests once that is fixed.
pod test: now that #6328 is fixed, use 'podman pod inspect --format'
instead of relying on jq
Various other tests: add an explanation of why test is disabled
so we can more easily distinguish "this will never be meaningful
under remote" vs "hey, doesn't work for now, but maybe someday".
Signed-off-by: Ed Santiago <santiago@redhat.com>
Run `podman auto-update` in the systemd system tests. Note that this is
a first step to at least exercise parts of `auto-update` in the CI. The
service won't get updated just yet as we need to set up a local
registry, and push a new image. I do not have enough time at the moment
to do that but consider this change already as an improvement.
We are experiencing some issues in #6793 w.r.t. to auto-updates but
couldn't track down the root cause yet.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Giuseppe Scrivano
Paul Holzinger
Valentin Rothberg
Ed Santiago
Ygal Blum
Toshiki Sonoda
Daniel J Walsh
Valentin Rothberg
Boaz Shuster
Paul Holzinger
Parker Van Roy