Add support for simple rollbacks during `podman auto-update`. Rollbacks
are enabled by default. If a systemd unit cannot be restarted after an
update, the previous image will be retagged and the unit will be
restarted a second time.
Add system tests for rollbacks. Also fix a bug in the restart sequence;
we have to use the channel to actually know whether the restart was
successful or not.
NOTE: To make rollbacks really useful, users must run their containers
with `--sdnotify=container` such that the containers send the ready
message over the (mounted) socket. This way, restarting the systemd
units during auto update will block until the message has been received
(or a timeout kicked in).
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Require the network to be online in all (generated) systemd units to
make sure that containers and Podman run only after the network has been
fully configured.
Fixes: #10655
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We should not be exposing the store outside of Libpod. We want to
encapsulate it as an internal implementation detail - there's no
reason functions outside of Libpod should directly be
manipulating container storage. Convert the last use to invoke a
method on Libpod instead, and remove the function.
[NO TESTS NEEDED] as this is just a refactor.
Signed-off-by: Matthew Heon <mheon@redhat.com>
LISTEN_FDNAMES is optional, the docs for sd_listen_fds() says:
This information is read from the $LISTEN_FDNAMES variable, which
**may** contain a colon-separated list of names.
emphasis mine (indeed, the cited coreos code also suggests it is optional).
This actually results in bug, since the default
/contrib/systemd/system/podman.socket file doesn't set a
FileDescriptorName=. podman when run with this systemd configuration
*always* starts in unix socket mode since SocketActivated() will return
false because the name is missing.
The bug is a race with a very small window: between when podman does the
unlink() and when it re-binds the socket later in the code, requests made
during this time will fail since nothing is listening. There's another
small race when the service stops and systemd realizes it and starts
listening again.
However, small this window we managed to hit it :).
Let's fix this by ignoring LISTEN_FDNAMES. Since the code in
cmd/podman/system/service_abi.go:restService() ignores this value anyway
when setting up the socket activated stuff, there's no real loss here.
Signed-off-by: Tycho Andersen <tycho@tycho.pizza>
Change the type of units generated with --new from "forking" to
"notify". This brings Podman closer to systemd and opens up
Podman to a number of use cases (see #5572).
Units generated without --new remain with `type=forking`. I
experimented a bit with adding a `--sdnotify` flag to `podman start` but
it doesn't really work well since we're competing with the default
sdnotify mode set during container creation.
Fixes: #5572
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Commit 748826fc88 fixed a bug where slow mounting of the runroot was
causing issues when the units are started at boot. The fix was to add
the container's runroot to the required mounts; the graph root has been
added as well.
Hard-coding the run- and graphroot to the required mounts, however,
breaks the portability of units generated with --now. Those units are
intended to be running on any machine as, theoreticaly, any user.
Make the mounts portable by using the `%t` macro for the run root.
Since the graphroot's location varies across root and ordinary users,
drop it from the list of required mounts. The graphroot was not causing
issues.
Fixes: #10493
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The with --new generated systemd unit loses the environment variables
when the create command only contains the key without the value. Since
podman tries to lookup those values from the environment the unit can
fail.
This commits ensures that we will add the environment variables to the
unit file when this is the case. The container environment variables are
looked up in the container spec.
Fixes#10101
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
podman generate systemd --new inserts extra idfile arguments. The
generated unit can break when the user did provide their own idfile
arguments as they overwrite the arguments added by generate systemd.
This also happens when a user tries to generate the systemd unit on
a container already create with a --new unit. This should now
create a identical unit. The solution is to remove all user provided
idfile arguments.
This commit also ensures that we do not remove arguments that are part
off the containers entrypoint.
Fixes#9776
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
It is rare but possible that storage locations for the graphroot and the
runroot are not mounted at boot time, and therefore might race when
doing container operations. An example we've seen in the wild is that a
slow tmpfs mount for the runroot would suddenly mount over /run, causing
the container to lose all currently-running data, requiring a system
refresh to get it back.
This patch adds RequiresMountsFor= to the systemd.unit header to ensure
the paths for both the graphroot and runroot are mounted prior to
starting any generated unit files.
Signed-off-by: Robb Manes <rmanes@redhat.com>
Some packages used by the remote client imported the libpod package.
This is not wanted because it adds unnecessary bloat to the client and
also causes problems with platform specific code(linux only), see #9710.
The solution is to move the used functions/variables into extra packages
which do not import libpod.
This change shrinks the remote client size more than 6MB compared to the
current master.
[NO TESTS NEEDED]
I have no idea how to test this properly but with #9710 the cross
compile should fail.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
We missed bumping the go module, so let's do it now :)
* Automated go code with github.com/sirkon/go-imports-rename
* Manually via `vgrep podman/v2` the rest
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The unit generation accidentally escaped the %t in the pod id file path.
This is a regression caused by #9178. This was not caught by the tests
because the test itself was wrong. It used a full path instead of the
systemd variable %t like the actual code does.
Fixes#9373
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
In a systemd unit dollar and percent signs are used for variables. A backslash
is used for escape sequences. If any of these characters are used in the create
command we have to properly escape them so systemd does not try to interpret them.
Fixes#9176
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
If the container create command contains an argument with double
curly braces the golang template parsing can fail since it tries
to interpret the value as variable. To fix this change the default
delimiter for the internal template to `{{{{`.
Fixes#9034
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
First, use the pflag library to parse the flags. With this we can
handle all corner cases such as -td or --detach=false.
Second, preserve the root args with --new. They are used for all podman
commands in the unit file. (e.g. podman --root /tmp run alpine)
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
Systemd is now complaining or mentioning /var/run as a legacy directory.
It has been many years where /var/run is a symlink to /run on all
most distributions, make the change to the default.
Partial fix for https://github.com/containers/podman/issues/8369
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
`KillMode=none` has been deprecated in systemd and is now throwing big
warnings when being used. Users have reported the issues upstream
(see #8615) and on the mailing list.
This deprecation was mainly motivated by an abusive use of third-party
vendors causing all kinds of undesired side-effects. For instance, busy
mounts that delay reboot.
After talking to the systemd team, we came up with the following plan:
**Short term**: we can use TimeoutStopSec and remove KillMode=none which
will default to cgroup.
**Long term**: we want to change the type to sdnotify. The plumbing for
Podman is done but we need it for conmon. Once sdnotify is working, we
can get rid of the pidfile handling etc. and let Podman handle it.
Michal Seklatar came up with a nice idea that Podman increase the time
out on demand. That's a much cleaner way than hard-coding the time out
in the unit as suggest in the short-term solution.
This change is executing the short-term plan and sets a minimum timeout
of 60 seconds. User-specified timeouts are added to that.
Fixes: #8615
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Our users are missing certain warning messages that would
make debugging issues with Podman easier.
For example if you do a podman build with a Containerfile
that contains the SHELL directive, the Derective is silently
ignored.
If you run with the log-level warn you get a warning message explainging
what happened.
$ podman build --no-cache -f /tmp/Containerfile1 /tmp/
STEP 1: FROM ubi8
STEP 2: SHELL ["/bin/bash", "-c"]
STEP 3: COMMIT
--> 7a207be102a
7a207be102aa8993eceb32802e6ceb9d2603ceed9dee0fee341df63e6300882e
$ podman --log-level=warn build --no-cache -f /tmp/Containerfile1 /tmp/
STEP 1: FROM ubi8
STEP 2: SHELL ["/bin/bash", "-c"]
STEP 3: COMMIT
WARN[0000] SHELL is not supported for OCI image format, [/bin/bash -c] will be ignored. Must use `docker` format
--> 7bd96fd25b9
7bd96fd25b9f755d8a045e31187e406cf889dcf3799357ec906e90767613e95f
These messages will no longer be lost, when we default to WARNing level.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Allow automatic generation for shell completion scripts
with the internal cobra functions (requires v1.0.0+).
This should replace the handwritten completion scripts
and even adds support for fish. With this approach it is
less likley that completions and code are out of sync.
We can now create the scripts with
- podman completion bash
- podman completion zsh
- podman completion fish
To test the completion run:
source <(podman completion bash)
The same works for podman-remote and podman --remote and
it will complete your remote containers/images with
the correct endpoints values from --url/--connection.
The completion logic is written in go and provided by the
cobra library. The completion functions lives in
`cmd/podman/completion/completion.go`.
The unit test at cmd/podman/shell_completion_test.go checks
if each command and flag has an autocompletion function set.
This prevents that commands and flags have no shell completion set.
This commit does not replace the current autocompletion scripts.
Closes#6440
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
The systemd generator looks for certain flags in the containers' create
commands to determine which flags need to be added. In case of named
containers, the generator adds the `--replace` flag to prevent name
conflicts at container creation. Fix the generator to not only cover
the `--name foo` syntax but also the `--name=foo` one.
Fixes: #7157
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add support for generating systemd units
via the api and podman-remote.
Change the GenerateSystemdReport type to return the
units as map[string]string with the unit name as key.
Add `--format` flag to `podman generate systemd`
to allow the output to be formatted as json.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
Make sure that arguments with whitespace are properly quoted so they are
interpreted as one (and not multiple ones) by systemd.
Now `-e tz="america/new york"` will be generated as `-e "tz=america/new york"`.
The quotes are moving but the argument is still correct.
Fixes: #7285
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
With the advent of Podman 2.0.0 we crossed the magical barrier of go
modules. While we were able to continue importing all packages inside
of the project, the project could not be vendored anymore from the
outside.
Move the go module to new major version and change all imports to
`github.com/containers/libpod/v2`. The renaming of the imports
was done via `gomove` [1].
[1] https://github.com/KSubedi/gomove
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Allow manual restarts of container units that are part of a pod.
This allows for configuring these containers for auto updates.
Fixes: #6770
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When generating systemd unit for pods, we need to remove certain
pod-related flags from the containers' create commands. Make sure
to account for all the syntax including a single argument with key and
value being split by `=`.
Fixes: #6766
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add an `ExecStopPost` run even for units generated without `--new`.
Although it may seem redundant to run `container/pod stop` twice at
first glance, we really need the post run. If the main PID (i.e.,
conmon) is killed, systemd will not execute `ExecStop` but only the
post one. We made this obeservation in a customer issue and could
reproduce the behavior consistently. Hence, the post run is needed
to properly clean up when conmon is killed and it's pretty much a
NOP in all other cases.
Credits to Ulrich Obergfell for throrough and detailed analyses,
which ultimately lead to this fix.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Use `--replace` for named containers and pods. This will clean up
previous containers and podsthat may not have been removed after a
system crash.
Fixes: #5485
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Create a new template for generating a pod unit file. Eventually, this
allows for treating and extending pod and container generation
seprately.
The `--new` flag now also works on pods.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Refactor the systemd-unit generation code and move all the logic into
`pkg/systemd/generate`. The code was already hard to maintain but I
found it impossible to wire the `--new` logic for pods in all the chaos.
The code refactoring in this commit will make maintaining the code
easier and should make it easier to extend as well. Further changes and
refactorings may still be needed but they will easier.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Rename the container ID file from "cid" to "ctr-id" to make the
generated unit files a) easier to read and to b) pro-actively
avoid any confusion when pod ID files are being added in the
future.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
* Fix support for socket activation, on remote and service
$ systemd-socket-activate -l 8083 --fdname=podman bin/podman system service --log-level=debug --time=30
$ bin/podman-remote --remote=tcp://[::]:8083 image ls
Or, use the podman.{socket,service} unit files
$ bin/podman-remote --remote=unix:///run/podman/podman.sock image ls
Signed-off-by: Jhon Honce <jhonce@redhat.com>
We need to consistently use --time rather then --timeout throughout the code.
Fix locations where timeout defaults are not set correctly as well.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add support to auto-update containers running in systemd units as
generated with `podman generate systemd --new`.
`podman auto-update` looks up containers with a specified
"io.containers.autoupdate" label (i.e., the auto-update policy).
If the label is present and set to "image", Podman reaches out to the
corresponding registry to check if the image has been updated. We
consider an image to be updated if the digest in the local storage is
different than the one of the remote image. If an image must be
updated, Podman pulls it down and restarts the container. Note that the
restarting sequence relies on systemd.
At container-creation time, Podman looks up the "PODMAN_SYSTEMD_UNIT"
environment variables and stores it verbatim in the container's label.
This variable is now set by all systemd units generated by
`podman-generate-systemd` and is set to `%n` (i.e., the name of systemd
unit starting the container). This data is then being used in the
auto-update sequence to instruct systemd (via DBUS) to restart the unit
and hence to restart the container.
Note that this implementation of auto-updates relies on systemd and
requires a fully-qualified image reference to be used to create the
container. This enforcement is necessary to know which image to
actually check and pull. If we used an image ID, we would not know
which image to check/pull anymore.
Fixes: #3575
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Move the dbus-connection code from libpod's healthcheck to pkg/systemd
to allow for sharing the logic. Needed for the auto-updates work.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
flouthoc
Valentin Rothberg
Matthew Heon
Tycho Andersen
Paul Holzinger
Daniel J Walsh
Robb Manes
OpenShift Merge Robot
Jakub Guzik
Ed Santiago
Josh Soref
Jhon Honce
Yuan-Hao Chen