podman/test/system
Valentin Rothberg 79e21b5b16 kube play: sd-notify integration
Integrate sd-notify policies into `kube play`.  The policies can be
configured for all contianers via the `io.containers.sdnotify`
annotation or for indidivual containers via the
`io.containers.sdnotify/$name` annotation.

The `kube play` process will wait for all containers to be ready by
waiting for the individual `READY=1` messages which are received via
the `pkg/systemd/notifyproxy` proxy mechanism.

Also update the simple "container" sd-notify test as it did not fully
test the expected behavior which became obvious when adding the new
tests.

Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2022-08-10 21:12:39 +02:00
..
000-TEMPLATE system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
001-basic.bats podman-remote pull: fix duplicate progress outputs 2022-07-21 10:04:45 +02:00
005-info.bats Bump VMs, to Ubuntu 2204 with cgroups v1 2022-07-21 20:08:32 -06:00
010-images.bats podman rmi: improve error message for build containers 2022-08-04 16:42:02 +02:00
011-image.bats --authfile command line argument for image sign command. 2021-11-11 15:11:19 -05:00
015-help.bats fix "podman -h" help output 2022-06-16 16:59:47 +02:00
020-tag.bats
030-run.bats With --rm option remove container if podman run fails 2022-07-28 05:54:58 -04:00
035-logs.bats podman wait can take multiple conditions 2022-07-08 05:26:56 -04:00
037-runlabel.bats
040-ps.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
045-start.bats Output messages display rawInput 2022-08-02 18:28:37 +09:00
050-stop.bats Output messages display rawInput 2022-08-02 18:28:37 +09:00
055-rm.bats Add rm --filter option 2022-07-30 10:59:59 +09:00
060-mount.bats Fix spelling "read only" -> "read-only" 2022-07-02 08:37:43 +02:00
065-cp.bats pkg: switch to golang native error wrapping 2022-07-08 08:54:47 +02:00
070-build.bats Merge pull request #14600 from edsantiago/argh 2022-06-16 16:01:45 +00:00
075-exec.bats Cirrus: enable Fedora 36 aarch64 tasks on EC2 2022-07-27 15:27:52 -04:00
080-pause.bats Revamp Libpod state strings for Docker compat 2022-01-17 13:56:07 -05:00
090-events.bats Add podman events -f to be alias for --filter 2022-07-13 10:55:27 -04:00
110-history.bats Add CreatedSince & CreatedAt format fields to podman image history 2022-04-27 15:15:03 -04:00
120-load.bats buildah vendor treadmill 2022-07-18 10:49:33 -04:00
125-import.bats Add --time out for podman * rm -f commands 2021-10-04 07:07:56 -04:00
130-kill.bats podman wait: return 0 if container never ran 2022-07-11 13:06:40 +02:00
140-diff.bats
150-login.bats Cirrus: enable Fedora 36 aarch64 tasks on EC2 2022-07-27 15:27:52 -04:00
160-volumes.bats When removing objects specifying --force,podman should exit with 0 2022-07-26 16:00:42 -04:00
170-run-userns.bats Bump VMs, to Ubuntu 2204 with cgroups v1 2022-07-21 20:08:32 -06:00
180-blkio.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
190-run-ipcns.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
200-pod.bats skip pod resource limits test on aarch64 2022-08-03 08:29:59 -04:00
220-healthcheck.bats Cleanup: remove obsolete/misleading bug workaround 2022-02-07 06:58:19 -07:00
250-systemd.bats system tests: new system-service bats file 2022-07-18 13:06:09 -06:00
251-system-service.bats Bump VMs, to Ubuntu 2204 with cgroups v1 2022-07-21 20:08:32 -06:00
255-auto-update.bats test/system/255-auto-update.bats: add an SELinux comment 2022-08-04 13:07:58 +02:00
260-sdnotify.bats kube play: sd-notify integration 2022-08-10 21:12:39 +02:00
270-socket-activation.bats Revert "Switch all rootful to rootfull" 2022-04-25 09:14:04 -04:00
271-tcp-cors-server.bats Enable /debug/pprof API service endpoints 2021-10-08 13:57:20 -07:00
272-system-connection.bats Report correct RemoteURI 2022-05-04 12:11:32 -04:00
300-cli-parsing.bats provide better error on invalid flag 2022-02-21 19:12:12 +01:00
320-system-df.bats
330-corrupt-images.bats System tests: tighten 'is' operator 2021-09-30 13:32:51 -06:00
400-unprivileged-access.bats Bump VMs, to Ubuntu 2204 with cgroups v1 2022-07-21 20:08:32 -06:00
410-selinux.bats Cirrus: enable Fedora 36 aarch64 tasks on EC2 2022-07-27 15:27:52 -04:00
420-cgroups.bats test: use new helper 2021-09-30 13:59:46 +02:00
450-interactive.bats replace golint with revive linter 2022-04-22 15:12:33 +02:00
500-networking.bats When removing objects specifying --force,podman should exit with 0 2022-07-26 16:00:42 -04:00
520-checkpoint.bats Output messages display rawInput 2022-08-02 18:28:37 +09:00
600-completion.bats shell completion: fix problems with container path completion 2022-06-09 13:46:55 +02:00
700-play.bats add "podman kube down" command 2022-08-02 16:27:01 +01:00
750-trust.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
800-config.bats Propagate $CONTAINERS_CONF to conmon 2022-02-18 12:35:25 +11:00
900-ssh.bats podman ssh work, using new c/common interface 2022-08-09 14:00:58 -04:00
README.md Cirrus: Add netavark/aardvark system test task 2022-02-03 13:07:56 -05:00
TODO.md docs: update the podman logo 2022-08-07 09:11:53 +08:00
build-testimage system test image: bump to 20220615 2022-06-15 13:29:08 -06:00
helpers.bash kube play: sd-notify integration 2022-08-10 21:12:39 +02:00
helpers.systemd.bash libpod: improve check to create conmon cgroup 2022-06-15 15:43:12 +02:00
helpers.t system tests: new random_free_port helper 2021-09-08 11:25:42 -06:00

README.md

Quick overview of podman system tests. The idea is to use BATS, but with a framework for making it easy to add new tests and to debug failures.

Quick Start

Look at 030-run.bats for a simple but packed example. This introduces the basic set of helper functions:

  • setup (implicit) - resets container storage so there's one and only one (standard) image, and no running containers.

  • parse_table - you can define tables of inputs and expected results, then read those in a while loop. This makes it easy to add new tests. Because bash is not a programming language, the caller of parse_table sometimes needs to massage the returned values; 015-run.bats offers examples of how to deal with the more typical such issues.

  • run_podman - runs command defined in $PODMAN (default: 'podman' but could also be './bin/podman' or 'podman-remote'), with a timeout. Checks its exit status.

  • is - compare actual vs expected output. Emits a useful diagnostic on failure.

  • die - output a properly-formatted message to stderr, and fail test

  • skip_if_rootless - if rootless, skip this test with a helpful message.

  • skip_if_remote - like the above, but skip if testing podman-remote

  • random_string - returns a pseudorandom alphanumeric string

Test files are of the form NNN-name.bats where NNN is a three-digit number. Please preserve this convention, it simplifies viewing the directory and understanding test order. In particular, 00x tests should be reserved for a first-pass fail-fast subset of tests:

bats test/system/00*.bats || exit 1
bats test/system

...the goal being to provide quick feedback on catastrophic failures without having to wait for the entire test suite.

Running tests

To run the tests locally in your sandbox, you can use one of these methods:

  • make;PODMAN=./bin/podman bats ./test/system/070-build.bats # runs just the specified test
  • make;PODMAN=./bin/podman bats ./test/system # runs all
  • make;PODMAN=./bin/podman NETWORK_BACKEND=netavark bats ./test/system # Assert & enable netavark testing

To test as root:

  • $ PODMAN=./bin/podman sudo --preserve-env=PODMAN bats test/system

Analyzing test failures

The top priority for this scheme is to make it easy to diagnose what went wrong. To that end, podman_run always logs all invoked commands, their output and exit codes. In a normal run you will never see this, but BATS will display it on failure. The goal here is to give you everything you need to diagnose without having to rerun tests.

The is comparison function is designed to emit useful diagnostics, in particular, the actual and expected strings. Please do not use the horrible BATS standard of [ x = y ]; that's nearly useless for tracking down failures.

If the above are not enough to help you track down a failure:

Debugging tests

Some functions have dprint statements. To see the output of these, set PODMAN_TEST_DEBUG="funcname" where funcname is the name of the function or perhaps just a substring.

Requirements

The jq tool is needed for parsing JSON output.

Further Details

TBD. For now, look in helpers.bash; each helper function has (what are intended to be) helpful header comments. For even more examples, see and/or run helpers.t; that's a regression test and provides a thorough set of examples of how the helpers work.