podman/test/system
Valentin Rothberg 7f8d5e5654 speed up image listing
As found in #17828, image listing does not scale well with a growing
number of local images.  Make use of recent improvements in libimage
that allow for computing the dangling and parent data with _one_ layer
tree.  Prior, the layer tree had to be recomputed _twice_ for each
image.

[NO NEW TESTS NEEDED] as it's a non-functional performance change.

Fixes: #17828
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2023-03-30 09:45:44 +02:00
..
000-TEMPLATE system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
001-basic.bats CI: test and confirm DESIRED_DATABASE 2023-03-16 15:32:56 -06:00
005-info.bats add CONTAINERS_CONF_OVERRIDE 2023-03-23 15:16:59 +01:00
010-images.bats Logs follow-until tests: loosen checks 2023-02-22 06:24:51 -07:00
011-image.bats
012-manifest.bats Add hidden podman manifest inspect -v option 2022-11-11 15:22:31 -05:00
015-help.bats Cleanup: fix problems reported by shell lint 2022-09-15 20:10:34 -06:00
020-tag.bats system tests: prevent leading tabs 2023-02-15 13:50:17 -07:00
030-run.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
032-sig-proxy.bats install sigproxy before start/attach 2023-02-13 15:13:42 +01:00
035-logs.bats fix "podman logs --since --follow" flake 2023-02-28 13:31:28 +01:00
037-runlabel.bats system tests: runlabel: use podman-under-test 2022-10-26 05:35:10 -06:00
040-ps.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
045-start.bats System tests: fix three races 2022-09-14 10:49:18 -06:00
050-stop.bats Output messages display rawInput 2022-08-02 18:28:37 +09:00
055-rm.bats system tests: prevent leading tabs 2023-02-15 13:50:17 -07:00
060-mount.bats Fix spelling "read only" -> "read-only" 2022-07-02 08:37:43 +02:00
065-cp.bats system tests: prevent leading tabs 2023-02-15 13:50:17 -07:00
070-build.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
075-exec.bats System tests: reenable some skipped aarch64 tests 2022-09-21 14:07:22 -06:00
080-pause.bats Revamp Libpod state strings for Docker compat 2022-01-17 13:56:07 -05:00
090-events.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
110-history.bats Add CreatedSince & CreatedAt format fields to podman image history 2022-04-27 15:15:03 -04:00
120-load.bats test/system: Move network-related helpers to helpers.network.bash 2022-11-08 00:16:36 +01:00
125-import.bats
130-kill.bats podman wait: return 0 if container never ran 2022-07-11 13:06:40 +02:00
140-diff.bats
150-login.bats Run make codespell 2023-03-21 16:00:54 -04:00
160-volumes.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
170-run-userns.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
180-blkio.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
190-run-ipcns.bats Correct output when inspecting containers created with --ipc 2023-01-24 12:18:39 -05:00
200-pod.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
220-healthcheck.bats fix --health-on-failure=restart in transient unit 2023-03-20 13:56:00 +01:00
250-systemd.bats cgroupns: private cgroupns on cgroupv1 breaks --systemd 2023-03-14 12:34:52 +01:00
251-system-service.bats system service --log-level=trace: support hijack 2023-03-14 14:33:52 +01:00
252-quadlet.bats Quadlet - treat paths starting with systemd specifiers as absolute 2023-03-28 10:30:33 +03:00
255-auto-update.bats test/system/255-auto-update.bats: add debug logs 2023-03-16 11:41:47 +01:00
260-sdnotify.bats system tests: fix racey sdnotify test 2023-03-27 10:47:49 -06:00
270-socket-activation.bats testimage: add iproute2 & socat, for pasta networking 2022-10-18 11:50:48 -06:00
271-tcp-cors-server.bats test/system: Move network-related helpers to helpers.network.bash 2022-11-08 00:16:36 +01:00
272-system-connection.bats test/system: fix wait_for_port() to wait for bind 2023-03-14 14:24:04 +01:00
280-update.bats System tests for podman-update 2022-09-01 15:42:35 -06:00
300-cli-parsing.bats provide better error on invalid flag 2022-02-21 19:12:12 +01:00
320-system-df.bats Build and use a newer systemd image 2023-01-02 13:26:46 -07:00
330-corrupt-images.bats speed up image listing 2023-03-30 09:45:44 +02:00
400-unprivileged-access.bats System tests: teardown: clean up volumes 2022-10-26 09:56:58 -06:00
410-selinux.bats Support running nested SELinux container separation 2023-03-13 14:21:12 -04:00
420-cgroups.bats System tests: fix three races 2022-09-14 10:49:18 -06:00
450-interactive.bats replace golint with revive linter 2022-04-22 15:12:33 +02:00
500-networking.bats fix slirp4netns resolv.conf ip with a userns 2023-03-28 15:52:33 +02:00
505-networking-pasta.bats Revert "Revert "pasta: Use two connections instead of three in TCP range forward tests"" 2023-03-16 05:11:30 +01:00
520-checkpoint.bats test: podman checkpoint/restore the latest container 2023-03-22 22:00:37 +08:00
600-completion.bats shell completion: fix problems with container path completion 2022-06-09 13:46:55 +02:00
610-format.bats Add podman system events alias to podman events 2022-11-11 12:57:49 -05:00
620-option-conflicts.bats New system tests for conflicting options 2022-12-21 06:23:29 -05:00
700-play.bats Add debug to --wait test 2023-03-28 10:01:02 -05:00
710-kube.bats Remove ReservedAnnotations from kube generate specification 2023-01-18 08:46:24 -05:00
750-trust.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
800-config.bats system tests: use CONTAINERS_CONF_OVERRIDE 2023-03-27 15:18:09 -06:00
900-ssh.bats ssh system test: skip until it becomes a test 2022-11-21 08:02:00 -07:00
950-preexec-hooks.bats rootless: rename auth-scripts to preexec-hooks 2023-01-17 10:58:46 +01:00
README.md Cirrus: Add netavark/aardvark system test task 2022-02-03 13:07:56 -05:00
TODO.md docs: update the podman logo 2022-08-07 09:11:53 +08:00
build-systemd-image e2e tests: stop using UBI images 2023-01-06 08:21:50 -07:00
build-testimage testimage: add iproute2 & socat, for pasta networking 2022-10-18 11:50:48 -06:00
helpers.bash cgroupns: private cgroupns on cgroupv1 breaks --systemd 2023-03-14 12:34:52 +01:00
helpers.network.bash test/system: fix wait_for_port() to wait for bind 2023-03-14 14:24:04 +01:00
helpers.systemd.bash System tests: quadlet network test 2023-01-03 17:25:58 +02:00
helpers.t System tests: assert(): friendlier failure messages 2023-02-16 17:47:10 -07:00

README.md

Quick overview of podman system tests. The idea is to use BATS, but with a framework for making it easy to add new tests and to debug failures.

Quick Start

Look at 030-run.bats for a simple but packed example. This introduces the basic set of helper functions:

  • setup (implicit) - resets container storage so there's one and only one (standard) image, and no running containers.

  • parse_table - you can define tables of inputs and expected results, then read those in a while loop. This makes it easy to add new tests. Because bash is not a programming language, the caller of parse_table sometimes needs to massage the returned values; 015-run.bats offers examples of how to deal with the more typical such issues.

  • run_podman - runs command defined in $PODMAN (default: 'podman' but could also be './bin/podman' or 'podman-remote'), with a timeout. Checks its exit status.

  • is - compare actual vs expected output. Emits a useful diagnostic on failure.

  • die - output a properly-formatted message to stderr, and fail test

  • skip_if_rootless - if rootless, skip this test with a helpful message.

  • skip_if_remote - like the above, but skip if testing podman-remote

  • random_string - returns a pseudorandom alphanumeric string

Test files are of the form NNN-name.bats where NNN is a three-digit number. Please preserve this convention, it simplifies viewing the directory and understanding test order. In particular, 00x tests should be reserved for a first-pass fail-fast subset of tests:

bats test/system/00*.bats || exit 1
bats test/system

...the goal being to provide quick feedback on catastrophic failures without having to wait for the entire test suite.

Running tests

To run the tests locally in your sandbox, you can use one of these methods:

  • make;PODMAN=./bin/podman bats ./test/system/070-build.bats # runs just the specified test
  • make;PODMAN=./bin/podman bats ./test/system # runs all
  • make;PODMAN=./bin/podman NETWORK_BACKEND=netavark bats ./test/system # Assert & enable netavark testing

To test as root:

  • $ PODMAN=./bin/podman sudo --preserve-env=PODMAN bats test/system

Analyzing test failures

The top priority for this scheme is to make it easy to diagnose what went wrong. To that end, podman_run always logs all invoked commands, their output and exit codes. In a normal run you will never see this, but BATS will display it on failure. The goal here is to give you everything you need to diagnose without having to rerun tests.

The is comparison function is designed to emit useful diagnostics, in particular, the actual and expected strings. Please do not use the horrible BATS standard of [ x = y ]; that's nearly useless for tracking down failures.

If the above are not enough to help you track down a failure:

Debugging tests

Some functions have dprint statements. To see the output of these, set PODMAN_TEST_DEBUG="funcname" where funcname is the name of the function or perhaps just a substring.

Requirements

The jq tool is needed for parsing JSON output.

Further Details

TBD. For now, look in helpers.bash; each helper function has (what are intended to be) helpful header comments. For even more examples, see and/or run helpers.t; that's a regression test and provides a thorough set of examples of how the helpers work.