automation-tests/test/system
cdoern 6d3520e8b7 podman image scp remote support & podman image scp tagging
add support for podman-remote image scp as well as direct access via the API. This entailed
a full rework of the layering of image scp functions as well as the usual API plugging and type creation

also, implemented podman image scp tagging. which makes the syntax much more readable and allows users t tag the new image
they are loading to the local/remote machine:

allow users to pass a "new name" for the image they are transferring
`podman tag` as implemented creates a new image im `image list` when tagging, so this does the same
meaning that when transferring images with tags, podman on the remote machine/user will load two images
ex: `podman image scp computer1::alpine computer2::foobar` creates alpine:latest and localhost/foobar on the remote host

implementing tags means removal of the flexible syntax. In the currently released podman image scp, the user can either specify
`podman image scp source::img dest::` or `podman image scp dest:: source::img`. However, with tags this task becomes really hard to check
which is the image (src) and which is the new tag (dst). Removal of that streamlines the arg parsing process

Signed-off-by: Charlie Doern <cdoern@redhat.com>
2022-06-28 08:54:19 -04:00
..
000-TEMPLATE system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
001-basic.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
005-info.bats Add support for passing --volumepath 2022-04-23 08:03:36 -04:00
010-images.bats system test image: bump to 20220615 2022-06-15 13:29:08 -06:00
011-image.bats
015-help.bats fix "podman -h" help output 2022-06-16 16:59:47 +02:00
020-tag.bats
030-run.bats conmon: silence json-file error 2022-06-23 09:11:57 -04:00
035-logs.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
037-runlabel.bats
040-ps.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
045-start.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
050-stop.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
055-rm.bats
060-mount.bats podman image mount: print pretty table 2022-05-24 16:07:39 +02:00
065-cp.bats podman cp: do not overwrite non-dirs with dirs and vice versa 2022-06-10 09:42:19 +02:00
070-build.bats Merge pull request #14600 from edsantiago/argh 2022-06-16 16:01:45 +00:00
075-exec.bats Add support for --userns=nomap 2022-04-21 15:29:04 -04:00
080-pause.bats
090-events.bats Merge pull request #13971 from rhatdan/codespell 2022-04-22 06:34:32 -04:00
110-history.bats Add CreatedSince & CreatedAt format fields to podman image history 2022-04-27 15:15:03 -04:00
120-load.bats podman image scp remote support & podman image scp tagging 2022-06-28 08:54:19 -04:00
125-import.bats
130-kill.bats
140-diff.bats
150-login.bats Fix "podman run port forward range" flake 2022-06-27 14:25:24 +02:00
160-volumes.bats Use Regexp in volume ls --filter name 2022-06-27 21:34:39 +03:00
170-run-userns.bats --userns=keep-id,nomap are not allowed in rootful mode 2022-06-08 09:59:13 -04:00
180-blkio.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
190-run-ipcns.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
200-pod.bats pod: ps does not race with rm 2022-06-28 10:38:12 +02:00
220-healthcheck.bats Cleanup: remove obsolete/misleading bug workaround 2022-02-07 06:58:19 -07:00
250-systemd.bats system tests: avoid rmi -a ... plus cleanup 2022-06-15 10:47:42 -06:00
255-auto-update.bats auto update: create an event 2022-05-23 14:11:29 +02:00
260-sdnotify.bats k8systemd: run k8s workloads in systemd 2022-05-17 10:18:58 +02:00
270-socket-activation.bats Revert "Switch all rootful to rootfull" 2022-04-25 09:14:04 -04:00
271-tcp-cors-server.bats
272-system-connection.bats Report correct RemoteURI 2022-05-04 12:11:32 -04:00
300-cli-parsing.bats provide better error on invalid flag 2022-02-21 19:12:12 +01:00
320-system-df.bats
330-corrupt-images.bats
400-unprivileged-access.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
410-selinux.bats test/system/410-selinux: fix for newer runc 2022-06-15 20:26:31 -07:00
420-cgroups.bats
450-interactive.bats replace golint with revive linter 2022-04-22 15:12:33 +02:00
500-networking.bats Fix "podman run port forward range" flake 2022-06-27 14:25:24 +02:00
520-checkpoint.bats Pass '--file-locks' to OCI runtime at restoring 2022-06-09 09:49:53 -04:00
600-completion.bats shell completion: fix problems with container path completion 2022-06-09 13:46:55 +02:00
700-play.bats k8systemd: run k8s workloads in systemd 2022-05-17 10:18:58 +02:00
750-trust.bats system tests: add assert(), and start using it 2022-04-20 16:14:42 -06:00
800-config.bats Propagate $CONTAINERS_CONF to conmon 2022-02-18 12:35:25 +11:00
README.md Cirrus: Add netavark/aardvark system test task 2022-02-03 13:07:56 -05:00
TODO.md
build-testimage system test image: bump to 20220615 2022-06-15 13:29:08 -06:00
helpers.bash Fix "podman run port forward range" flake 2022-06-27 14:25:24 +02:00
helpers.systemd.bash libpod: improve check to create conmon cgroup 2022-06-15 15:43:12 +02:00
helpers.t

README.md

Quick overview of podman system tests. The idea is to use BATS, but with a framework for making it easy to add new tests and to debug failures.

Quick Start

Look at 030-run.bats for a simple but packed example. This introduces the basic set of helper functions:

  • setup (implicit) - resets container storage so there's one and only one (standard) image, and no running containers.

  • parse_table - you can define tables of inputs and expected results, then read those in a while loop. This makes it easy to add new tests. Because bash is not a programming language, the caller of parse_table sometimes needs to massage the returned values; 015-run.bats offers examples of how to deal with the more typical such issues.

  • run_podman - runs command defined in $PODMAN (default: 'podman' but could also be './bin/podman' or 'podman-remote'), with a timeout. Checks its exit status.

  • is - compare actual vs expected output. Emits a useful diagnostic on failure.

  • die - output a properly-formatted message to stderr, and fail test

  • skip_if_rootless - if rootless, skip this test with a helpful message.

  • skip_if_remote - like the above, but skip if testing podman-remote

  • random_string - returns a pseudorandom alphanumeric string

Test files are of the form NNN-name.bats where NNN is a three-digit number. Please preserve this convention, it simplifies viewing the directory and understanding test order. In particular, 00x tests should be reserved for a first-pass fail-fast subset of tests:

bats test/system/00*.bats || exit 1
bats test/system

...the goal being to provide quick feedback on catastrophic failures without having to wait for the entire test suite.

Running tests

To run the tests locally in your sandbox, you can use one of these methods:

  • make;PODMAN=./bin/podman bats ./test/system/070-build.bats # runs just the specified test
  • make;PODMAN=./bin/podman bats ./test/system # runs all
  • make;PODMAN=./bin/podman NETWORK_BACKEND=netavark bats ./test/system # Assert & enable netavark testing

To test as root:

  • $ PODMAN=./bin/podman sudo --preserve-env=PODMAN bats test/system

Analyzing test failures

The top priority for this scheme is to make it easy to diagnose what went wrong. To that end, podman_run always logs all invoked commands, their output and exit codes. In a normal run you will never see this, but BATS will display it on failure. The goal here is to give you everything you need to diagnose without having to rerun tests.

The is comparison function is designed to emit useful diagnostics, in particular, the actual and expected strings. Please do not use the horrible BATS standard of [ x = y ]; that's nearly useless for tracking down failures.

If the above are not enough to help you track down a failure:

Debugging tests

Some functions have dprint statements. To see the output of these, set PODMAN_TEST_DEBUG="funcname" where funcname is the name of the function or perhaps just a substring.

Requirements

The jq tool is needed for parsing JSON output.

Further Details

TBD. For now, look in helpers.bash; each helper function has (what are intended to be) helpful header comments. For even more examples, see and/or run helpers.t; that's a regression test and provides a thorough set of examples of how the helpers work.