The system test refactor[0] replaced the 'run_toolbox' helper function
with 'run toolbox', which is a normal invocation of Toolbox. This makes
it impossible to override Toolbox used during the tests using env var.
[0] https://github.com/containers/toolbox/pull/693
The bats-support[0] and bats-assert[1] libraries extend the
capabilities of bats[2]. Mainly, bats-assert is very useful for clean
checking of values/outputs/return codes.
Apart from updating the cases to use the libraries, the test cases have
been restructured in a way that they don't depend on each other anymore.
This required major changes in the helpers.bats file.
Overall, the tests are cleaner to read and easier to extend due to the
test cases being independent.
Some slight changes were made to the test cases themselves. Should not
alter their final behaviour.
There will be a follow up commit that will take care of downloading of
the tested images locally and caching them using Skopeo to speedup the
tests and try to resolve network problems when pulling the images that
we experienced in the past.
[0] https://github.com/bats-core/bats-support
[1] https://github.com/bats-core/bats-assert
[2] https://github.com/bats-core/bats-core
In the Go implementation, when the 'rm' and 'rmi' commands fail to
remove a container or image, they don't use a non-zero exit code.
There's currently no nice fix for this. So, the tests have been
adjusted as a temporary measure.
https://github.com/containers/toolbox/pull/507
The tests introduced by commit b5cdc57ae3 have proven to be
rather unstable due to mistakes in their design. The tests were quite
chaotically structured, and because of that images were deleted and
pulled too often, causing several false positives [1, 2].
This changes the structure of the tests in a major way. The tests
(resp. commands) are now run in a manner that better simulates the way
Toolbox is actually used. From a clean state, through creating
containers, using them and in the end deleting them. This should
reduce the strain on the bandwidth and possibly even speed up the
tests themselves.
[1] https://github.com/containers/toolbox/pull/372
[2] https://github.com/containers/toolbox/pull/374https://github.com/containers/toolbox/pull/375
Debarshi Ray
Ondřej Míchal
Juanje Ojeda