cmd/podman needs that functionality as well, so I figured it makes sense
to break it into a separate package to reduce code clones.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Support pruning images in use by external containers (e.g., build
containers). Requires extending the containers filter, adding a
callback to libpod and a new options for image removal.
Tests will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Following flag allows user to configure RemoveImages in a manner so that
remove func becomes exclusive to removing manifests when needed and not
removing underlying referenced images.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
If matching images resolves to a manifest list, return manifest list
instead of resolving to image instance, if manifest list is not found
try resolving image.
Signed-off-by: Aditya Rajan <flouthoc.git@gmail.com>
Fix a bug where not all removed images were actually reported as such.
A regression test will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When it was first introduced, the blob info cache's location didn't
change from the system-wide default location when we were running in
rootless mode, so we started setting its location ourselves to avoid
triggering permissions errors when updating it.
The image library has since started taking into account that it was
running in rootless mode, but its hardwired default isn't the same as
the one we were setting, so we ended up creating a second cache file.
Stop doing that.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When writing LookupImage, I thought that it's a good idea to always
attempt to match an image against the local (or requested) platform.
The use case I had in mind is multi-arch support:
`$ podman run image` should only match `image` if it matches the local
platform. We may have previously pulled `image` for another
architecture.
The core criteria for these checks is that images set their platform
(arch/os/variant) correctly. As it turned out that is not the case.
We recently performed a number of fixes to better support multi-arch
images and this change should put the last nail in the coffin.
Hence, entirely remove the `IgnorePlatform` option and only perform
platform matches if the arch, os or variant is specified explicitly via
the LookupImageOptions or the runtime's system context (as Buildah likes
to do it).
Note that this is a breaking change, so I need to update Buildah and
Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add a method to the libimage runtime to access (a copy of) its
types.SystemContext. That can be helpful for callers which may need to
access the system context.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We must ignore the platform of a local image when doing lookups. Some
images set an incorrect or even invalid platform (see
containers/podman/issues/10682). Doing the lookup while ignoring the
platform checks prevents redundantly downloading the same image.
Note that this has the consequence that a `--pull-never --arch=hurz` may
chose a local image of another architecture. However, I estimate the
benefit of continuing to allow potentially invalid images higher than
not running them (and breaking workloads).
The changes required to touch the corrupted checks. I used the occasion
to make the corrupted checks a bit cheaper.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Currently if the caller does not specify the BigFilesTemporaryDir,
Podman and Buildah users expect this to default TMPDIR environment
variable or /var/tmp if not set.
Moving to libimage caused a regression in this functionality.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Allow for looking up images via customizable arch, os and variant.
This prevents `podman run --arch=xxx` from redundantly pulling down the
image if needed.
Context: containers/podman/issues/10648
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Commit 7f038138c3 introduced a regression to Exists() which would
return an error if the image does not exist.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
While various execution paths in libimage already handle corrupted
images, `(*Runtime).Exists()` did not and would list an image to exist
in the storage even if it is corrupted.
Some corruptions can only be detected when accessing the individual
data. A reliable way of accessing such data is to inspect an image.
Hence, an image will only be listed to exist if a) it has been found
and b) can be inspected. If the inspection fails, the image will be
reported to not exists but without an error. That allows for users
of libimage to properly recover and repull.
Further, add a new unit tests that forces a data corruption and
gradually recovers from it.
Podman will now behave as follows:
```
$ ./bin/podman run -d --rm nginx ls
ERRO[0000] Image nginx exists in local storage but may be corrupted: layer not known
ERRO[0000] Looking up nginx in local storage: layer not known
Resolved "nginx" as an alias (/home/vrothberg/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 596b1d696923 skipped: already exists
Copying blob 30afc0b18f67 skipped: already exists
Copying blob febe5bd23e98 skipped: already exists
Copying blob 69692152171a skipped: already exists
Copying blob 8283eee92e2f skipped: already exists
Copying blob 351ad75a6cfa done
Copying config d1a364dc54 done
Writing manifest to image destination
Storing signatures
56b65883c3c32b67277bcc173bd9f26c27cbbdbc6d3aacf6c552be796eb7a337
```
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
For the sake of Docker compatibility, support the tag@digest notation.
In that case, the tag is stripped off the reference and the digest is
the sole source of truth.
Add a number of tests to make sure we're behaving as expected.
Context: containers/podman/issues/6721
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Recent changes in the image-lookup logic will, in many cases, yield a
check whether an image is a manifest list. This had caused a regression
in Podman's test/system/330-corrupt-images.bats system tests where we're
attempting to delete a corrupted image with a missing manifest. Since
the manifest is missing, the manifest list check fails.
To make the image lookups more tolerant towards this specific error
case, we need to ignore the error but emit a warning, similar to what
we're already doing in the parent-child checks.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Commit 724e7c92b5 fixed an issue when pushing images from of a
platform different than the current machine. That required to disable
the platform matching logic when looking up the image before pushing it.
It also required some restructuring of the code such that manifest lists
are resolved and their instances looked up.
The restructuring in turn introduced a regression when looking up bare
manifest lists. To fix the regression and keep the code simple,
introduce an internal field in the LookupImageOptions that indicates
whether we're looking up a bare manifest list or not.
Now we have clearer separation of concerns between looking up images or
manfifests and whether the looked up image needs to match the current
platform or not.
Add some unit tests exercising the manifest-list code to make sure we're
not regressing again.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When pushing an image, make sure to ignore the platform of the image to
push exactly what the user wishes to. Add a test to make sure we're not
regressing in the future.
To preserve previous behaviour with respect to attempting to push a
manifest list, move the platform check below resolving to a manifest
list.
Fixes: #containers/podman/issues/10344
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add tests for exercising pushing images to various transports and
attempt to pull from the destinations.
Fix an error determining the storage reference and image name when
pushing to containers-storage.
Fix a bug in `RemoveImages`: leaving `names` empty and specifying no
filters should remove *all* images.
Please note that the tests are currently not exercising pushing to a
registry. That requires a local registry but since CI is currently
running inside a container, we cannot do much just yet. Once CI runs
in another environment, I will go back and extend the tests.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We got some crashes in Fedora 34 testing.
Second part of this PR is to fix the config_test which is accitendtly
picking up a containers.conf settings from the host. This change forces
the default settings.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add an event system to libimage. Callers can opt-in to using events by
requesting an event channel via `(*Runtime).EventChannel()`. The
returned channel has a buffer of size 100 which should be sufficient
even under high loads. But, to be on the safe side, writing an event
will time out after 2 seconds to prevent operations from blocking.
Currently, the only user of such an event system is Podman which will
need to convert the `Event` type to what's used internally in libpod.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The following changes were not split into smaller commits since the
entire package is still work in progress and I want to keep moving:
* Various small fixes.
* The internal image cache has been removed as it's a recipe for
inconsistencies for longer running processes. This should make
libimage easier to use for CRI-O and a Podman service.
* LookupImage now returns storage.ErrUnknownImage rather than nil.
This simplifies the callers and makes sure we have a consistent
error.
* LookupImage is now able to handle manifests lists. Unless the
platform is explicitly ignored via the options, the matching
image within the manifest list is now returned. This greatly
simplifies the spec generation in Podman; no callers should have
to worry about this kind of detail.
* LookupImage has been refactored into smaller-sized and easier to
read functions.
* RemoveImages has been changed to assemble the data of removed or
untagged images. This comes in handy for pruning images. I am
heavily against having a dedicated API for pruning since the it's
really just a combination of filtering and removing images which
RemoveImages already supports. Hence these changes to satisfy
the needs of `podman image prune`.
Furthermore, it now returns an []error slice rather than a single
error. Again to make Podman happy which needs to inspect *all*
errors for setting the appropriate exit code.
* A rather large refactoring of the removal code along with very
verbose comments. Those were largely absent in the Podman code base
but there many rules and contracts embedded that I partially could
only reconstruct by manually tests and comparing to Docker.
* Add a new `containers={true,false}` filter which allows filtering
images whether they are used by containers (=true) or if no container
is using them (=false). This filter is required for pruning images
in Podman.
* `libimage/types` has been merged into `libimage`. Podman has to do
_a lot of_ massaging for the remote client already and the types
are pretty much nailed down for the remote API. Hence, I prefer to
do some translation between `libimage` types and what Podman needs
rather than splitting `libimage` in half without an obvious reason.
This way the package is self-contained allowing for an easier
navigation and maintenance.
* `libimage.PullPolicy` has been merged into `pkg/config.PullPolicy`
to have _one_ central place to deal with pull policies. The type
system in `pkg/config` sets "always" as the default unfortunately
but I think consistency is more important at that point.
* Added `CopyOptions.DirForceCompress` to enforce layer compression
when copying to a `dir` destination.
* We now use `github.com/disiqueira/gotree` for pretty printing image
trees. That greatly simplifies the code and we don't have to worry
about the logic of printing a tree. Note that trees are now always
printed top down!
* Added a new `libimage.ManifestList` type along with an API for local
lookups and performing certain operations on it to wrap around
`libimage/manifests` as previously done in `libpod/image` and other
places in Podman.
* Correct caching of `(*Image).Inspect`.
* In addition to username, password and credentials, allow for
speciying an identity token for copying images. That's needed for
Podman's remote API.
* Make image removal more tolerant toward corrupted images.
* A new "until=timestamp" filter that can be used by all APIs
supporting filtering.
* An empty string now resolves to PullPolicyMissing.
* `(*Runtime) systemContextCopy()` returns a deep copy of the runtime's
system context. Golang's shallow copies are very dangerous for long
running processes such as Podman's system service. Hence, we need to
make sure that base data is not altered over time. That adds another
external dependency but I do not see a way around that. Long term,
I desire a `(*containers/image/types.SystemContext).Copy()` function.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The new `libimage` package is an attempt to consolidate the code for
managing container images and performing operations on them such as
pulling, pushing, saving, searching, local lookups, removing etc.
Over time, Buildah, CRI-O and Podman diverged with respect to managing
container images resulting in a high amount of code duplication
rendering the tools harder to maintain (e.g., bug fixes) and harder to
extend (e.g., adding new features) than necessary.
The desire to share all that code in a common library grew and this is
an attempt to address the it.
The changes as they are now pass Buildah CI [1]. Once merged into
Buildah, I expect follow up changes when migrating Podman over to
`libimage`.
Miscellaneous changes:
* Copy `podman/pkg/signal` to `pkg/signal`.
* Copy `buildah/manifests` to `image/manifests`. Note that the unit
tests require root privileges. Skip()'s are added when running
rootless. Currently excluded from linting.
* Copy `buildah/pkg/manifests` to `pkg/manifests`. Currently excluded
from linting.
* Copy `buildah/pkg/supplemented` to `pkg/supplemented`. Currently
excluded from linting.
[1] github.com/containers/buildah/pull/3148
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Valentin Rothberg