Computing the layer tree requires listing all images. Certain code
paths have all images at hand already, so let's optimize a bit to
avoid listing them redundantly.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Checking whether an image is dangling and finding a parent image
requires building a layer tree. Computing a layer tree is expensive, so
add options to `ListImages` to pre-compute the dangling and parent
information ahead of time; that requires 1 layer tree instead of N.
Context: containers/podman/issues/17828
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
It turns out only counting the layers size is not sufficient for
`podman systemd df` as it excludes the size of the manifests, configs
and potentially attached meta data in the storage.
Instead, sum the image sizes but distract redundant layers. That indeed
gives the expected result and does not yield negative results for
reclaimable space.
Remove the unrelease LayersDiskUsage function again to avoid expensive
recalculation of the layer tree. We are still below 1.0, and I am
convinced the total image size belongs into DiskUsage.
NOTE: The DiskUsage function does not have test coverage in libimage.
This should be addressed at some point but in the interest of
time I leverage podman's system tests.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Wrap in a `nil` check to make sure that consumers not using events are
not bothered with log messages. It's probably worth moving the check
into the function but I do not want start Yak shaving in a quick fix.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
As reported in containers/podman/issues/15485, Docker sends untag events
prior to removing the image. Follow that example for compatibility
reasons.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Trying to remove an image that is in use by containers is a confusing
experience for users who may not be aware of "external" containers which
are not displayed in `podman ps` by default (see containers/podman/issues/15006).
Add some context to the error from containers/storage to guide the user
into listing external containers and force-removing the image.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
`github.com/pkg/errors` is deprecated since quite some time so we now
use the native error wrapping for more idiomatic golang.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Wherever possible slices should be allocated with the correct capacity to
avoid unnecessary memory allocations.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The previously inherited behavior from Podman was matching too
aggressively. Now, the filter matches the exact behavior of
Docker, simplifies the code and is tested directly in libimage.
Context: containers/podman#11905
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure to enforce the "latest" tag when looking up images in the
local storage. Also make sure that digested short-names are subject
to the extended digest lookups.
Context: containers/podman/issues/11964
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
With commit e983ccadc6 the dangling checks have been changed to be
compatible with Docker. Since then, the dangling also need to compute
children.
Speed up the dangling and intermediate checks by computing the layer
tree *once* instead of for each filter invocation.
**Before:**
real 0m10.837s
user 0m11.308s
sys 0m4.231s
**After:**
real 0m0.476s
user 0m0.478s
sys 0m0.151s
Context: github.com/containers/podman/issues/11997
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Support pruning images in use by external containers (e.g., build
containers). Requires extending the containers filter, adding a
callback to libpod and a new options for image removal.
Tests will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Allow for passing down credentials when comparing a local image with a
remote one. The linked BZ relates to a regression in `podman auto-update`
but while reading the code I noticed it's also impacting pull policies.
BZ: bugzilla.redhat.com/show_bug.cgi?id=2000943
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure to check an image for corruption before running disk usage on
it. Such checks are already applied on various execution paths but not
yet on disk usage.
Further update the corrupted-image error to include that the image
should be removed to resolve the error. This should ultimately guide
users to resolve the issue.
Fixes: containers/common#751
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Relax the digest checks when untagging. Podman CI relies on `rmi
foo@digest` to actually work even when only untagging the image.
The behavior is wrong since the digest is not getting removed from the
image at all BUT there's currently no way in our stack to do that.
To tackle the problem at the source, we need a way in c/storage to alter
the digests of an image, similar to `SetNames()` for altering the
names/tags of an image. Once that's done, Podman can behave as Docker
does and allow for altering the digests.
For now, to unblock ongoing work, let's just relax the checks and leave
a FIXME note.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure that tag and untag reject digested input. Also add unit tests
for both to make sure we're not regressing in the future.
Fixes: containers/common#710
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When copying images into local storage, parse the manifest of the copied
image and then look up the IDs of the matching image.
There's a short period of time, between when we copy the image into
local storage and when we subsequently go to look for it using the name
that we specified for it when we copied it, when the name we wanted to
assign to the image could have been assigned to another image by another
process.
The manifest that we copied as part of the image that we copied will
still be in the right image regardless, and we can use that to find the
image's ID, and from there fill out our own Image structure that we
return to our caller.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fix a bug where not all removed images were actually reported as such.
A regression test will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
As discussed in github.com/containers/podman/issues/10832 the definition
of a "dangling" image in Podman has historically been incorrect. While
the Docker docs describe a dangling image as an image without a tag, and
Podman implemented the filters as such, Docker actually implemented the
filters for images without a tag and without children.
Refine the dangling filters and hence `IsDangling()` to only return true
if an image is untagged and has no children.
Also correct the comments of `IsIntermediate()`.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Some users rely on events being written *after* the operation ran.
Hence, defer all event writes.
Context: containers/podman/issues/10812
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We must ignore the platform of a local image when doing lookups. Some
images set an incorrect or even invalid platform (see
containers/podman/issues/10682). Doing the lookup while ignoring the
platform checks prevents redundantly downloading the same image.
Note that this has the consequence that a `--pull-never --arch=hurz` may
chose a local image of another architecture. However, I estimate the
benefit of continuing to allow potentially invalid images higher than
not running them (and breaking workloads).
The changes required to touch the corrupted checks. I used the occasion
to make the corrupted checks a bit cheaper.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When removing an image by name, do not remove the image and all its
tags, even if force is set. Instead, just untag the specified name.
Note: adjust the load test to preserve the order in the untagged field.
Also vendor in the latest HEAD in containers/image to fix a bug revealed
in Podman CI.
Context: containers/podman/issues/10685
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add unit tests for exercising all kinds of function of an `Image`
object.
Also remove an unused, redundant (and incomplete) `MountPoint` function.
`Mountpoint` is used by Podman instead.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When comparing the digests of a local image and a remote image, we've
been reading the manifest from the remote image using NewImage(), which
may or may not return a list, and have been comparing the digest of that
manifest to only one of the local image's manifests.
Start checking if the remote reference points to a manifest list, and if
it is, find the image in the list that we'd choose to pull, and use its
manifest digest for the comparison.
When looking up the digest of the local image to compare to the remote
image, consider them to be the same image if any of the manifests in the
image record has the same digest as the remote manifest, which is now
known to not be a list.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an event system to libimage. Callers can opt-in to using events by
requesting an event channel via `(*Runtime).EventChannel()`. The
returned channel has a buffer of size 100 which should be sufficient
even under high loads. But, to be on the safe side, writing an event
will time out after 2 seconds to prevent operations from blocking.
Currently, the only user of such an event system is Podman which will
need to convert the `Event` type to what's used internally in libpod.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The following changes were not split into smaller commits since the
entire package is still work in progress and I want to keep moving:
* Various small fixes.
* The internal image cache has been removed as it's a recipe for
inconsistencies for longer running processes. This should make
libimage easier to use for CRI-O and a Podman service.
* LookupImage now returns storage.ErrUnknownImage rather than nil.
This simplifies the callers and makes sure we have a consistent
error.
* LookupImage is now able to handle manifests lists. Unless the
platform is explicitly ignored via the options, the matching
image within the manifest list is now returned. This greatly
simplifies the spec generation in Podman; no callers should have
to worry about this kind of detail.
* LookupImage has been refactored into smaller-sized and easier to
read functions.
* RemoveImages has been changed to assemble the data of removed or
untagged images. This comes in handy for pruning images. I am
heavily against having a dedicated API for pruning since the it's
really just a combination of filtering and removing images which
RemoveImages already supports. Hence these changes to satisfy
the needs of `podman image prune`.
Furthermore, it now returns an []error slice rather than a single
error. Again to make Podman happy which needs to inspect *all*
errors for setting the appropriate exit code.
* A rather large refactoring of the removal code along with very
verbose comments. Those were largely absent in the Podman code base
but there many rules and contracts embedded that I partially could
only reconstruct by manually tests and comparing to Docker.
* Add a new `containers={true,false}` filter which allows filtering
images whether they are used by containers (=true) or if no container
is using them (=false). This filter is required for pruning images
in Podman.
* `libimage/types` has been merged into `libimage`. Podman has to do
_a lot of_ massaging for the remote client already and the types
are pretty much nailed down for the remote API. Hence, I prefer to
do some translation between `libimage` types and what Podman needs
rather than splitting `libimage` in half without an obvious reason.
This way the package is self-contained allowing for an easier
navigation and maintenance.
* `libimage.PullPolicy` has been merged into `pkg/config.PullPolicy`
to have _one_ central place to deal with pull policies. The type
system in `pkg/config` sets "always" as the default unfortunately
but I think consistency is more important at that point.
* Added `CopyOptions.DirForceCompress` to enforce layer compression
when copying to a `dir` destination.
* We now use `github.com/disiqueira/gotree` for pretty printing image
trees. That greatly simplifies the code and we don't have to worry
about the logic of printing a tree. Note that trees are now always
printed top down!
* Added a new `libimage.ManifestList` type along with an API for local
lookups and performing certain operations on it to wrap around
`libimage/manifests` as previously done in `libpod/image` and other
places in Podman.
* Correct caching of `(*Image).Inspect`.
* In addition to username, password and credentials, allow for
speciying an identity token for copying images. That's needed for
Podman's remote API.
* Make image removal more tolerant toward corrupted images.
* A new "until=timestamp" filter that can be used by all APIs
supporting filtering.
* An empty string now resolves to PullPolicyMissing.
* `(*Runtime) systemContextCopy()` returns a deep copy of the runtime's
system context. Golang's shallow copies are very dangerous for long
running processes such as Podman's system service. Hence, we need to
make sure that base data is not altered over time. That adds another
external dependency but I do not see a way around that. Long term,
I desire a `(*containers/image/types.SystemContext).Copy()` function.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The new `libimage` package is an attempt to consolidate the code for
managing container images and performing operations on them such as
pulling, pushing, saving, searching, local lookups, removing etc.
Over time, Buildah, CRI-O and Podman diverged with respect to managing
container images resulting in a high amount of code duplication
rendering the tools harder to maintain (e.g., bug fixes) and harder to
extend (e.g., adding new features) than necessary.
The desire to share all that code in a common library grew and this is
an attempt to address the it.
The changes as they are now pass Buildah CI [1]. Once merged into
Buildah, I expect follow up changes when migrating Podman over to
`libimage`.
Miscellaneous changes:
* Copy `podman/pkg/signal` to `pkg/signal`.
* Copy `buildah/manifests` to `image/manifests`. Note that the unit
tests require root privileges. Skip()'s are added when running
rootless. Currently excluded from linting.
* Copy `buildah/pkg/manifests` to `pkg/manifests`. Currently excluded
from linting.
* Copy `buildah/pkg/supplemented` to `pkg/supplemented`. Currently
excluded from linting.
[1] github.com/containers/buildah/pull/3148
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Valentin Rothberg