To prevent the podman remote client from using libimage which causes a
lot of bloat due the c/image and c/storage dependencies add the
`!remote` tag.
This will cause a hard compile time failure if the remote client ends up
pulling in libimage.
Fixes#1702
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
podman untag should error out of a name is given which does not exists
for the given image. This regression was added in commit a4a7c28aef.
There was even a test which meant to check for it but unfortunately it
did not actually check for what it should. The doNotExist check failed
early to the upper case in the repo name.
The tests have been updated to check for actual error messages to show
ensure it is failing for the right reason.
This also showed that `normalizing name` message was included twice so
I removed one case to not stutter.
Fixes a4a7c28aef ("libimage: support parallel tag/untag")
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The c/storage SetNames API is depracated because it is not race free to
first get the list of names and then append our new name then write the
full list back. Instead a better Add/RemovesNames API has been added.
Tag and Untag should use these to prevent race conditions that can be
easily reproduce using podman tag in parallel. Tests have been added to
ensure it is working correctly.
Fixes https://github.com/containers/podman/issues/17515
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Move the platform code into its own directory
under libimage so it can be used by farm build in
podman without causing the binary size to increase a lot.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
The id and digest filters for podman images was broken
as it would only match on the full ID or Digest, so when
users would give a substring of either not results would be
given even though there were matches.
Fix to check on prefix now instead.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Make sure to close all `ImageCloser` to avoid potential memory leaks.
Fixes: containers/common#1533
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
We don't need a reference.NamedTagged now.
That also makes the namedTagged variable in the caller more local.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Right now that's not simpler, but it will enable simplification of the caller.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Scary features should have scary names. Also add a comment
to make it less likely that this semantics will spread.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
To fix a BZ on auto-updates, we need to improve the system tests.
To improve the system tests, we need to use the existing test
infrastructure. To use the existing test infrastructure, we need
to be able to skip TLS verification in tests. To be able to skip TLS
verification in the tests, we need a new --tls-verify flag for
podman-auto-update. And for that to work, we need a way to pass that
down to libimage when checking if there's a new image on the registry.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2218315
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
When looking up an image by digest, make sure that the entire repository
of the specified value is considered. Previously, both the repository
and the tag have been ignored and we looked for _some_ image with a
matching digest.
As outlined in #1248, Docker stopped ignoring the repository with
version v20.10.20 (Oct '22) which is a compelling reason to do the same.
To be clear, previously `something@digest` would look for any image with
`digest` while `something` is entirely ignored. With this change, both
`something` and `digest` must match the image.
This change breaks two e2e tests in Podman CI which relied on the
previous behavior. There is a risk of breaking users but there is a
strong security argument to perform this change: if the repository does
not match the (previously) returned issue, there is a fair chance of a
user error.
Fixes: containers/common#1248
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Commit 3e9cdd9c53 took a first stab at making operating on the layer
tree more resilient by detecting unknown-image errors. Testing in
containers/podman/pull/18980 revealed that we also need to detect
unknown-layer and -size errors as well.
Move the errors checks into a convenience function and update the
relevant call sites to facilitate future changes. Export the
function since Podman needs the very same checks when operating
on images, for instance, when looking up image labels.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Computing the layer tree requires listing all images. Certain code
paths have all images at hand already, so let's optimize a bit to
avoid listing them redundantly.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Checking whether an image is dangling and finding a parent image
requires building a layer tree. Computing a layer tree is expensive, so
add options to `ListImages` to pre-compute the dangling and parent
information ahead of time; that requires 1 layer tree instead of N.
Context: containers/podman/issues/17828
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
It turns out only counting the layers size is not sufficient for
`podman systemd df` as it excludes the size of the manifests, configs
and potentially attached meta data in the storage.
Instead, sum the image sizes but distract redundant layers. That indeed
gives the expected result and does not yield negative results for
reclaimable space.
Remove the unrelease LayersDiskUsage function again to avoid expensive
recalculation of the layer tree. We are still below 1.0, and I am
convinced the total image size belongs into DiskUsage.
NOTE: The DiskUsage function does not have test coverage in libimage.
This should be addressed at some point but in the interest of
time I leverage podman's system tests.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Wrap in a `nil` check to make sure that consumers not using events are
not bothered with log messages. It's probably worth moving the check
into the function but I do not want start Yak shaving in a quick fix.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
As reported in containers/podman/issues/15485, Docker sends untag events
prior to removing the image. Follow that example for compatibility
reasons.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Trying to remove an image that is in use by containers is a confusing
experience for users who may not be aware of "external" containers which
are not displayed in `podman ps` by default (see containers/podman/issues/15006).
Add some context to the error from containers/storage to guide the user
into listing external containers and force-removing the image.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
`github.com/pkg/errors` is deprecated since quite some time so we now
use the native error wrapping for more idiomatic golang.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Wherever possible slices should be allocated with the correct capacity to
avoid unnecessary memory allocations.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The previously inherited behavior from Podman was matching too
aggressively. Now, the filter matches the exact behavior of
Docker, simplifies the code and is tested directly in libimage.
Context: containers/podman#11905
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure to enforce the "latest" tag when looking up images in the
local storage. Also make sure that digested short-names are subject
to the extended digest lookups.
Context: containers/podman/issues/11964
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
With commit e983ccadc6 the dangling checks have been changed to be
compatible with Docker. Since then, the dangling also need to compute
children.
Speed up the dangling and intermediate checks by computing the layer
tree *once* instead of for each filter invocation.
**Before:**
real 0m10.837s
user 0m11.308s
sys 0m4.231s
**After:**
real 0m0.476s
user 0m0.478s
sys 0m0.151s
Context: github.com/containers/podman/issues/11997
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Support pruning images in use by external containers (e.g., build
containers). Requires extending the containers filter, adding a
callback to libpod and a new options for image removal.
Tests will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Allow for passing down credentials when comparing a local image with a
remote one. The linked BZ relates to a regression in `podman auto-update`
but while reading the code I noticed it's also impacting pull policies.
BZ: bugzilla.redhat.com/show_bug.cgi?id=2000943
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure to check an image for corruption before running disk usage on
it. Such checks are already applied on various execution paths but not
yet on disk usage.
Further update the corrupted-image error to include that the image
should be removed to resolve the error. This should ultimately guide
users to resolve the issue.
Fixes: containers/common#751
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Relax the digest checks when untagging. Podman CI relies on `rmi
foo@digest` to actually work even when only untagging the image.
The behavior is wrong since the digest is not getting removed from the
image at all BUT there's currently no way in our stack to do that.
To tackle the problem at the source, we need a way in c/storage to alter
the digests of an image, similar to `SetNames()` for altering the
names/tags of an image. Once that's done, Podman can behave as Docker
does and allow for altering the digests.
For now, to unblock ongoing work, let's just relax the checks and leave
a FIXME note.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make sure that tag and untag reject digested input. Also add unit tests
for both to make sure we're not regressing in the future.
Fixes: containers/common#710
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When copying images into local storage, parse the manifest of the copied
image and then look up the IDs of the matching image.
There's a short period of time, between when we copy the image into
local storage and when we subsequently go to look for it using the name
that we specified for it when we copied it, when the name we wanted to
assign to the image could have been assigned to another image by another
process.
The manifest that we copied as part of the image that we copied will
still be in the right image regardless, and we can use that to find the
image's ID, and from there fill out our own Image structure that we
return to our caller.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fix a bug where not all removed images were actually reported as such.
A regression test will be added to Podman.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
As discussed in github.com/containers/podman/issues/10832 the definition
of a "dangling" image in Podman has historically been incorrect. While
the Docker docs describe a dangling image as an image without a tag, and
Podman implemented the filters as such, Docker actually implemented the
filters for images without a tag and without children.
Refine the dangling filters and hence `IsDangling()` to only return true
if an image is untagged and has no children.
Also correct the comments of `IsIntermediate()`.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Some users rely on events being written *after* the operation ran.
Hence, defer all event writes.
Context: containers/podman/issues/10812
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We must ignore the platform of a local image when doing lookups. Some
images set an incorrect or even invalid platform (see
containers/podman/issues/10682). Doing the lookup while ignoring the
platform checks prevents redundantly downloading the same image.
Note that this has the consequence that a `--pull-never --arch=hurz` may
chose a local image of another architecture. However, I estimate the
benefit of continuing to allow potentially invalid images higher than
not running them (and breaking workloads).
The changes required to touch the corrupted checks. I used the occasion
to make the corrupted checks a bit cheaper.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When removing an image by name, do not remove the image and all its
tags, even if force is set. Instead, just untag the specified name.
Note: adjust the load test to preserve the order in the untagged field.
Also vendor in the latest HEAD in containers/image to fix a bug revealed
in Podman CI.
Context: containers/podman/issues/10685
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add unit tests for exercising all kinds of function of an `Image`
object.
Also remove an unused, redundant (and incomplete) `MountPoint` function.
`Mountpoint` is used by Podman instead.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When comparing the digests of a local image and a remote image, we've
been reading the manifest from the remote image using NewImage(), which
may or may not return a list, and have been comparing the digest of that
manifest to only one of the local image's manifests.
Start checking if the remote reference points to a manifest list, and if
it is, find the image in the list that we'd choose to pull, and use its
manifest digest for the comparison.
When looking up the digest of the local image to compare to the remote
image, consider them to be the same image if any of the manifests in the
image record has the same digest as the remote manifest, which is now
known to not be a list.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an event system to libimage. Callers can opt-in to using events by
requesting an event channel via `(*Runtime).EventChannel()`. The
returned channel has a buffer of size 100 which should be sufficient
even under high loads. But, to be on the safe side, writing an event
will time out after 2 seconds to prevent operations from blocking.
Currently, the only user of such an event system is Podman which will
need to convert the `Event` type to what's used internally in libpod.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The following changes were not split into smaller commits since the
entire package is still work in progress and I want to keep moving:
* Various small fixes.
* The internal image cache has been removed as it's a recipe for
inconsistencies for longer running processes. This should make
libimage easier to use for CRI-O and a Podman service.
* LookupImage now returns storage.ErrUnknownImage rather than nil.
This simplifies the callers and makes sure we have a consistent
error.
* LookupImage is now able to handle manifests lists. Unless the
platform is explicitly ignored via the options, the matching
image within the manifest list is now returned. This greatly
simplifies the spec generation in Podman; no callers should have
to worry about this kind of detail.
* LookupImage has been refactored into smaller-sized and easier to
read functions.
* RemoveImages has been changed to assemble the data of removed or
untagged images. This comes in handy for pruning images. I am
heavily against having a dedicated API for pruning since the it's
really just a combination of filtering and removing images which
RemoveImages already supports. Hence these changes to satisfy
the needs of `podman image prune`.
Furthermore, it now returns an []error slice rather than a single
error. Again to make Podman happy which needs to inspect *all*
errors for setting the appropriate exit code.
* A rather large refactoring of the removal code along with very
verbose comments. Those were largely absent in the Podman code base
but there many rules and contracts embedded that I partially could
only reconstruct by manually tests and comparing to Docker.
* Add a new `containers={true,false}` filter which allows filtering
images whether they are used by containers (=true) or if no container
is using them (=false). This filter is required for pruning images
in Podman.
* `libimage/types` has been merged into `libimage`. Podman has to do
_a lot of_ massaging for the remote client already and the types
are pretty much nailed down for the remote API. Hence, I prefer to
do some translation between `libimage` types and what Podman needs
rather than splitting `libimage` in half without an obvious reason.
This way the package is self-contained allowing for an easier
navigation and maintenance.
* `libimage.PullPolicy` has been merged into `pkg/config.PullPolicy`
to have _one_ central place to deal with pull policies. The type
system in `pkg/config` sets "always" as the default unfortunately
but I think consistency is more important at that point.
* Added `CopyOptions.DirForceCompress` to enforce layer compression
when copying to a `dir` destination.
* We now use `github.com/disiqueira/gotree` for pretty printing image
trees. That greatly simplifies the code and we don't have to worry
about the logic of printing a tree. Note that trees are now always
printed top down!
* Added a new `libimage.ManifestList` type along with an API for local
lookups and performing certain operations on it to wrap around
`libimage/manifests` as previously done in `libpod/image` and other
places in Podman.
* Correct caching of `(*Image).Inspect`.
* In addition to username, password and credentials, allow for
speciying an identity token for copying images. That's needed for
Podman's remote API.
* Make image removal more tolerant toward corrupted images.
* A new "until=timestamp" filter that can be used by all APIs
supporting filtering.
* An empty string now resolves to PullPolicyMissing.
* `(*Runtime) systemContextCopy()` returns a deep copy of the runtime's
system context. Golang's shallow copies are very dangerous for long
running processes such as Podman's system service. Hence, we need to
make sure that base data is not altered over time. That adds another
external dependency but I do not see a way around that. Long term,
I desire a `(*containers/image/types.SystemContext).Copy()` function.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Oleksandr Redko
openshift-merge-bot[bot]
Paul Holzinger
Urvashi Mohnani
Valentin Rothberg
Miloslav Trmač
OpenShift Merge Robot
Karthik Elango
Sascha Grunert
Valentin Rothberg
Daniel J Walsh
Nalin Dahyabhai