Commit Graph

132 Commits

Author SHA1 Message Date
OpenShift Merge Robot 030366b89e Merge pull request #1024 from Luap99/linters
enable unparam, exportloopref and revive linters
2022-05-06 08:57:16 -04:00
Paul Holzinger cc110440e4 enable unparam, exportloopref and revive linters
unparam and exportloopref already work without changes.
For revive I had to silence many naming issues. I decided to silence them
instead of changing the name because I didn't want to break any code.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2022-05-06 13:32:35 +02:00
Daniel J Walsh aec9f8f79b libimage should be using containers.conf for tmpdir
if image_copy_tmp_dir is set in containers.conf it needs to be used in
the systemcontext for BigFilesTemporaryDir value.

Fixes: https://github.com/containers/podman/issues/14091

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2022-05-04 16:55:25 -04:00
Kir Kolyshkin e368436e5f libimage: suppress appendAssign gocritic warning
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 17:03:19 -07:00
Kir Kolyshkin 381b103ad2 libimage/manifests: fix ifelse gocritic warning
This one:

	libimage/manifests/manifests.go:387:10: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic)
			} else {
			       ^

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 17:02:33 -07:00
Kir Kolyshkin b951b72412 Gofumpt the code
gofumpt is a stricter version of gofmt, basically making the code more
readable, and fixing the gocritic's octalLiterar warnings like this one:

	pkg/util/util_supported.go:26:17: octalLiteral: use new octal literal style, 0o722 (gocritic)
		return (perm & 0722) == 0700
			       ^

Generated by gofumpt -w .

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 16:50:11 -07:00
Kir Kolyshkin bbe3459db0 libimage/manifests: fix a gosimple linter warning
> libimage/manifests/manifests.go:408:3: S1033: unnecessary guard around call to delete (gosimple)
> 		if _, needToDelete := l.instances[instanceDigest]; needToDelete {
> 		^

Indeed, we can just call delete right away.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 15:20:01 -07:00
Kir Kolyshkin 4393468343 *_test.go: fix or suppress errcheck warnings
Where we can check the error, do it. Where we can not, ignore it.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 15:18:22 -07:00
Kir Kolyshkin ca15ecaf45 Silence errcheck linter warnings
Error checking is important. In these two cases, though, we don't have a
way to return an error, so make it explicit that we ignore the error.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-04-09 15:17:32 -07:00
Kir Kolyshkin 2ddd7fd725 libimage: nit
Rename hist to history.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2022-03-30 11:39:49 -07:00
Aditya R 4057001d4e libimage: import should not ignore configured variant if any
`image-spec` already supports variant and if `ImportOptions` contains
any configured `Variant` it should be set on the imported image as well.

What does this fix:

Allows `podman import --variant <something> some.tar image-name` to
become functional from `no-op`

Ref: https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/config.go#L93

Signed-off-by: Aditya R <arajan@redhat.com>
2022-03-25 14:27:10 +05:30
rvandernoort 5ba5855c6f Remove container related filter tests -> move to podman
Signed-off-by: rvandernoort <s.r.vandernoort@student.tudelft.nl>
2022-03-23 13:14:51 +01:00
Valentin Rothberg 2696c75e6d libimage: RemoveImages: add Ignore field
Add a field to `RemoveImages` that would ingore if a specified image
does not exist and not throw an error.

The intended use case is adding a `podman rmi --ignore` flag.

Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2022-03-17 14:55:16 +01:00
Valentin Rothberg 80d883a8d7 libimage: image lookup: fix ID vs short name
When looking up an image by a short name that prefixes another image's
ID, the one matching the short name should be returned.

This means that we need to do a final lookup in the storage with the
specified name (without normalization) to continue matching short IDs.

Since it's common that users of libimage (e.g., Buildah) internally
refer to images by full ID, let's make sure that we check for that
first.  This way, we'll match full IDs on first lookup and keep the
expected performance.

Note that a name starting with `sha2556:` must be followed by a 64-byte
hex value; something we didn't check for before.

Fixes: containers/podman/issues/12761
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2022-02-24 14:42:37 +01:00
Valentin Rothberg 18ad7e4f77 libimage: tests: redirect docker.io to GCR mirror
Unfortunately, it the table was commented out, so it was a NOP.
Fix one test where the image being pulled was not present on
the GCR mirror.

Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2022-02-24 14:41:56 +01:00
Nalin Dahyabhai 630d2df7f2 Add libimage/manifests.PushOptions.SourceFilter
Add a SourceFilter hook that allows a caller to intercept and filter
attempts to read source blobs when pushing a manifest list.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2022-02-03 17:52:09 -05:00
Valentin Rothberg 8ba7e55c56 libimage: pull: platform checks for non-local platform
After containers/podman/issues/10682, we decided to always re-pull
images of non-local platforms and match *any* local image. Over time, we
refined this logic to not *always* pull the image but only if there is a
*newer* one. This has slightly changed the semantics and requires to
perform platform checks when looking up a local image. Otherwise, bogus
values would match a local image and mistakenly return it.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2022-01-12 10:15:09 +01:00
OpenShift Merge Robot e4d8d9ba27 Merge pull request #880 from vrothberg/platform-pull-policy
libimage: refine pull-policy enforcement for custom platforms
2022-01-10 17:53:09 +01:00
Valentin Rothberg e369277804 libimage: refine pull-policy enforcement for custom platforms
When pulling down an image with a user-specified custom platform, we
try to make sure that user gets what they are asking for.  An inherent
issue with multi-arch images is that there are many images in the wild
which do not get the platform right (see containers/podman/issues/10682).
That means we need to pessimistically assume that the local image is
wrong and pull the "correct" one down from the registry; in the worst case
that is redundant work but we have a guarantee of correctness.

Motivated by containers/podman/issues/12707 I had another look at the
code and found some space for optimizations.  Previously, we enforced
the pull policy to "always" but that may be too aggressive since we may
be running in an airgapped environment and the local image is correct.

With this change, we enforce the pull policy to "newer" which makes
errors non-fatal in case a local image has been found; this seems like a
good middleground between making sure we are serving the "correct" image
and user friendliness.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2022-01-10 16:27:10 +01:00
OpenShift Merge Robot 94502d0861 Merge pull request #879 from vrothberg/path-pull
pull: fix pulling from dir transport
2022-01-10 16:22:53 +01:00
Valentin Rothberg 3aa83d3500 libimage filters: allow duplicates with the same value
Podman CI wants that for pruning. The pruning code unconditionally sets
the dangling filter since that's crucial for pruning.

Also fix a typo in the error message:
`s/more then/more than/`

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2022-01-10 14:32:16 +01:00
Valentin Rothberg 95872a5d60 pull: fix pulling from dir transport
Path-based transports may contain characters that are invalid for a
reference.  In such cases, we should pessimistically generate an ID
and not attempt to look at the (possibly path-based) string within the
transport.

This fixes an error when running `podman run dir:/tmp/CapitalChar` and
will prevent the same issue for the upcoming SIF transport.

Extend the tests to make sure we're not going to regress in the future.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2022-01-10 13:34:36 +01:00
Valentin Rothberg f80a7f6c96 image lookup: apply checks for matching digest
When looking up an image by digest, make sure that the matching image is
also subject to further attribute checks (e.g., manifest list checks) as
specified by the user.

Fixes: containers/podman/issues/12729
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2022-01-10 13:18:42 +01:00
Paul Holzinger 9255c2eec0 Revert "pull: fix pulling from dir transport"
This reverts commit 913fe2c9cb.

Both podman and buildah CI are failing because of this change. Since
Valentin is out and I need to get my network changes merged I want to
revert this for now.

Example failure from buildah test:
```
[+2471s] not ok 507 pull-from-local-directory
[+2471s] # (from function `assert' in file ./helpers.bash, line 342,
[+2471s] #  from function `expect_output' in file ./helpers.bash, line 369,
[+2471s] #  in test file ./pull.bats, line 89)
[+2471s] #   `expect_output --substring "localhost${TESTDIR}/buildahtest:latest"' failed
[+2471s] # /var/tmp/go/src/github.com/containers/buildah/tests /var/tmp/go/src/github.com/containers/buildah/tests
[+2471s] # $ /var/tmp/go/src/github.com/containers/buildah/tests/./../bin/buildah pull --signature-policy /var/tmp/go/src/github.com/containers/buildah/tests/./policy.json alpine
[+2471s] # Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
[+2471s] # Trying to pull docker.io/library/alpine:latest...
[+2471s] # Getting image source signatures
[+2471s] # Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
[+2471s] # Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
[+2471s] # Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # Writing manifest to image destination
[+2471s] # Storing signatures
[+2471s] # 961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # $ /var/tmp/go/src/github.com/containers/buildah/tests/./../bin/buildah push --signature-policy /var/tmp/go/src/github.com/containers/buildah/tests/./policy.json docker.io/library/alpine:latest dir:/var/tmp/buildah_tests.c47drl/buildahtest
[+2471s] # Getting image source signatures
[+2471s] # Copying blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
[+2471s] # Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # Writing manifest to image destination
[+2471s] # Storing signatures
[+2471s] # $ /var/tmp/go/src/github.com/containers/buildah/tests/./../bin/buildah rmi alpine
[+2471s] # untagged: docker.io/library/alpine:latest
[+2471s] # 961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # $ /var/tmp/go/src/github.com/containers/buildah/tests/./../bin/buildah pull --signature-policy /var/tmp/go/src/github.com/containers/buildah/tests/./policy.json dir:/var/tmp/buildah_tests.c47drl/buildahtest
[+2471s] # Getting image source signatures
[+2471s] # Copying blob sha256:f3ac2f9422602f687109cc39270c47da673c6a61b5ca4958bc232adbc65912e7
[+2471s] # Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # Writing manifest to image destination
[+2471s] # Storing signatures
[+2471s] # 961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
[+2471s] # $ /var/tmp/go/src/github.com/containers/buildah/tests/./../bin/buildah images --format {{.Name}}:{{.Tag}}
[+2471s] # <none>:<none>
[+2471s] # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[+2471s] # #|     FAIL: buildah images --format {{.Name}}:{{.Tag}}
[+2471s] # #| expected: =~ 'localhost/var/tmp/buildah_tests.c47drl/buildahtest:latest'
[+2471s] # #|   actual:    '<none>:<none>'
[+2471s] # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2022-01-07 14:18:32 +01:00
Valentin Rothberg 281201d87d libimage: normalize platform
Buildah, containers and probably other container engines are normalizing
the platform parameters to support common values.  For instance, "x86_64"
is normalized to the OCI conformant "amd64".

Use the same normalization when copying images and looking up local
images.  Also add some debug logs to facilitate future debugging.

Fixes: containers/podman/issues/12680
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-23 13:09:03 +01:00
OpenShift Merge Robot 9f0f225581 Merge pull request #864 from vrothberg/dir-pull
pull: fix pulling from dir transport
2021-12-20 15:40:31 +01:00
Valentin Rothberg 913fe2c9cb pull: fix pulling from dir transport
Path-based transports may contain characters that are invalid for a
reference.  In such cases, we should pessimistically generate an ID
and not attempt to look at the (possibly path-based) string within the
transport.

This fixes an error when running `podman run dir:/tmp/CapitalChar` and
will prevent the same issue for the upcoming SIF transport.

Extend the tests to make sure we're not going to regress in the future.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-20 14:55:36 +01:00
Daniel J Walsh 862adf81e6 Add filters for image and manifest
For compatibility mode, we need to have a filter that gets rid of
manifest lists from the list of all images.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2021-12-14 10:19:57 -05:00
Valentin Rothberg f9d367b72f libimage: filter: check if image is corrupted in case of error
Further turn applying the filters into a method of `type Image`.
Also change the loop when filtering images to not append the
loop variable to the result slice; it's always the same pointer.

Fixes: containers/podman/issues/12582
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-14 15:42:27 +01:00
Daniel J Walsh 009078de4c Support matching filters disjoint filters
The same filters types should be disjoint, while
each selected filter type is required. This will allow callers to pass
multiple reference filters and if an image matches it is returned.

Other filters the image has to match all filters to be returned.

Specifying "since", "after", "before", "containers", "dangling","intermediate",  "readonly", "until" more then once is an error.

[NO NEW TESTS NEEDED] I will open up a validation against buildah to
make sure this passes before merging.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2021-12-13 14:34:21 -05:00
Valentin Rothberg c08c28e17c load: support buildkit archives
Archives generated with buildkit have some kind of "hybrid" layout which
is the same for OCI and Docker archives.  OCI ones ship with a
manifest.json but set the image's reference in the index.json but in a
custom annotation and not the one the OCI image spec wants.  Archives
in the Docker format set the reference in `RepoTags` of the
manifest.json.

To support these archives, simply look for the custom containerd
annotation *and* change the order back to give OCI archives precedence.

Fixes: containers/podman/issues/12560
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-10 13:24:05 +01:00
Valentin Rothberg 1deba4dcd4 libimage: fix reference filters
It turns out that FamiliarMatch is only useful for matching Docker Hub
but we should not limit it to that and match values against registry.

For instance, FamiliarMatch is *not* able to match a FQN reference
against a Docker Hub image.  I am convinced that we should *not* behave
as Docker does in this case.

This brings us back to the behavior prior to commit c15939746c but
with a fixed matching algorithm.  The specified value will now be
matched against 1) the FQN 2) without domain 3) without domain and path.
If specified also a second time without digest/tag.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-05 16:16:52 +01:00
OpenShift Merge Robot 929fd4f165 Merge pull request #847 from vrothberg/reference-filter
reference filter: match exact behavior of Docker
2021-12-03 16:18:47 +01:00
Valentin Rothberg c15939746c reference filter: match exact behavior of Docker
The previously inherited behavior from Podman was matching too
aggressively.  Now, the filter matches the exact behavior of
Docker, simplifies the code and is tested directly in libimage.

Context: containers/podman#11905
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-12-03 10:52:47 +01:00
Daniel J Walsh 1cb44dc5a1 If we tell system to never pull we should not pull
Fixes: https://github.com/containers/buildah/issues/3596

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2021-11-30 15:00:17 -05:00
Valentin Rothberg a5a6620b33 libimage: search custom registries
Add a new search option to specify custom search registries.
The search tests live in Podman and I currently do not have
enough cycles to add completely new ones to libimage.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-11-25 14:28:45 +01:00
Valentin Rothberg a5f253cddf libimage: enforce "latest" tag when looking up images
Make sure to enforce the "latest" tag when looking up images in the
local storage.  Also make sure that digested short-names are subject
to the extended digest lookups.

Context: containers/podman/issues/11964
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-11-22 15:19:38 +01:00
OpenShift Merge Robot 0cffd00486 Merge pull request #818 from nalind/manifest-variant
libimage/manifests.list.Add(): pull variant info from configs
2021-11-11 01:39:35 +01:00
OpenShift Merge Robot 01a7c761db Merge pull request #825 from vrothberg/download
pkg/download: for downloading files into tmp
2021-11-10 15:37:43 +01:00
Valentin Rothberg 8eb373cb0a pkg/download: for downloading files into tmp
cmd/podman needs that functionality as well, so I figured it makes sense
to break it into a separate package to reduce code clones.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-11-10 14:41:42 +01:00
Valentin Rothberg 1b87e6d155 libimage: inspect: extract healthchecks from configs
buildkit is setting the health check in the image's config while Docker
and Podman set it in the image's container config.  Hence, if the
container config's healthcheck is nil, have a look at the config.

Fixes: #containers/podman/issues/12226
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-11-09 13:02:54 +01:00
Nalin Dahyabhai a248c4faa6 libimage/manifests.list.Add(): pull variant info from configs
When we're given an image reference to add to a manifest list, if we're
setting the instance info's architecture using information from the
image, set its variant field, too.  Likewise, if we're setting the OS
using information from the image, set the OS version and features
fields.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2021-11-04 14:51:14 -04:00
Valentin Rothberg 7cd4a8a161 libimage: speed up image filters
With commit e983ccadc6 the dangling checks have been changed to be
compatible with Docker.  Since then, the dangling also need to compute
children.

Speed up the dangling and intermediate checks by computing the layer
tree *once* instead of for each filter invocation.

**Before:**
real    0m10.837s
user    0m11.308s
sys     0m4.231s

**After:**
real    0m0.476s
user    0m0.478s
sys     0m0.151s

Context: github.com/containers/podman/issues/11997
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-10-19 15:30:06 +02:00
OpenShift Merge Robot 27e2c6604d Merge pull request #802 from jelly/search_registry_truncated
libimage: Return the full registry domain when searching
2021-10-14 19:27:59 +02:00
Valentin Rothberg 607e1273f1 libimage: Inspect: add InspectOptions
Add an InspectOptions struct for inspecting images.  This is a breaking
change but I think it's worth it since a considerable amount of CPU time
is spent in computing the image's parent (i.e., computing the layer is
costly) while this data is oftentimes not needed.

This cuts off 10ms of container-creation time in Podman.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-10-08 13:45:54 +02:00
Valentin Rothberg cbb69a9e64 libimage: Pull: fix logs
Exchange two debug logs in the pull-never policy.  Those were probably
not moved along with the code and could lead to confusions when reading
debug logs.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-10-08 10:34:48 +02:00
Jelle van der Waa 3534572787 libimage: Include Index in listTags response
A search without the listTags option normally returns the Index, but as
searchRepositoryTags builds the SearchResult struct the Index was not
included.

Signed-off-by: Jelle van der Waa <jvanderwaa@redhat.com>
2021-10-07 17:45:43 +02:00
Jelle van der Waa ecbdf8b9ce libimage: Return the full registry domain when searching
Searching for images in registry.fedoraproject.org returns
fedoraproject.org as registry in the search results. When relying on the
Index to group results from different registries this is an issue.

Signed-off-by: Jelle van der Waa <jvanderwaa@redhat.com>
2021-10-05 11:05:38 +02:00
Valentin Rothberg 45bb4d8d48 libimage: load: try docker-archive first
Make sure to try loading the docker-archive first to account for the
"hybrid tarballs" created by buildkit.  docker-archive gets the name
right while oci-archive will not.

Fixes: github.com/containers/podman/pull/11619
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-09-30 14:15:51 +02:00
Valentin Rothberg 3dc26d743a libimage: use faster json library
Use github.com/json-iterator/go instead of the JSON library.
It is faster and battle tested.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2021-09-29 14:18:01 +02:00