Podman and other tools already add Error: to the front of returned error
message, and this ends up as a stutter.
podman pull fedora.io/fred
Trying to pull fedora.io/fred:latest...
Error: Error initializing image from source docker://fedora.io/fred:latest: invalid character '<' looking for beginning of value
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add DockerLogMirrorChoice to types.SystemContext for keeping log level of the physical pull source of images.
Crio could set it log the image source if it's from a mirror.
Signed-off-by: Qi Wang <qiwan@redhat.com>
While container/image is not affected by CVE-2020-15157 future
generations may wonder why authentication is turned off on additional
URLs provided by certain blobs.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Set default rootless sigstore to ~/.local/share/containers/sigstore if the caller is non-root.
Export the func ConfiguredSignatureStorageBase() for Podman image sign implementation.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Format the error message use the response body if the original error is returned as client.UnexpectedHTTPResponseError type
Signed-off-by: Qi Wang <qiwan@redhat.com>
Often, manifests fail to pull because of a MIME type mismatch and have
to be debugged manually to figure out what the server/proxy is sending
the client. This is much easier to see when it's just printed for you.
Signed-off-by: Robb Manes <rmanes@redhat.com>
Restrict the sizes of blobs which are copied into memory such as the
manifest, the config, signatures, etc. This will protect consumers of
c/image from rogue or hijacked registries that return too big blobs in
hope to cause an OOM DOS attack.
Note that error message should be improved in a future change to make
sure that it's clear in which code path we hit a limit.
Fixes: CVE-2020-1702
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1792796
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that the debug log has timely information about
in-progress pulls, instead of only summarizing it at the end.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We don't say "pull" elsewhere, and it is a bad match (consider
(skopeo inspect) or a cross-registry copy).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Don't change the error format when no mirrors are involved, but record
all errors when they are.
An example error:
(First tried mirrors: [docker.io/library/busybox:thisdoesnotexist: Error reading manifest thisdoesnotexist in docker.io/library/busybox: manifest unknown: manifest unknown]): does-not-exist-primary.example.com/library/busybox:thisdoesnotexist: error pinging docker registry does-not-exist-primary.example.com: Get https://does-not-exist-primary.example.com/v2/: dial tcp: lookup does-not-exist-primary.example.com on 192.168.64.1:53: no such host
This could obviously be improved with various user-focused heuristics
(e.g. prefer mirror TLS/authentication errors to primary DNS not found),
or by dropping unnecessary context in common cases of the individual errors,
but we can tinker with that later.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... now that their scope is smaller and we don't need to disambiguate
so much or to be so explicit about their respective purposes.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
i.e. try the following mirrors.
The currently fatal cases are:
- newReference fails, i.e. input was completely invalid or :tag@digest.
That should never happen, but if it did, it could well be pullSource-specific.
- newDockerClientFromEnv fails, which can be several cases,
but notably the registry being blocked; in that case we should definitely
try another endpoint.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add the manifest.List interface, and implementations for OCIv1 Index and
Docker Schema2List documents.
Add an instanceDigest parameter to PutManifest(), PutSignatures(), and
LayerInfosForCopy, for symmetry with GetManifest() and GetSignatures().
Return an error if the instanceDigest is supplied to destinations which
don't support them, and add stubs that do so even to the transports
which would support it, so that we don't break compilation here.
Add a MultipleImages flag to copy.Options, and if the source for a copy
operation contains multiple images, copy all of the images if we can.
If we can't copy them all, but we were told to, return an error.
Use the generic manifest list API to select a single image to copy from
a list, so that we aren't just limited to the Docker manifest list
format for those cases.
When guessing at the type of a manifest, if the manifest contains a list
of manifests, use its declared MIME type if it included one, else assume
it's an OCI index, because an OCI index doesn't include its MIME type.
When copying, switch from using an encode-then-compare of the original
and updated versions of the list to checking if the instance list was
changed (one of the things we might have changed) or if its type has
changed due to conversion (the other change we might have made). If
neither has changed, then we don't need to change the encoded value of
the manifest.
When copying, when checking for a digest mismatch in a target image
reference, ignore a mismatch between the digest in the reference and the
digest of the main manifest if we're copying one element from a list,
and the digest in the reference matches the digest of the manifest list.
When copying, if conversion of manifests for single images is being
forced, convert manifest lists to the corresponding list types.
When copying, supply the unparsed top level to Commit() by attaching the
value to the context.Context.
Support manifest lists in the directory transport by using the instance
digest as a prefix of the filename used to store a manifest or a piece
of signature data.
Support manifest lists in the oci-layout transport by accepting indexes
as we do images, and stop guessing about Platform values to add to the
top-level index.
Support storing manifest lists to registries in the docker: transport by
using the manifest digest when we're writing one image as part of
pushing a list of them, and by using the instance digest when reading or
writing signature data, when one is specified, or the cached digest of
the non-instanced digest when one is not specified.
Add partial support for manifest lists to the storage transport: when
committing one image from a list into storage, also add a copy of the
manifest list by extracting it from the context.Context. The logic is
already in place to enable locating an image using any of multiple
manifest digests.
When writing an image that has an instanceDigest value (meaning it's a
secondary image), don't try to generate a canonical reference to add to
the image's list of names if the reference for the primary image doesn't
contain a name. That should only happen if we're writing using just an
image ID, which is unlikely, but we still need to handle it.
Avoid computing the digest of the manifest, or retrieving the
either-a-tag-or-a-digest value from the target reference, if we're given
an instanceDigest, which would override them anyway.
Move the check for non-nil instanceDigest values up into the main
PutSignatures() method instead of duplicating it in the per-strategy
helpers.
Add mention of the instanceDigest parameter and its use to various
PutManifest, PutSignatures, and LayerInfosForCopy implementations and
their declarations in interfaces.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Consolidate checking the http-status codes to allow for a more uniform
error handling. Also treat code 429 (too many requests) as a known
error instead of an invalid status code.
When hitting 429, perform an exponential back off starting a 2 seconds
for at most 5 iterations. If the http.Response set the `Retry-Header`
then use the provided value or date to compute the delay until the
next attempt. Note that the maximum delay is 60 seconds.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Revert the public docstring to only say what the caller needs to know.
Drop most of the contents entirely, as this should be clear from the
code or from containers-registries.conf.5.md; move a bit inside
the function.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add a new mirror-by-digest-only attribute to the registries configuration.
If set true, mirrors will only be consulted when the passed reference
includes a digest.
To facilitate using the API, add a new method
`PullSourcesFromReference(reference.Named) ([]PullSource, Error)`.
The new `PullSource` type contains an `Endpoint` and a written
reference. `Endpoint.RewriteReference(...)...` is now private.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This can never happen with the current code, but making sure that will
continue to be the case in the future does not cost any extra lines.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The control flow is slightly less clear because the success
path is not on the very bottom, OTOH we never return a
(failingUnusableImageSource, errorUsingIt) now.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is more consistent with how Registry values are normalized after parsing
from the configuration file, and removes the need for a prefix == "" special case
in RewriteReference.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This commit enables private registry mirror support for the docker
client. Major changes and additions related to an image pull are:
- If mirrors are specified via the configuration (registries.conf), then
the first working (where the manifest is available remotely) mirror
will be used for the pull.
- The main registry will be the fallback if no mirror works.
- The insecure/skipVerify flags will be used from system context and
from the corresponding registry/mirror as fallback too.
- Configuring a prefix will be considered, too.
- Update documentation for registry mirror support within
`registries.conf` as well as the man page.
- Renamed the `url` to `location` within system_registries_v2
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
makeRequestToResolvedURL() can return {nil, err} for all
given urls. In that case, resp.Body can cause SIGSEGV
because resp is nil.
Signed-off-by: Arjun Sreedharan <arjun024@gmail.com>
Make the extra token scope a parameter instead of a struct member. This
allows a parallel execution of TryReusingBlob without the need to
serialize accesses.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
GetBlob() can now be executed concurrently. Let HasThreadSafeGetBlob()
return true to make use of it.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add a HasThreadSafeGetBlob() method to the ImageSource interface and all
its implementations to indicate whether the corresponding GetBlob()
method can be executed concurrently. This is a first step to enable
parallel image copying. By default, all transports are not thread-safe
and must be carefully migrated in later changes.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
For now, none of the transports actually use it, so should not change behavior.
copy.Image uses its existing cache object; config parses in image/* usually
use NoCache because they wouldn't appreciably benefit anyway.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
e.g. 504 (Gateway timeout) instead of just "504".
Note that for unknown status codes this will output "99999 ()"; this
seems not worth worrying about and building a helper for right now.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Change the order in which a search happens. Try the v1 endpoint
first without any authentication token and if that fails, get a
token and try the v2 endpoint.
Also add feature to search with an empty query, which only works
with the v2 endpoint. If the query is empty skip the v1 endpoint
and go straight to the v2 endpoint.
This helps fix issues we were seeing when searching quay.io and also
adds the ability to search a whole registry without a specific query
parameter.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
https://github.com/docker/distribution/blob/master/docs/spec/auth/scope.md
does not document the action necessary for deleting images.
For docker/distribution, we need "pull" to be able to read the manifest,
and "delete" to delete it. Until docker/distribution commit
ccb839e0e30c3b6992fb4084dfd6550d0ddd4d1a (Jan 3 2017), the action checked
for deleting manifests was "*".
For Quay.io, only "push" is sufficient for both; and although
https://github.com/docker/distribution/blob/master/docs/spec/auth/jwt.md
says that requesting permissions that the server does not grant should
not be an error, quay.io refuses to even parse a request which contains
a "delete" action and does not grant any token.
This should not affect OpenShift, where the token is an API
token and does not actually include any scopes or permissions;
(that is also probably why noone has noticed before).
Overall, "*" seems to be the only common action specification;
luckily both docker/distribution and Quay.io seem to treat it
as "all actions allowed" (which is _not_ documented in the spec,
but then again, nothing about actions for deleting images is
documented).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Network IO paths should react to cancels now.
- File IO paths generally still won't.
- `SystemContext` objects have been renamed to `sys` to leave `ctx`
available for the stdlib context objects.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
especially focus on those from client.HandleErrorResponse, which may
be very unhelpful, e.g. "unknown: Not Found".
In general, make sure the error includes an expanded Docker reference,
or a full URL, or a hostname + path.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add an LayerInfosForCopy() method to source images which gives them a
chance to provide updated values for the blobsums contained in the
image's manifest, if they want to. Returning `nil` implies that they
have no changes to suggest compared to what's in the manifest.
When copying an image, if we can update the manifest with those new
values during copying, do so. If we have new values, but we can't
update the manifest, copying fails.
Update storageImageSource to return its manifest and reference in
unmodified form, and supply updated blob digests via LayerInfosForCopy()
so that copying images from storage works.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This mirrors GetManifest, and allows / requires signatures to be per-instance.
Also add implementations in docker: and atomic:, the only transports which
support both manifest lists and signatures.
Does not change behavior yet, the only user always specifies nil
instanceDigest.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will make the code paths more uniform for consumers of the
primary manifest and the manifest instances.
(Having an explicit support for manifest instances is necessary
for transports like docker-daemon: / oci-archive:, which
contain several images but setting up an ImageSource is very
expensive, or which don't even allow referencing images by digest.)
This is a direct replacement of GetTargetManifest, and should
not change behavior; notably the OCI implementation is still
blindly guessing the manifest type although it is probably
available in the index.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This package is used in authenticating a user for kpod login
and can be used for authentication in kpod push, pull etc.
Signed-off-by: umohnani8 <umohnani@redhat.com>
This has two impacts:
1. In the case where a request succeeds, break out of the loop so as not to
continue fetching unnecessary blobs.
2. Previously it was possible for a request to fail after succeeding in a
previous iteration, which would error the function, despite a successful
fetch.
Signed-off-by: Will Martin <wmartin@pivotal.io>
The requestedManifestMIMETypes parameter was added because a destination
might not support all manifest MIME types that the the source supports,
but the original use case now passes all manifest types and lets
containers/image convert internally. In generally, internal conversion
may be more comprehensive, is more predictable, and avoids bypassing
internal checks.
Fixes: containers/image#331
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
If we are supplied a digested reference, use it instead of possibly
fetching the manifest. This allows using signatures which are not
protected even if the manifest is, in some situations; in ordinary
operation accessing the whole image, this does not make any difference
because we are fetching the manifest either way.
To ensure consistent behavior, only use the reference digest if it uses
the digest.Canonical algorithm, which has been used so far. This could
be perhaps relaxed in the future.
Note that this does not promise that GetSignatures will never need
authentication—e.g. for X-Registry-Supports-Signatures this already
depends on the server, and for the sigstore lookaside authentication
support may be added in the future.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is primarily the only documentation of the sigstore layout;
in addition it comments on the OpenShift API master REST API and the
OpenShift docker/distribution API extension.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is provided by the OpenShift-integrated registry. This is
equivalent to the atomic: transport (in the “openshift”) subpackage, but
it requires less code and notably does not require an OpenShift login
context to be configured.
See https://github.com/openshift/origin/pull/12504 and
https://github.com/openshift/openshift-docs/pull/3556 for more
information on this API extension.
To preserve compatibility, we always check for a configured lookaside
sigstore first; if that is set up, we use the lookaside and ignore the
registry-native signature storage. Usually the user would not bother to
set up the lookaside, and use the native mechanism.
The code is mostly trivial; the only non-obvious aspect is the loop in
putSignaturesToAPIExtension, which is a pretty direct translation of
openshiftImageDestination.PutSignatures.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will allow using paths starting with /extensions/v2/ in the future.
Also, we can now name the parameter “path” instead of the incorrect
“url”, and the “path” name actually matches (there is no magic
/v2/ adding involved).
Also eliminates duplication of the ping paths (in logging code only).
Does not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows us to provide in the image interfaces a method of providing
an error at close time. This is only currently used in a few situations.
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
This replaces the copy of github.com/docker/docker/reference in the same
place, which we have just gotten rid of, and allows using this package
even in consumers which insist on an incompatible version of
docker/distribution.
The copy has been edited to drop a reference to
github.com/docker/distribution/digestset .
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is an intermediate step which will eventually go away.
The goal of this PR is to get rid of c/i/docker/daemon/reference and to
replace uses of it by direct calls to docker/distribution/reference.
We can't do that safely and easily, because the two have different
semantics for reference.Named.Name() and reference.Named.String(): we
return a minimized version, e.g. "busybox", upstream returns an expanded
version, e.g. "docker.io/library/busybox".
BEFORE this commit the difference is hidden by using
docker/distribution/reference.WithName, which allows using the minimized
version, and works with it correctly; but because we want to use the
upstream canonicalization code, which will change semantics, we can't
just mix and match.
To make the distinction explicit, this commmit adds an X to ALL public
names from c/i/docker/daemon/reference. E.g. a reference.XNamed type,
which has methods XName and XString.
This is pretty large, but does not change behavior at all. By
inspection it is clear to see that reference.XNamed and subtypes does
not expose any of the non-X, conflicting, method names.
Using e.g.
> git diff --word-diff-regex=.|grep -F '{+'|grep -v '^\([^{]\|{+X+}\)*{\?$'
it is possible to see that most lines in this diff only add a single X
letter, and manually inspect the few lines which don't match the regexp.
The only REALLY new code is an explicit definition of namedRef.XName()
and namedRef.XString(), and two newly added casts to namedRef in cases
where we need to use the underlying distreference.Reference within
a reference.XNamed value. Strictly speaking these changes change
behavior, in that third-party implementations of reference.XNamed are no
longer accepted; but we broke them by renaming at all.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
golint nowadays rejects it with
> error strings should not be capitalized or end with punctuation or a newline
(… but it accepts leading capitalization anyway, so this does not change
any of the many instances of that.)
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
https://github.com/containers/image/pull/139 has changed the semantics
to always expect a value from an ImageSource; that’s not really great
when the source has no idea, but we don’t have optionals in Golang to
distinguish between a nonsensical "" and unavailable, so this is at
least making the documentation consistent.
Hopefully ImageSources which call manifest.GuessMIMEType in GetManifest
will not proliferate…
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
* It's quite useful to have error types exported so that libraries
can avoid substring matchers
Signed-off-by: Claudia Beresford <cberesford@pivotal.io>
Allows configuring "sigstore" (for RW) and "sigstore-write" (write-only)
for images/repositories/namespaces/registries in
/etc/containers/repositories.d/*.yaml .
If configured, uses them to store signatures in the docker: transport.
Also includes documentation in docs/registries.d.md.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Use -1 because 0 is, technically, a valid size of a blob.
Also remove a FIXME about (docker save) tarballs, now we know that we
will need to make a temporary, seekable, copy.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
For lookaside signature store, and separating the read and write URLs,
we need to set up read-only and read-write states differently; having
read-write “delete” in dockerImageSource is incovenient.
In tue future, ImageSource.Delete will be a really poor fit for
docker-daemon:, where initializing the ImageSource causes the tarball
to be copied from the daemon. We could instead implement the
docker-daemon source so that it only copies the tarball on demand, but
not sharing the object is much simpler.
This leaves the Docker implementation in docker_image_src.go to make
reviewing easier.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows the selection to be consistent across GetManifest and
GetSignatures (which will be needed by Docker lookaside).
The API change causes lots of churn, but ultimately it just moves the
real origin of the value from image.FromSource() to transport.NewImageSource(),
both of which are static for the life of the ImageSource.
Does not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... instead of Docker-specific certPath and tlsVerify.
Also invert the sense of tlsVerify to make the default secure.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This improves the dockerReference encapsulation, and makes it more clear
that the reference.IsNameOnly case can not happen.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This minimizes transport-specific knowledge in image name parsing
(as in cmd/skopeo/utils.go) and allows separation of reference parsing
and their use.
Existing public NewImage... API has been removed; callers are expected
to use any of
* types.ImageTransport.ParseReference().NewImage...
(if they have a general string)
* transportpackage.ParseReference().NewImage...
(if they have a transport-specific string)
* transportpackage.NewReference().NewImage...
(if they have transport-specific raw values)
This usually adds an extra error checking step for the
ParseReference/NewReference call compared to the previous code; this is
considered not a big loss, especially because reporting “the reference
is invalid” and “the reference looks valid but connecting/using it
failed” as distinct failure modes seems quite useful for users.
The references are currently one-way (you can get a types.Image* from an
ImageReference, but not the other way around); that will be fixed soon.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is somewhat better typed, and avoids unnecessary roundtrips using
strings when both the producer and consumer want a reference.Named value
(like in PolicyContext.requirementsForImage).
This also forces us to explicitly handle IntendedDockerReference()
returning nil, when before we could rely on it returning "", which would
then be rejected by reference.ParseNamed as invalid input; anyway,
handling that case specially just allows for better error messages.
This adds two FIXMEs about error messages which do not tell the user
which image is being rejected; that will be fixed in the future
generalized reference work.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The tag is a derived value from the reference, so storing it separately
is redundant.
More importantly, this also modifies CanonicalDockerReference and
IntendedDockerReference to call the reference.Named.String() method for
formatting, so that we no longer corrupt
busybox@sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
to busybox:sha256:a599…
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
OpenShift returns "application/json; charset=utf-8"; with greatest
respect to the strict need to care about character encoding, we don't
want to get into the business of implementing support for charset
encoding, so just drop that on the floor.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Giuseppe Scrivano
Daniel J Walsh
Valentin Rothberg
xiaoqing
Qi Wang
Robb Manes
Miloslav Trmač
knqyf263
Nalin Dahyabhai
Sascha Grunert
Arjun Sreedharan
Urvashi Mohnani
Mike Lundy
Tiago Scolari
Owen W. Taylor
Mrunal Patel
Michal Fojtik
Erik Hollensbe
Antonio Murdaca
George Lestaris
Crazykev
Alberto Leal