CloudFront returns an unquoted ':' in the boundary, special handle it.
It is needed for partial pulls from quay.io.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Per https://github.com/containers/image/issues/1010 , it seems
the header is not populated by AWS ECR.
We were actually computing the digest from the manifest bytes
already, so this is both more robust and simpler.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Mostly just name changes that should not change behavior,
apart from ioutil.ReadDir -> os.ReadDir avoiding per-item
lstat(2) in some cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... to be at least a bit protected against credentials in logs.
I did try to find all uses, but it's possible I have missed some.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is, sadly, wasteful, because NewRequestWithContext() only accepts
a string and parses it again, but it gives us more type safety, and simplifies
at least some callers.
Most importantly, this will also allow us to call url.Redacted() for logging.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Following the now-established pattern, introduce private.ImageSource
(with a new SupportsGetBlobAt() method), implement it in
dockerImageSource, and use a wrapped version in c/image/copy.
This gives us private.ImageSource and private.ImageDestination,
to allow for future features.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The way we actually use this, injecting a progress-reporting
proxy, the PutBlobPartial method doesn't have access to the
full ImageSource; so, differentiate more clearly between
this small interface and an ImageSource.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The goal is to establish a practical/convenient naming for
internal-only interfaces, now that we are going to start using them
much more frequently.
Design concerns:
- Don't duplicate any public package name (to avoid the
publicTypes/internalTypes names in every single user).
Rules out "types", "image" (as in "image.Destination").
- Don't just use "internal", to keep that around for
very localized private utilities, like oci/internal
"private", as a more or less synonym to "internal", seems
to work acceptably well: we will have types.ImageDestination
and private.ImageDestination.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
No point in handling them in generic code, and callers that
aren't expecting StatusPartialContent could be confused.
Should not change behavior of GetBlobAt; might in change behavior of
other callers if they unexpectedly received the relevant HTTP status
codes.
This ~mechanically moves the code without much restructuring.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
ghcr.io converts a multirange request to a 200 response when the
client request too much data.
If the server replies with a 200 status to a partial request then
split the body ignoring any additional content that wasn't requested
by the caller.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
add inline documentation to clarify the expectation for the chunks
specified to the GetBlobAt() function.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
... just for the general principle of avoiding hard-coded
copy&pasted data.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Podman and other tools already add Error: to the front of returned error
message, and this ends up as a stutter.
podman pull fedora.io/fred
Trying to pull fedora.io/fred:latest...
Error: Error initializing image from source docker://fedora.io/fred:latest: invalid character '<' looking for beginning of value
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add DockerLogMirrorChoice to types.SystemContext for keeping log level of the physical pull source of images.
Crio could set it log the image source if it's from a mirror.
Signed-off-by: Qi Wang <qiwan@redhat.com>
While container/image is not affected by CVE-2020-15157 future
generations may wonder why authentication is turned off on additional
URLs provided by certain blobs.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Set default rootless sigstore to ~/.local/share/containers/sigstore if the caller is non-root.
Export the func ConfiguredSignatureStorageBase() for Podman image sign implementation.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Format the error message use the response body if the original error is returned as client.UnexpectedHTTPResponseError type
Signed-off-by: Qi Wang <qiwan@redhat.com>
Often, manifests fail to pull because of a MIME type mismatch and have
to be debugged manually to figure out what the server/proxy is sending
the client. This is much easier to see when it's just printed for you.
Signed-off-by: Robb Manes <rmanes@redhat.com>
Restrict the sizes of blobs which are copied into memory such as the
manifest, the config, signatures, etc. This will protect consumers of
c/image from rogue or hijacked registries that return too big blobs in
hope to cause an OOM DOS attack.
Note that error message should be improved in a future change to make
sure that it's clear in which code path we hit a limit.
Fixes: CVE-2020-1702
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1792796
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that the debug log has timely information about
in-progress pulls, instead of only summarizing it at the end.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We don't say "pull" elsewhere, and it is a bad match (consider
(skopeo inspect) or a cross-registry copy).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Don't change the error format when no mirrors are involved, but record
all errors when they are.
An example error:
(First tried mirrors: [docker.io/library/busybox:thisdoesnotexist: Error reading manifest thisdoesnotexist in docker.io/library/busybox: manifest unknown: manifest unknown]): does-not-exist-primary.example.com/library/busybox:thisdoesnotexist: error pinging docker registry does-not-exist-primary.example.com: Get https://does-not-exist-primary.example.com/v2/: dial tcp: lookup does-not-exist-primary.example.com on 192.168.64.1:53: no such host
This could obviously be improved with various user-focused heuristics
(e.g. prefer mirror TLS/authentication errors to primary DNS not found),
or by dropping unnecessary context in common cases of the individual errors,
but we can tinker with that later.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... now that their scope is smaller and we don't need to disambiguate
so much or to be so explicit about their respective purposes.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
i.e. try the following mirrors.
The currently fatal cases are:
- newReference fails, i.e. input was completely invalid or :tag@digest.
That should never happen, but if it did, it could well be pullSource-specific.
- newDockerClientFromEnv fails, which can be several cases,
but notably the registry being blocked; in that case we should definitely
try another endpoint.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add the manifest.List interface, and implementations for OCIv1 Index and
Docker Schema2List documents.
Add an instanceDigest parameter to PutManifest(), PutSignatures(), and
LayerInfosForCopy, for symmetry with GetManifest() and GetSignatures().
Return an error if the instanceDigest is supplied to destinations which
don't support them, and add stubs that do so even to the transports
which would support it, so that we don't break compilation here.
Add a MultipleImages flag to copy.Options, and if the source for a copy
operation contains multiple images, copy all of the images if we can.
If we can't copy them all, but we were told to, return an error.
Use the generic manifest list API to select a single image to copy from
a list, so that we aren't just limited to the Docker manifest list
format for those cases.
When guessing at the type of a manifest, if the manifest contains a list
of manifests, use its declared MIME type if it included one, else assume
it's an OCI index, because an OCI index doesn't include its MIME type.
When copying, switch from using an encode-then-compare of the original
and updated versions of the list to checking if the instance list was
changed (one of the things we might have changed) or if its type has
changed due to conversion (the other change we might have made). If
neither has changed, then we don't need to change the encoded value of
the manifest.
When copying, when checking for a digest mismatch in a target image
reference, ignore a mismatch between the digest in the reference and the
digest of the main manifest if we're copying one element from a list,
and the digest in the reference matches the digest of the manifest list.
When copying, if conversion of manifests for single images is being
forced, convert manifest lists to the corresponding list types.
When copying, supply the unparsed top level to Commit() by attaching the
value to the context.Context.
Support manifest lists in the directory transport by using the instance
digest as a prefix of the filename used to store a manifest or a piece
of signature data.
Support manifest lists in the oci-layout transport by accepting indexes
as we do images, and stop guessing about Platform values to add to the
top-level index.
Support storing manifest lists to registries in the docker: transport by
using the manifest digest when we're writing one image as part of
pushing a list of them, and by using the instance digest when reading or
writing signature data, when one is specified, or the cached digest of
the non-instanced digest when one is not specified.
Add partial support for manifest lists to the storage transport: when
committing one image from a list into storage, also add a copy of the
manifest list by extracting it from the context.Context. The logic is
already in place to enable locating an image using any of multiple
manifest digests.
When writing an image that has an instanceDigest value (meaning it's a
secondary image), don't try to generate a canonical reference to add to
the image's list of names if the reference for the primary image doesn't
contain a name. That should only happen if we're writing using just an
image ID, which is unlikely, but we still need to handle it.
Avoid computing the digest of the manifest, or retrieving the
either-a-tag-or-a-digest value from the target reference, if we're given
an instanceDigest, which would override them anyway.
Move the check for non-nil instanceDigest values up into the main
PutSignatures() method instead of duplicating it in the per-strategy
helpers.
Add mention of the instanceDigest parameter and its use to various
PutManifest, PutSignatures, and LayerInfosForCopy implementations and
their declarations in interfaces.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Consolidate checking the http-status codes to allow for a more uniform
error handling. Also treat code 429 (too many requests) as a known
error instead of an invalid status code.
When hitting 429, perform an exponential back off starting a 2 seconds
for at most 5 iterations. If the http.Response set the `Retry-Header`
then use the provided value or date to compute the delay until the
next attempt. Note that the maximum delay is 60 seconds.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Revert the public docstring to only say what the caller needs to know.
Drop most of the contents entirely, as this should be clear from the
code or from containers-registries.conf.5.md; move a bit inside
the function.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add a new mirror-by-digest-only attribute to the registries configuration.
If set true, mirrors will only be consulted when the passed reference
includes a digest.
To facilitate using the API, add a new method
`PullSourcesFromReference(reference.Named) ([]PullSource, Error)`.
The new `PullSource` type contains an `Endpoint` and a written
reference. `Endpoint.RewriteReference(...)...` is now private.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This can never happen with the current code, but making sure that will
continue to be the case in the future does not cost any extra lines.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Giuseppe Scrivano