Mostly just name changes that should not change behavior,
apart from ioutil.ReadDir -> os.ReadDir avoiding per-item
lstat(2) in some cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... to be at least a bit protected against credentials in logs.
I did try to find all uses, but it's possible I have missed some.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is, sadly, wasteful, because NewRequestWithContext() only accepts
a string and parses it again, but it gives us more type safety, and simplifies
at least some callers.
Most importantly, this will also allow us to call url.Redacted() for logging.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add internal/streamdigest package to stream layers to temporary files
for populating Digest and Size in BlobInfos and use in docker tarfile
and image dest. When precompute digests option is set, layers are
never uploaded that already exist on the destination registry, in
exchange for streaming layers to temporary files when digests are
unknown (ex. compressing "on the fly").
Signed-off-by: Paul Fisher <pfisher@lyft.com>
In the rare case where TryReusingBlob (is not called, or) doesn't find
a blob in the destination but the following PutBlob does, do record its
location in the BlobInfoCache.
To do that, split the relevant part of TryReusingBlob into a separate function
instead of calling all of TryReusingBlob with restrictive parameters (and in
particular, NoCache, which is why we were not recording the location).
This should currently only matter in the rare race of TryReusingBlob vs.
registry changes; https://github.com/containers/image/pull/1393 proposes
adding another case that would be much easier to hit.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Introduce internal/putblobdigest.Digester to encapsulate
the two alternatives, so that the PutBlob implementations only
need to plug it in.
Then use it throughout: let PutBlob use caller-provided digest
values, if any (and they use the right algorithm).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Mostly remove some extra variables, instead modify stream
by wrapping it in io.TeeReader()s.
In dockerImageDestination, move more of the pipeline
construction outside of the uploadReader closure, and
use consecutiveio.TeeReader()s instead of io.MultiWriter.
Should not change (observable) behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The "" value is not really a valid digest.Digest value anyway
(it fails Digest.Validate()), so comparing String() with "" is,
if anything, _less_ valid use of the API than comparing the raw
digest.Digest with "".
So, consistently use the shorter and easier-to-read version
throughout the codebase.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This was always sort of implied, and we now rely on that explicitly
to avoid computing the digests; it turns out the digest computation
can consume a very noticeable amount of CPU time.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... just for the general principle of avoiding hard-coded
copy&pasted data.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Podman and other tools already add Error: to the front of returned error
message, and this ends up as a stutter.
podman pull fedora.io/fred
Trying to pull fedora.io/fred:latest...
Error: Error initializing image from source docker://fedora.io/fred:latest: invalid character '<' looking for beginning of value
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
... and acknowledge that various tests are strictly speaking
invalid, to reinforce that real callers must not pass nil.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Extend the blob info cache to also cache the name of the type of
compression used on a blob that we've seen, or specific values that
indicate that we know the blob was not compressed, or that we don't
know whether or not it was compressed.
New methods for adding known blob-compression pairs and reading
candidate locations including compression information are part of a new
internal BlobInfoCache2 interface which the library's BlobInfoCache
implementors also implement.
When we copy a blob, try to record the state of compression for the
source blob, and if we applied any changes, the blob we produced.
Make sure that when TryReusingBlob successfully uses a blob from the
blob info cache, that it provides compression information in the
BlobInfo that it returns, so that manifests can be updated to describe
layers using the correct MIME types.
When attempting to write a manifest, if a manifest can't be written
because layers were compressed using an algorithm which can't be
expressed using that manifest type, continue on to trying other manifest
formats.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Set default rootless sigstore to ~/.local/share/containers/sigstore if the caller is non-root.
Export the func ConfiguredSignatureStorageBase() for Podman image sign implementation.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Format the error message use the response body if the original error is returned as client.UnexpectedHTTPResponseError type
Signed-off-by: Qi Wang <qiwan@redhat.com>
Unlike the typical instanceDigest, the value is not nil for top-level
manifests, so pass it by value and name it manifestDigest, consistent
with other uses of such a value.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Before, we would typically only fail with a
"http: no Location header in response" error, discarding the underlying
reason and confusing users about the server state.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Otherwise net/http might (regardless of returning from http.Client.Do,
and regardless of possibly canceling ctx) continue reading the input
stream from a separate goroutine.
This
- ensures digester.Digest() is safe to use; otherwise it might work on
an in-progress digester state, causing a "panic: d.nx != 0".
- ensures the source of the data is not called after PutBlob returns,
violating the expectations of the caller.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Restrict the sizes of blobs which are copied into memory such as the
manifest, the config, signatures, etc. This will protect consumers of
c/image from rogue or hijacked registries that return too big blobs in
hope to cause an OOM DOS attack.
Note that error message should be improved in a future change to make
sure that it's clear in which code path we hit a limit.
Fixes: CVE-2020-1702
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1792796
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This commit adds DockerDisableDestSchema1MIMETypes to types.Context
which, if set to true, will omit DockerV2Schema1SignedMediaType and
DockerV2Schema1MediaType from
dockerImageDestination.SupportedManifestMIMETypes.
The motivation for this is that newer versions of the docker registry
(2.7.1 and later) disable schema1 by default. If a user attempts to
copy a v2schema1 image into the newer registry, the result is a
500 error. With this change, when calling copy.Image(), setting
options.DestinationCtx.DockerDisableDestSchema1MIMETypes to true
will allow copying of a schema1 image to a destination registry with
schema1 disabled. The image will be converted to schema2.
Signed-off-by: Scott Seago <sseago@redhat.com>
This has arguably been implied (OTOH, also arguably, it's a breaking change),
make it explicit.
This does not yet implement the semantics.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add the manifest.List interface, and implementations for OCIv1 Index and
Docker Schema2List documents.
Add an instanceDigest parameter to PutManifest(), PutSignatures(), and
LayerInfosForCopy, for symmetry with GetManifest() and GetSignatures().
Return an error if the instanceDigest is supplied to destinations which
don't support them, and add stubs that do so even to the transports
which would support it, so that we don't break compilation here.
Add a MultipleImages flag to copy.Options, and if the source for a copy
operation contains multiple images, copy all of the images if we can.
If we can't copy them all, but we were told to, return an error.
Use the generic manifest list API to select a single image to copy from
a list, so that we aren't just limited to the Docker manifest list
format for those cases.
When guessing at the type of a manifest, if the manifest contains a list
of manifests, use its declared MIME type if it included one, else assume
it's an OCI index, because an OCI index doesn't include its MIME type.
When copying, switch from using an encode-then-compare of the original
and updated versions of the list to checking if the instance list was
changed (one of the things we might have changed) or if its type has
changed due to conversion (the other change we might have made). If
neither has changed, then we don't need to change the encoded value of
the manifest.
When copying, when checking for a digest mismatch in a target image
reference, ignore a mismatch between the digest in the reference and the
digest of the main manifest if we're copying one element from a list,
and the digest in the reference matches the digest of the manifest list.
When copying, if conversion of manifests for single images is being
forced, convert manifest lists to the corresponding list types.
When copying, supply the unparsed top level to Commit() by attaching the
value to the context.Context.
Support manifest lists in the directory transport by using the instance
digest as a prefix of the filename used to store a manifest or a piece
of signature data.
Support manifest lists in the oci-layout transport by accepting indexes
as we do images, and stop guessing about Platform values to add to the
top-level index.
Support storing manifest lists to registries in the docker: transport by
using the manifest digest when we're writing one image as part of
pushing a list of them, and by using the instance digest when reading or
writing signature data, when one is specified, or the cached digest of
the non-instanced digest when one is not specified.
Add partial support for manifest lists to the storage transport: when
committing one image from a list into storage, also add a copy of the
manifest list by extracting it from the context.Context. The logic is
already in place to enable locating an image using any of multiple
manifest digests.
When writing an image that has an instanceDigest value (meaning it's a
secondary image), don't try to generate a canonical reference to add to
the image's list of names if the reference for the primary image doesn't
contain a name. That should only happen if we're writing using just an
image ID, which is unlikely, but we still need to handle it.
Avoid computing the digest of the manifest, or retrieving the
either-a-tag-or-a-digest value from the target reference, if we're given
an instanceDigest, which would override them anyway.
Move the check for non-nil instanceDigest values up into the main
PutSignatures() method instead of duplicating it in the per-strategy
helpers.
Add mention of the instanceDigest parameter and its use to various
PutManifest, PutSignatures, and LayerInfosForCopy implementations and
their declarations in interfaces.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Consolidate checking the http-status codes to allow for a more uniform
error handling. Also treat code 429 (too many requests) as a known
error instead of an invalid status code.
When hitting 429, perform an exponential back off starting a 2 seconds
for at most 5 iterations. If the http.Response set the `Retry-Header`
then use the provided value or date to compute the delay until the
next attempt. Note that the maximum delay is 60 seconds.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows consumers who don't need the defaulting logic to pick
their desired implementation without pulling in unrelated
dependencies. For example, it allows you to consume the memory or
NoCache implementations without pulling in github.com/boltdb.
I'd initially included blobinfocache wrappers to preserve backwards
compatibility with the old API while consumers updated to the new
APIs, but Miloslav said we don't have any backwards-compatibility
commitments at the moment [1].
Having single-implementation subpackages also allows for more generic
names (e.g. NewMemoryCache is now just New), because memory.New is
clear enough while memory.NewMemoryCache stutters.
[1]: https://github.com/containers/image/pull/602#issuecomment-471570789
Signed-off-by: W. Trevor King <wking@tremily.us>
Uploading a spec-compliant OCI manifest which contains no top-level
media type fails on AWS ECR with an Unsupported error code and
an 'Invalid JSON syntax' error message. Catch this case as an
invalid manifest error so that we automatically fallback to other
accepted media types in copyOneImage().
Addresses libpod issue #1719
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Mark dockerImageDestination as thread safe to allow parallel copy
operations of an image's blobs to a registry.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Make the extra token scope a parameter instead of a struct member. This
allows a parallel execution of TryReusingBlob without the need to
serialize accesses.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Mark dockerImageDestination to not be thread safe as there is a race
condition in dockerClient.extraScope causing copy.Copy() to error out
with `[...]dockerClient.extraScope was set before TryReusingBlob`,
which must be addressed in future changes.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
PutBlob() can now be executed concurrently. Let HasThreadSafePutBlob()
return true to make use of it.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add a HasThreadSafePutBlob() method to the ImageDestination interface
and all its implementations to indicate whether the corresponding
PutBlob() method can be executed concurrently. This is a first step to
enable parallel image copying. By default, all transports are not
thread-safe and must be carefully migrated in later changes.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This will allow TryReusingBlob to substitute the required blob with an equivalent
based on cache information if possible, or not doing so if it would break signatures.
In addition, we set canSubstitute to false when signing an image, to make 100% sure
the signed blob is safe (e.g. that we are not signing a third-party-compressed version
which has been maliciously designed to attack some decompressor implementations).
Nothing implements this, so does not change behavior yet.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
For now, none of the transports actually use it, so should not change behavior.
copy.Image uses its existing cache object; config parses in image/* usually
use NoCache because they wouldn't appreciably benefit anyway.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will, primarily, make it easier for the transport to use
alternate locations without having to somehow carry state from
HasBlob to ReapplyBlob.
Also, all instances of ReapplyBlob have been either trivial or redundant,
and there is a single primary caller of HasBlob/ReapplyBlob (+ a few
in-transport users who are even a bit cleaner with the move to
TryReusingBlob).
Should not change behavior, apart from not doing the
storageImageDestination.HasBlob check redundantly in
storageImageDestination.ReapplyBlob.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
e.g. 504 (Gateway timeout) instead of just "504".
Note that for unknown status codes this will output "99999 ()"; this
seems not worth worrying about and building a helper for right now.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Change the order in which a search happens. Try the v1 endpoint
first without any authentication token and if that fails, get a
token and try the v2 endpoint.
Also add feature to search with an empty query, which only works
with the v2 endpoint. If the query is empty skip the v1 endpoint
and go straight to the v2 endpoint.
This helps fix issues we were seeing when searching quay.io and also
adds the ability to search a whole registry without a specific query
parameter.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
This allows a destination to opt out of updating the embedded name:tag in schema1
manifests without violating the ImageDestination.Reference / ImageReference API
expectations.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Miloslav Trmač