... to prevent panics if the value does not contain a :, or other unexpected
values (e.g. a path traversal).
Don't bother on paths where we computed the digest ourselves, or it is already trusted
for other reasons.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
TryReusingBlob now contains a new option `RequiredCompression` which
filters the blob by checking against a compression of the blob which is
considerd to be resued, in case `RequiredCompression` is set and `info`
of the blob being reused does not matches, no blob is returned.
Signed-off-by: Aditya R <arajan@redhat.com>
This is a subset of types.BlobInfo the transports
actually should deal with, to be much more explicit
about what the transports are responsible for.
Should not change behavior, in practice.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is a subset of types.BlobInfo the transports
actually should deal with. Also tighten the semantics a bit,
to be much more explicit about what the transports are responsible for.
This allows us to stop tracking MIME types in storage.addedLayerInfo,
so simplify that.
Then fix the generic code so that blob annotations are not discarded
on blob reuse.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
`github.com/pkg/errors` is deprecated since quite some time so we now
use the native error wrapping for more idiomatic golang.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
NOTE: This makes implementation choices:
- dir, docker LOOKASIDE, ostree and storage are going to store the
generic-format blobs, and can represent any future formats
of signatures.
- docker X-R-S-S and OpenShift only accept simple signatures,
and fail when copying the other kind.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
It's very annoying that Go allows silent zero-initilization,
and we have neither mandatory constructors nor parameter labels,
i.e. a forgotten field does not cause a compile-time error. For _private_
transport implementations, relying on human review is just about acceptable.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This sets up the precedent that all transports should primarily implement
the private interface; that will allow us to make future changes to the
private interface easier, because we can just change the public interface
wrappers in a single place instead of modifying transports - especially
as more stubs are added soonish.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This makes it clearer that dirImageDestination is only
used in a valid state.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will allow imports of errors and pkg/errors to
coexist in a single package, avoid adding accidental
new uses via errors.New(), make any potential new
additions more visible in reviews.
Files that can import errors only (typically
because they only use errors.New) have been updated to
that import instead of renaming.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This means we won't save the stack, which is cheaper
(and possibly might break callers' format strings that
want to print the stack, but we never promised the stack
to be available).
Use either fmt.Errorf, or errors.New (usually as a local
edit, not carring about errors.new vs. pkg/errors.New;
that's going to be cleaned up later).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Mostly just name changes that should not change behavior,
apart from ioutil.ReadDir -> os.ReadDir avoiding per-item
lstat(2) in some cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Introduce internal/putblobdigest.Digester to encapsulate
the two alternatives, so that the PutBlob implementations only
need to plug it in.
Then use it throughout: let PutBlob use caller-provided digest
values, if any (and they use the right algorithm).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Mostly remove some extra variables, instead modify stream
by wrapping it in io.TeeReader()s.
In dockerImageDestination, move more of the pipeline
construction outside of the uploadReader closure, and
use consecutiveio.TeeReader()s instead of io.MultiWriter.
Should not change (observable) behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This was always sort of implied, and we now rely on that explicitly
to avoid computing the digests; it turns out the digest computation
can consume a very noticeable amount of CPU time.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Podman and other tools already add Error: to the front of returned error
message, and this ends up as a stutter.
podman pull fedora.io/fred
Trying to pull fedora.io/fred:latest...
Error: Error initializing image from source docker://fedora.io/fred:latest: invalid character '<' looking for beginning of value
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
... and acknowledge that various tests are strictly speaking
invalid, to reinforce that real callers must not pass nil.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Extend the blob info cache to also cache the name of the type of
compression used on a blob that we've seen, or specific values that
indicate that we know the blob was not compressed, or that we don't
know whether or not it was compressed.
New methods for adding known blob-compression pairs and reading
candidate locations including compression information are part of a new
internal BlobInfoCache2 interface which the library's BlobInfoCache
implementors also implement.
When we copy a blob, try to record the state of compression for the
source blob, and if we applied any changes, the blob we produced.
Make sure that when TryReusingBlob successfully uses a blob from the
blob info cache, that it provides compression information in the
BlobInfo that it returns, so that manifests can be updated to describe
layers using the correct MIME types.
When attempting to write a manifest, if a manifest can't be written
because layers were compressed using an algorithm which can't be
expressed using that manifest type, continue on to trying other manifest
formats.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This has arguably been implied (OTOH, also arguably, it's a breaking change),
make it explicit.
This does not yet implement the semantics.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add the manifest.List interface, and implementations for OCIv1 Index and
Docker Schema2List documents.
Add an instanceDigest parameter to PutManifest(), PutSignatures(), and
LayerInfosForCopy, for symmetry with GetManifest() and GetSignatures().
Return an error if the instanceDigest is supplied to destinations which
don't support them, and add stubs that do so even to the transports
which would support it, so that we don't break compilation here.
Add a MultipleImages flag to copy.Options, and if the source for a copy
operation contains multiple images, copy all of the images if we can.
If we can't copy them all, but we were told to, return an error.
Use the generic manifest list API to select a single image to copy from
a list, so that we aren't just limited to the Docker manifest list
format for those cases.
When guessing at the type of a manifest, if the manifest contains a list
of manifests, use its declared MIME type if it included one, else assume
it's an OCI index, because an OCI index doesn't include its MIME type.
When copying, switch from using an encode-then-compare of the original
and updated versions of the list to checking if the instance list was
changed (one of the things we might have changed) or if its type has
changed due to conversion (the other change we might have made). If
neither has changed, then we don't need to change the encoded value of
the manifest.
When copying, when checking for a digest mismatch in a target image
reference, ignore a mismatch between the digest in the reference and the
digest of the main manifest if we're copying one element from a list,
and the digest in the reference matches the digest of the manifest list.
When copying, if conversion of manifests for single images is being
forced, convert manifest lists to the corresponding list types.
When copying, supply the unparsed top level to Commit() by attaching the
value to the context.Context.
Support manifest lists in the directory transport by using the instance
digest as a prefix of the filename used to store a manifest or a piece
of signature data.
Support manifest lists in the oci-layout transport by accepting indexes
as we do images, and stop guessing about Platform values to add to the
top-level index.
Support storing manifest lists to registries in the docker: transport by
using the manifest digest when we're writing one image as part of
pushing a list of them, and by using the instance digest when reading or
writing signature data, when one is specified, or the cached digest of
the non-instanced digest when one is not specified.
Add partial support for manifest lists to the storage transport: when
committing one image from a list into storage, also add a copy of the
manifest list by extracting it from the context.Context. The logic is
already in place to enable locating an image using any of multiple
manifest digests.
When writing an image that has an instanceDigest value (meaning it's a
secondary image), don't try to generate a canonical reference to add to
the image's list of names if the reference for the primary image doesn't
contain a name. That should only happen if we're writing using just an
image ID, which is unlikely, but we still need to handle it.
Avoid computing the digest of the manifest, or retrieving the
either-a-tag-or-a-digest value from the target reference, if we're given
an instanceDigest, which would override them anyway.
Move the check for non-nil instanceDigest values up into the main
PutSignatures() method instead of duplicating it in the per-strategy
helpers.
Add mention of the instanceDigest parameter and its use to various
PutManifest, PutSignatures, and LayerInfosForCopy implementations and
their declarations in interfaces.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add a HasThreadSafePutBlob() method to the ImageDestination interface
and all its implementations to indicate whether the corresponding
PutBlob() method can be executed concurrently. This is a first step to
enable parallel image copying. By default, all transports are not
thread-safe and must be carefully migrated in later changes.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This will allow TryReusingBlob to substitute the required blob with an equivalent
based on cache information if possible, or not doing so if it would break signatures.
In addition, we set canSubstitute to false when signing an image, to make 100% sure
the signed blob is safe (e.g. that we are not signing a third-party-compressed version
which has been maliciously designed to attack some decompressor implementations).
Nothing implements this, so does not change behavior yet.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
For now, none of the transports actually use it, so should not change behavior.
copy.Image uses its existing cache object; config parses in image/* usually
use NoCache because they wouldn't appreciably benefit anyway.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will, primarily, make it easier for the transport to use
alternate locations without having to somehow carry state from
HasBlob to ReapplyBlob.
Also, all instances of ReapplyBlob have been either trivial or redundant,
and there is a single primary caller of HasBlob/ReapplyBlob (+ a few
in-transport users who are even a bit cleaner with the move to
TryReusingBlob).
Should not change behavior, apart from not doing the
storageImageDestination.HasBlob check redundantly in
storageImageDestination.ReapplyBlob.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows a destination to opt out of updating the embedded name:tag in schema1
manifests without violating the ImageDestination.Reference / ImageReference API
expectations.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
It would be nice if those could be canceled, although so far we seem not
to have any users which would benefit.
Many would be easily handled with a cancelable variant of io.Copy,
but a few might be much more involved.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Network IO paths should react to cancels now.
- File IO paths generally still won't.
- `SystemContext` objects have been renamed to `sys` to leave `ctx`
available for the stdlib context objects.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
The config file was being saved as digest.tar for the directory transport.
This is misleading as the config file is not a tar archive.
Dropped the .tar extension from all files now, including blobs.
The user can use a tool like file to determine the format of the files in the directory.
Signed-off-by: umohnani8 <umohnani@redhat.com>
Miloslav Trmač