... to prevent panics if the value does not contain a :, or other unexpected
values (e.g. a path traversal).
Don't bother on paths where we computed the digest ourselves, or it is already trusted
for other reasons.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We need something to wrap (format ID, actual data).
And a shared helper for a file/blob representation for things like
dir or c/storage.
This looks quite immature.
Tests of the non-simple signing will only be added with support for that format.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This sets up the precedent that all transports should primarily implement
the private interface; that will allow us to make future changes to the
private interface easier, because we can just change the public interface
wrappers in a single place instead of modifying transports - especially
as more stubs are added soonish.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This sets up the precedent that all transports should primarily implement
the private interface; that will allow us to make future changes to the
private interface easier, because we can just change the public interface
wrappers in a single place instead of modifying transports - especially
as more stubs are added soonish.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This means we won't save the stack, which is cheaper
(and possibly might break callers' format strings that
want to print the stack, but we never promised the stack
to be available).
Use either fmt.Errorf, or errors.New (usually as a local
edit, not carring about errors.new vs. pkg/errors.New;
that's going to be cleaned up later).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Mostly just name changes that should not change behavior,
apart from ioutil.ReadDir -> os.ReadDir avoiding per-item
lstat(2) in some cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Introduce internal/putblobdigest.Digester to encapsulate
the two alternatives, so that the PutBlob implementations only
need to plug it in.
Then use it throughout: let PutBlob use caller-provided digest
values, if any (and they use the right algorithm).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Use a different manifest and manifest list
- Write per-arch manifest before the top-level one
- Same for signatures
- Don't write the top-level manifest twice
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... and acknowledge that various tests are strictly speaking
invalid, to reinforce that real callers must not pass nil.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Add the manifest.List interface, and implementations for OCIv1 Index and
Docker Schema2List documents.
Add an instanceDigest parameter to PutManifest(), PutSignatures(), and
LayerInfosForCopy, for symmetry with GetManifest() and GetSignatures().
Return an error if the instanceDigest is supplied to destinations which
don't support them, and add stubs that do so even to the transports
which would support it, so that we don't break compilation here.
Add a MultipleImages flag to copy.Options, and if the source for a copy
operation contains multiple images, copy all of the images if we can.
If we can't copy them all, but we were told to, return an error.
Use the generic manifest list API to select a single image to copy from
a list, so that we aren't just limited to the Docker manifest list
format for those cases.
When guessing at the type of a manifest, if the manifest contains a list
of manifests, use its declared MIME type if it included one, else assume
it's an OCI index, because an OCI index doesn't include its MIME type.
When copying, switch from using an encode-then-compare of the original
and updated versions of the list to checking if the instance list was
changed (one of the things we might have changed) or if its type has
changed due to conversion (the other change we might have made). If
neither has changed, then we don't need to change the encoded value of
the manifest.
When copying, when checking for a digest mismatch in a target image
reference, ignore a mismatch between the digest in the reference and the
digest of the main manifest if we're copying one element from a list,
and the digest in the reference matches the digest of the manifest list.
When copying, if conversion of manifests for single images is being
forced, convert manifest lists to the corresponding list types.
When copying, supply the unparsed top level to Commit() by attaching the
value to the context.Context.
Support manifest lists in the directory transport by using the instance
digest as a prefix of the filename used to store a manifest or a piece
of signature data.
Support manifest lists in the oci-layout transport by accepting indexes
as we do images, and stop guessing about Platform values to add to the
top-level index.
Support storing manifest lists to registries in the docker: transport by
using the manifest digest when we're writing one image as part of
pushing a list of them, and by using the instance digest when reading or
writing signature data, when one is specified, or the cached digest of
the non-instanced digest when one is not specified.
Add partial support for manifest lists to the storage transport: when
committing one image from a list into storage, also add a copy of the
manifest list by extracting it from the context.Context. The logic is
already in place to enable locating an image using any of multiple
manifest digests.
When writing an image that has an instanceDigest value (meaning it's a
secondary image), don't try to generate a canonical reference to add to
the image's list of names if the reference for the primary image doesn't
contain a name. That should only happen if we're writing using just an
image ID, which is unlikely, but we still need to handle it.
Avoid computing the digest of the manifest, or retrieving the
either-a-tag-or-a-digest value from the target reference, if we're given
an instanceDigest, which would override them anyway.
Move the check for non-nil instanceDigest values up into the main
PutSignatures() method instead of duplicating it in the per-strategy
helpers.
Add mention of the instanceDigest parameter and its use to various
PutManifest, PutSignatures, and LayerInfosForCopy implementations and
their declarations in interfaces.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows consumers who don't need the defaulting logic to pick
their desired implementation without pulling in unrelated
dependencies. For example, it allows you to consume the memory or
NoCache implementations without pulling in github.com/boltdb.
I'd initially included blobinfocache wrappers to preserve backwards
compatibility with the old API while consumers updated to the new
APIs, but Miloslav said we don't have any backwards-compatibility
commitments at the moment [1].
Having single-implementation subpackages also allows for more generic
names (e.g. NewMemoryCache is now just New), because memory.New is
clear enough while memory.NewMemoryCache stutters.
[1]: https://github.com/containers/image/pull/602#issuecomment-471570789
Signed-off-by: W. Trevor King <wking@tremily.us>
For now, none of the transports actually use it, so should not change behavior.
copy.Image uses its existing cache object; config parses in image/* usually
use NoCache because they wouldn't appreciably benefit anyway.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Network IO paths should react to cancels now.
- File IO paths generally still won't.
- `SystemContext` objects have been renamed to `sys` to leave `ctx`
available for the stdlib context objects.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
This mirrors GetManifest, and allows / requires signatures to be per-instance.
Also add implementations in docker: and atomic:, the only transports which
support both manifest lists and signatures.
Does not change behavior yet, the only user always specifies nil
instanceDigest.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will make the code paths more uniform for consumers of the
primary manifest and the manifest instances.
(Having an explicit support for manifest instances is necessary
for transports like docker-daemon: / oci-archive:, which
contain several images but setting up an ImageSource is very
expensive, or which don't even allow referencing images by digest.)
This is a direct replacement of GetTargetManifest, and should
not change behavior; notably the OCI implementation is still
blindly guessing the manifest type although it is probably
available in the index.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The requestedManifestMIMETypes parameter was added because a destination
might not support all manifest MIME types that the the source supports,
but the original use case now passes all manifest types and lets
containers/image convert internally. In generally, internal conversion
may be more comprehensive, is more predictable, and avoids bypassing
internal checks.
Fixes: #331
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
This allows generic code to differentiate between transports where
bandwidth and storage costs are at premium (usually remote
repositories), and where memory/mmap and CPU time are more important
(usually local, especially temporary, storage) [or the dir: case
where we don’t want to modify the input at all if possible].
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This does not change behavior. Having a single struct for carrying
the (digest, size) pair around will make manipulating layers much
easier in the future.
... except that dockerImageDestination now logs the uploaded layer using
the realized digest, instead of the input digest = an empty string if the digest is
unknown.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows copy.go to bail early, without transferring any large blobs.
(This is an attribute of an ImageDestination and not an ImageTransport
because supporting signatures may depend on the individual destination,
e.g. depending on server version or whether a sigstore lookaside has
been configured for that destination.)
Also reorders the ImageDestination methods in some cases, to separate
basic management / capability query / upload groups.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This marks the image saving as "done". This is a separate concept from
ImageDestination.Close, which is called whenever the destination
should no longer be used, even if the caller has given up on the upload
process.
The separation between Close and Commit alllows a natural, simple flow:
> dest := ref.NewDestination()
> defer dest.Close()
> ... // do anything, simply return on error
> dest.Commit()
whereas a single Close would be more difficult to use:
> dest := ref.NewDestination()
> succeeded := false
> defer func() {
> dest.Close(succeeded)
> }()
> ... // do anything, simply return on error
> succeeded = true
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Submit this size in docker: destinations, validate it in dir: and oci:.
Preserve the size, if provided by ImageSource, when copying blobs.
Also document that PutBlob must be called before PutManifest and
PutSignatures; this is clearly the necessary ordering if the layer
tarballs are being generated on the fly, or if the manifest destination
wants to reject manifests referring to nonexistent blobs.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
For lookaside signature store, and separating the read and write URLs,
we need to set up read-only and read-write states differently; having
read-write “delete” in dockerImageSource is incovenient.
In tue future, ImageSource.Delete will be a really poor fit for
docker-daemon:, where initializing the ImageSource causes the tarball
to be copied from the daemon. We could instead implement the
docker-daemon source so that it only copies the tarball on demand, but
not sharing the object is much simpler.
This leaves the Docker implementation in docker_image_src.go to make
reviewing easier.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows the selection to be consistent across GetManifest and
GetSignatures (which will be needed by Docker lookaside).
The API change causes lots of churn, but ultimately it just moves the
real origin of the value from image.FromSource() to transport.NewImageSource(),
both of which are static for the life of the ImageSource.
Does not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... instead of Docker-specific certPath and tlsVerify.
Also invert the sense of tlsVerify to make the default secure.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is necessary to prevent using layers which have been tampered with.
To implement this, dir: and oci: now store data into temporary files,
and Rename() when all data is successfully copied and validated.
We do not do anything extra for docker:, hoping that the mandatory
digest= parameter in the API call is sufficient.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This also removes the error return value, only supporting nil
to indicate "Docker references not supported"; other kinds of
errors should have been handled when initially parsing or
creating the reference.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This minimizes transport-specific knowledge in image name parsing
(as in cmd/skopeo/utils.go) and allows separation of reference parsing
and their use.
Existing public NewImage... API has been removed; callers are expected
to use any of
* types.ImageTransport.ParseReference().NewImage...
(if they have a general string)
* transportpackage.ParseReference().NewImage...
(if they have a transport-specific string)
* transportpackage.NewReference().NewImage...
(if they have transport-specific raw values)
This usually adds an extra error checking step for the
ParseReference/NewReference call compared to the previous code; this is
considered not a big loss, especially because reporting “the reference
is invalid” and “the reference looks valid but connecting/using it
failed” as distinct failure modes seems quite useful for users.
The references are currently one-way (you can get a types.Image* from an
ImageReference, but not the other way around); that will be fixed soon.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is somewhat better typed, and avoids unnecessary roundtrips using
strings when both the producer and consumer want a reference.Named value
(like in PolicyContext.requirementsForImage).
This also forces us to explicitly handle IntendedDockerReference()
returning nil, when before we could rely on it returning "", which would
then be rejected by reference.ParseNamed as invalid input; anyway,
handling that case specially just allows for better error messages.
This adds two FIXMEs about error messages which do not tell the user
which image is being rejected; that will be fixed in the future
generalized reference work.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Miloslav Trmač