Now that all data for DockerReference comes from completeReference,
just return it directly, and eliminate the weird theoretical
failure paths where we can silently return incorrect data.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
They are mostly used only for boolean presence checking; only DockerReference
actually uses the value; in that case the transition to completeReference
may add a digest value, but a few lines below we do a, now redundant,
WithDigest computation which would have added the same digest anyway.
NOTE: This changes PolicyConfigurationIdentity and PolicyConfigurationNamespaces,
which now return values even if breakDockerReference. That seems to be the right
thing to do anyway.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
name:tag@digest is actually a valid input to reference.ParseNormalizedNamed;
the code used to set name:tag in s.name, but leave s.tag empty.
That happened to mostly work correctly, but for the wrong reasons. So, fix
setting s.tag for name:tag@digest inputs, and then update DockerReference
to handle such cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This starts a series of commits reworking storageReference to rely more on
reference.Named than on a set of independent string members.
First, introduce completeReference: this is a reference.Named, like
storageReference.name, but it includes both the tag and the digest, possibly
at the same time. For now, the field is only set, never used; users will be
ported one aspect at a time in the following commits; ultimately it will be
renamed to something shorter after we eliminate storageReference.name as well.
Also, to preserve the ImageDestination.DockerReference() ... interesting
behavior ..., add an explicit breakDockerReference boolean. This will allow
us to rely on completeDockerReference being always available, while keeping the
existing behavior unchanged until we can improve upon it.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Extract an imageMatchesRepo helper from resolveImage. This avoids
duplicating it, and a separate helper with "return" is easier to follow
than labeled breaks.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This should produce equivalent results, but Name() uses the fully-qualified
format (i.e. is safer against normalization changes), while FamiliarName is
primarily intended for UI output, and also more costly to compute.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... if PolicyConfigurationIdentity includes @id. At least the :tag form is clearly useful.
This does not handle the name:tag@digest case (which _can_ happen), where the tag is
currently recorded inside s.name but not s.tag; the possible code handling it would
be very non-obviously pointing out this difference. For now, only leave a FIXME in the test.
Maybe it should be handled by refusing such input instead; e.g. that's what
docker.Transport does.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Network IO paths should react to cancels now.
- File IO paths generally still won't.
- `SystemContext` objects have been renamed to `sys` to leave `ctx`
available for the stdlib context objects.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
Log the message about a given reference not being resolvable to an image
ID at the debug level rather than as an error, and wrap the error that
we return in case the caller wants to relay the reference's value back
as part of the error.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When attempting to locate an image, if we have a name+digest (i.e.,
canonical reference), use the store's new ImagesByDigest() method to
search for images that match a given digest, and which have at least one
name that matches the specified name, treating the reference as an
implicit name so long as it has an explicit name that matches (give or
take a tag).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Do not construct storageImageSource around storageImage around
storageUnnamedImageSource; instead, use the ~simple storageUnnamedImageSource
implementations, except using manifest.Manifest to edit layers in GetManifest.
Revert storageImage to only implementing .Size()
Still modifies the input reference, which breaks assumptions of signature validation.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
References can contain digests or tags, but not both, so fix our parser
to handle either type of name for use as image names.
Update PolicyConfigurationNamespaces() to discard tags and digests when
computing the list of alternate namespaces which can match an image.
Drop digests from a storageImageSource's copies of references, in case
we're reading and writing images using references which contain digests,
since schema1 images tend to fail verification of those.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This patch overhauls how we use containers/storage to store images,
dropping most of our own metadata in favor of facilities that are
now provided by default by the storage library. Additionally:
* storageImageDestination now caches blobs in a temporary directory
until Commit() is called
* storageImageDestination generates a barebones manifest if one isn't
supplied before Commit() is called
* storageImageDestination uses new APIs in containers/storage to
look for a local layer with the same contents of a blob, making it
better at noticing when a PutBlob() isn't necessary
* storageImageDestination sets the creation date for the image if it
can be determined during Commit()
* storageImageDestination defaults to using the hex part of the digest
of the image's configuration blob as an image's ID, making it better
at catching re-pulls of the same image
* storageImageDestination no longer discards names which have been set
for an image when reusing an ID
* storageImage now counts sizes of uncompressed data when determining
image size
* storageImage now counts the size of the configuration blob when
computing an image's size
* storageImage returns an updated image with the manifest listing
uncompressed layer blobs
* storageImageSource also returns such an updated manifest
* storageImageSource now always returns uncompressed layers
Test changes:
* storage tests now always write an image manifest
* the test for determining an image's size now actually writes the
configuration blob that it later tries to read
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
... and use them for choosing an image from a manifest list, and
verifying whether an image is acceptable in a MustMatchRuntimeOS()
destination. Propagate the types.SystemContext through the
call stack as necessary.
This adds no users and seems not all that important, but after
we re-enable fetching manifest lists, docker_transport_test.go does a
"//busybox".NewImage(), which nowadays mean copying from a manifest list,
and then we need this override to keep tests working on non-Linux platforms
for which Docker does not publish images.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Remove the .Close() methods from UnparsedImage/Image, which closed the
underlying ImageSource. Instead, just require the caller to ensure
that the ImageSource is not closed as long as the UnparsedImage/Image
are used.
This allows using several independent UnparsedImage/Image instances
for a shared ImageSource; notably independent Image objects for the
individual image instances in a manifest list. (copy.Image is already
simpler although it is only using a single instance.)
To keep ImageReference.NewImage simple and not to break all the external
callers of this, also add a simple ImageCloser wrapper which retains
the ImageSource closing functionality, and return it from image.FromSource
and ImageReference.NewImage implementations.
(It's very likely many of the NewImage callers would be surprised by how this
handles manifest lists, and it is very tempting to break this API, at least
by renaming, to force the callers to consider this; however, this would be
better done after eliminating the need of ImageReference.NewImage entirely,
by replacing the specialized types.Image extensions with something else, first.)
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The requestedManifestMIMETypes parameter was added because a destination
might not support all manifest MIME types that the the source supports,
but the original use case now passes all manifest types and lets
containers/image convert internally. In generally, internal conversion
may be more comprehensive, is more predictable, and avoids bypassing
internal checks.
Fixes: #331
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
Add methods to the storage transport to provide a way to set default UID
and GID maps to be used for all stores. Use them to let the unit tests
avoid permissions problems.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Expand the syntax of the store's location to allow it to also contain
the runRoot location (optional, after the root, separated by '+') and a
list of driver options (optional, after the runRoot, separated by ":",
and itself broken up using ",").
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Container storage moved its implementation from containers/storage/storage
to containers/storage. It also removed the store.Get* calls to store.*,
eliminating the Get Prefix.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
When writing an image to the storage transport, add an image name based
on the tag, if one was given. This should save most consumers of the
library from having to mess around with names after calling
copy.Image().
When reading or deleting an image, make sure that if the reference we're
using to find it includes a name, that the image has the name, for the
sake of cases where the reference was initialized with both a name and
an ID.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This replaces the copy of github.com/docker/docker/reference in the same
place, which we have just gotten rid of, and allows using this package
even in consumers which insist on an incompatible version of
docker/distribution.
The copy has been edited to drop a reference to
github.com/docker/distribution/digestset .
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This is an intermediate step which will eventually go away.
The goal of this PR is to get rid of c/i/docker/daemon/reference and to
replace uses of it by direct calls to docker/distribution/reference.
We can't do that safely and easily, because the two have different
semantics for reference.Named.Name() and reference.Named.String(): we
return a minimized version, e.g. "busybox", upstream returns an expanded
version, e.g. "docker.io/library/busybox".
BEFORE this commit the difference is hidden by using
docker/distribution/reference.WithName, which allows using the minimized
version, and works with it correctly; but because we want to use the
upstream canonicalization code, which will change semantics, we can't
just mix and match.
To make the distinction explicit, this commmit adds an X to ALL public
names from c/i/docker/daemon/reference. E.g. a reference.XNamed type,
which has methods XName and XString.
This is pretty large, but does not change behavior at all. By
inspection it is clear to see that reference.XNamed and subtypes does
not expose any of the non-X, conflicting, method names.
Using e.g.
> git diff --word-diff-regex=.|grep -F '{+'|grep -v '^\([^{]\|{+X+}\)*{\?$'
it is possible to see that most lines in this diff only add a single X
letter, and manually inspect the few lines which don't match the regexp.
The only REALLY new code is an explicit definition of namedRef.XName()
and namedRef.XString(), and two newly added casts to namedRef in cases
where we need to use the underlying distreference.Reference within
a reference.XNamed value. Strictly speaking these changes change
behavior, in that third-party implementations of reference.XNamed are no
longer accepted; but we broke them by renaming at all.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Rebase.
- Stop doing quite so much redundant work.
- Make storageImageDestination.ShouldCompressLayers() return false
instead of true.
- Rename storageImageSource.GetBlobAndLayerID to getBlobAndLayerID,
since it doesn't need to be public.
- Don't bother recomputing diff sizes most of the time if we don't have
a number already, since it's okay to return -1 when we don't know.
- If the reference includes an ID, the list of namespaces that might
contain it should also include the tagged version of the name.
- If an image is only referenced by ID, make sure it has a leading "@"
to disambiguate it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add containers/storage as a backend type called 'storage'.
The image's blobs are stored either as storage layers (if they look like
archived layers) or (alongside the manifest and signatures) as named big
data items associated with the image.
Inspection data and record-keeping (so that we can remember which blobs
were layers and which weren't) are encoded as a JSON object which is
stored in the storage image's metadata field.
When importing blobs, layer IDs are generated by concatenating the
parent's layer ID (if there is one) with the hex string representation
of the expected digest of the content blob, if one is known. If there
is no expected digest, the ID is randomly generated.
If we find ourselves importing a layer with the same ID as a layer that
we already have, we digest the incoming stream and compare it to the
cached digest of the already-present layer, and return an error only if
they don't match. If an expected blob digest is provided, the actual
digest of the blob is compared with it, and if they don't match, an
error is returned.
If we find ourselves importing a blob more than once, we track the IDs
of each of the resulting layers.
If we find ourselves importing an image that wants to be tagged with a
name that is already in use, the name is then assigned to the new image
and the old image remains otherwise unmodified. If that incoming image
claims to have the same ID as an image which we already have, the import
will fail.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Miloslav Trmač