Before, this only waited for the download to complete. There was no
guarantee that the layer had been registered in the graph and was ready
use. This is especially problematic with v2 pulls, which wait for all
downloads before extracting layers.
Change Broadcaster to allow an error value to be propagated from Close
to the waiters.
Make the wait stop when the extraction is finished, rather than just the
download.
This also fixes v2 layer downloads to prefix the pool key with "layer:"
instead of "img:". "img:" is the wrong prefix, because this is what v1
uses for entire images. A v1 pull waiting for one of these operations to
finish would only wait for that particular layer, not all its
dependencies.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Ensure that layers are not excluded from manifests based on previous pushes.
Continue skipping pushes on layers which were pushed by a previous tag.
Update push multiple tag tests.
Ensure that each tag pushed exists on the registry and is pullable.
Add output comparison on multiple tag push check.
fixes#15536
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The Docker Daemon should send actual actions client ask for to issue tokens,
not all the permissions that client is guaranteed.
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
This file was not well documented and had very high cyclomatic complexity.
This patch completely rearranges this file and the ImageDelete method to
be easier to follow and more maintainable in the future.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
- Rename to Broadcaster
- Document exported types
- Change Wait function to just wait. Writing a message to the writer and
adding the writer to the observers list are now handled by separate
function calls.
- Avoid importing logrus (the condition where it was used should never
happen, anyway).
- Make writes non-blocking
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Previously, its other return value was used even when it returned an
error. This is awkward and goes against the convention. It also could
have resulted in a nil pointer dereference when an error was returned
because of an unknown pool type. This changes the unknown pool type
error to a panic (since the pool types are hardcoded at call sites and
must always be "push" or "pull"), and returns a "found" boolean instead
of an error.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Based on #12874 from Sam Abed <sam.abed@gmail.com>. His original commit
was brought up to date by manually porting the changes in pull.go into
the new code in pull_v1.go and pull_v2.go.
Fixes#8385
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The practice of buffering to a tempfile during a pushing contributes massively
to slow V2 push performance perception. The protocol was actually designed to
avoid precalculation, supporting cut-through data push. This means we can
assemble the layer, calculate its digest and push to the remote endpoint, all
at the same time.
This should increase performance massively on systems with slow disks or IO
bottlenecks.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Windows: add support for images stored in alternate location.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
* use downloadInfo pointers everywhere
* use downloads slice only for things that we really download
* cleanup tmp files in all cases
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Make command like "docker images ubuntu:14.04" work and filter out the
image with the given tag.
Closes#8048.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This patch splits LookupEndpoints into LookupPullEndpoints and
LookupPushEndpoints so that mirrors added with --registry-mirror are
skipped in the list returned by LookupPushEndpoints.
Fixes https://github.com/docker/distribution/issues/823
Signed-off-by: Tibor Vass <tibor@docker.com>
The process of pulling an image spawns a new goroutine for each layer in the
image manifest. If any of these downloads fail we would stop everything and
return the error, even though other goroutines would still be running and
writing output through a progress reader which is attached to an http response
writer. Since the request handler had already returned from the first error,
the http server panics when one of these download goroutines makes a write to
the response writer buffer.
This patch prevents this crash in the daemon http server by waiting for all of
the download goroutines to complete, even if one of them fails. Only then does
it return, terminating the request handler.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
We noticed a regression since the 1.7.1 patch after some refactoring. This
patch corrects the behavior and adds integration tests for modified manifest
and rootfs layer blobs.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Some structures use int for sizes and UNIX timestamps. On some
platforms, int is 32 bits, so this can lead to the year 2038 issues and
overflows when dealing with large containers or layers.
Consistently use int64 to store sizes and UNIX timestamps in
api/types/types.go. Update related to code accordingly (i.e.
strconv.FormatInt instead of strconv.Itoa).
Use int64 in progressreader package to avoid integer overflow when
dealing with large quantities. Update related code accordingly.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Expand the godoc documentation for the graph package.
Centralize DefaultTag in the graphs/tag package instead of defining it
twice.
Remove some unnecessary "config" structs that are only used to pass
a few parameters to a function.
Simplify the GetParentsSize function - there's no reason for it to take
an accumulator argument.
Unexport some functions that aren't needed outside the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[v2: a separate aufs commit is merged into this one]
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
In 1.6.2 we were decoding inspect API response into interface{}.
time.Time fields were JSON encoded as RFC3339Nano in the response
and when decoded into interface{} they were just strings so the inspect
template treated them as just strings.
From 1.7 we are decoding into types.ContainerJSON and when the template
gets executed it now gets a time.Time and it's formatted as
2015-07-22 05:02:38.091530369 +0000 UTC.
This patch brings back the old behavior by typing time.Time fields
as string so they gets formatted as they were encoded in JSON -- RCF3339Nano
Signed-off-by: Antonio Murdaca <runcom@linux.com>
Add a trusted flag to force the cli to resolve a tag into a digest via the notary trust library and pull by digest.
On push the flag the trust flag will indicate the digest and size of a manifest should be signed and push to a notary server.
If a tag is given, the cli will resolve the tag into a digest and pull by digest.
After pulling, if a tag is given the cli makes a request to tag the image.
Use certificate directory for notary requests
Read certificates using same logic used by daemon for registry requests.
Catch JSON syntax errors from Notary client
When an uncaught error occurs in Notary it may show up in Docker as a JSON syntax error, causing a confusing error message to the user.
Provide a generic error when a JSON syntax error occurs.
Catch expiration errors and wrap in additional context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
* Add godoc documentation where it was missing
* Change identifier names that don't match Go style, such as INDEX_NAME
* Rename RegistryInfo to PingResult, which more accurately describes
what this structure is for. It also has the benefit of making the name
not stutter if used outside the package.
Updates #14756
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
pushV2Tag already deduplicates layers, but the scope of this
deduplication is only for a particular tag. If we are pushing all tags
in a repository, we may check layers several times. Fix this by moving
the layersSeen map from the pushV2Tag function to the v2Pusher struct.
In addition to avoiding some useless round-trips, this makes the "docker
push" output less confusing. It formerly could contain many repeated
lines like:
124e2127157f: Image already exists
124e2127157f: Image already exists
...
Add test coverage based on the "docker push" output: a hash should not
appear multiple times when pushing multiple tags.
Fixes#14873
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
with the current duplication of code in the grap.go split-up, this puts
all assembly/disassembly logic into isolated functions
Signed-off-by: Vincent Batts <vbatts@redhat.com>
if there is a tar-data.json.gz present for an image layer, then use it
to create the tar archive, instead of the traditional graphdriver Diff.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Conflicts:
graph/graph.go
Preserve the entries from the tar archive for layers added to the graph.
With these entries and relative filesystem path, the tar archives can be
reassembled later.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Currently canonical name gets set to the local name and displayed in the errors.
Canonical name should be the unique and canonical name for an image.
Use docker.io as the canonical domain for images on the public registry.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Currently the layer array is initialized with the first layer then the first layer is appened to the layer list. Adding the first layer twice causes the layer to appear twice in the manifest, making a duplicate push and pull attempt occur.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
During `(*Graph).Register, there was no protection on adding new layers
concurrently. In some cases, this resulted in corruption of a layer by creating
the directory but not the underlying data. This manifested in several different
IO errors reported in the client. This attempts to fix this by adding a mutex
by Image ID to protect the Register operation.
We do not completely understand the root cause of this corruption other than
the result is somehow tied to this particular function. This fix has been
confirmed to address the issue through testing.
Unfortunately, this fix does not address existing corruption. The user will
have to remove and re-pull the corrupt layer to stop the error from happening
in the future. This change only ensures that the layer will not become corrupt.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
An inspection of the graph package showed this function to be way out of place.
It is only depended upon by the daemon code. The function prepares a top-level
readonly layer used to provide a consistent runtime environment for docker
images.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Currently digests are not stored on pull, causing a simple re-tag or re-push to send up all layers. Storing the digests on pull will allow subsequent pushes to the same repository to not push up content.
This does not address pushing content to a new repository. When content is pushed to a new repository, the digest will be recalculated. Since only one digest is currently stored, it may cause a new content push to the original repository.
Fixes#13883
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Export image/container metadata stored in graph driver. Right now 3 fields
DeviceId, DeviceSize and DeviceName are being exported from devicemapper.
Other graph drivers can export fields as they see fit.
This data can be used to mount the thin device outside of docker and tools
can look into image/container and do some kind of inspection.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Update get and set functions to use digests.
Update push code to use the digest type instead of string
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Move graph related functions in image to graph package.
Consolidating graph functionality is the first step in refactoring graph into an image store model.
Subsequent refactors will involve breaking up graph into multiple types with a strongly defined interface.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Aaron Lehmann
Derek McGowan
Rozhnov Alexandr
Hu Keping
Vincent Batts
Tonis Tiigi
Arnaud Porterie
xiekeyang
Josh Hawn
Brian Goff
Sam Abed
David Calavera
Vincent Demeester
Sevki Hasirci
Alexander Morozov
Taylor Jones
Stephen J Day
Jessie Frazelle
Shijiang Wei
Stefan J. Wernli
Sebastiaan van Stijn
Alexander Morozov
Liu Hua
Kun Zhang
Zhang Wei
Tibor Vass
Tibor Vass
Srini Brahmaroutu
John Howard
Lei Jitang
Kir Kolyshkin
Srini Brahmaroutu
Antonio Murdaca
Doug Davis
Ma Shimiao
Stephen Day
Roman Strashkin
Phil Estes
Vincent Batts
Don Kjer
Vivek Goyal
Matt Moore
yuchangchun