Currently the layer array is initialized with the first layer then the first layer is appened to the layer list. Adding the first layer twice causes the layer to appear twice in the manifest, making a duplicate push and pull attempt occur.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
During `(*Graph).Register, there was no protection on adding new layers
concurrently. In some cases, this resulted in corruption of a layer by creating
the directory but not the underlying data. This manifested in several different
IO errors reported in the client. This attempts to fix this by adding a mutex
by Image ID to protect the Register operation.
We do not completely understand the root cause of this corruption other than
the result is somehow tied to this particular function. This fix has been
confirmed to address the issue through testing.
Unfortunately, this fix does not address existing corruption. The user will
have to remove and re-pull the corrupt layer to stop the error from happening
in the future. This change only ensures that the layer will not become corrupt.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
An inspection of the graph package showed this function to be way out of place.
It is only depended upon by the daemon code. The function prepares a top-level
readonly layer used to provide a consistent runtime environment for docker
images.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Currently digests are not stored on pull, causing a simple re-tag or re-push to send up all layers. Storing the digests on pull will allow subsequent pushes to the same repository to not push up content.
This does not address pushing content to a new repository. When content is pushed to a new repository, the digest will be recalculated. Since only one digest is currently stored, it may cause a new content push to the original repository.
Fixes#13883
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Export image/container metadata stored in graph driver. Right now 3 fields
DeviceId, DeviceSize and DeviceName are being exported from devicemapper.
Other graph drivers can export fields as they see fit.
This data can be used to mount the thin device outside of docker and tools
can look into image/container and do some kind of inspection.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Update get and set functions to use digests.
Update push code to use the digest type instead of string
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Move graph related functions in image to graph package.
Consolidating graph functionality is the first step in refactoring graph into an image store model.
Subsequent refactors will involve breaking up graph into multiple types with a strongly defined interface.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
To ensure manifest integrity when pulling by digest, this changeset ensures
that not only the remote digest provided by the registry is verified but also
that the digest provided on the command line is checked, as well. If this check
fails, the pull is cancelled as with an error. Inspection also should that
while layers were being verified against their digests, the error was being
treated as tech preview image signing verification error. This, in fact, is not
a tech preview and opens up the docker daemon to man in the middle attacks that
can be avoided with the v2 registry protocol.
As a matter of cleanliness, the digest package from the distribution project
has been updated to latest version. There were some recent improvements in the
digest package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The v2 registry will act as a pull-through cache, and needs to be
handled differently by the client to the v1 registry mirror.
See docker/distribution#459 for details
Configuration
Only one v2 registry can be configured as a mirror. Acceptable configurations
in this chanage are: 0...n v1 mirrors or 1 v2 mirror. A mixture of v1 and v2
mirrors is considered an error.
Pull
If a v2 mirror is configured, all pulls are redirected to that mirror. The
mirror will serve the content locally or attempt a pull from the upstream mirror,
cache it locally, and then serve to the client.
Push
If an image is tagged to a mirror, it will be pushed to the mirror and be
stored locally there. Otherwise, images are pushed to the hub. This is
unchanged behavior.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
This patch removes the need for requestFactories and decorators
by implementing http.RoundTripper transports instead.
It refactors some challenging-to-read code.
NewSession now takes an *http.Client that can already have a
custom Transport, it will add its own auth transport by wrapping
it.
The idea is that callers of http.Client should not bother
setting custom headers for every handler but instead it should
be transparent to the callers of a same context.
This patch is needed for future refactorings of registry,
namely refactoring of the v1 client code.
Signed-off-by: Tibor Vass <tibor@docker.com>
The intention of the user is to download a verified image if explicitly
pulling with a digest and falling back to v1 registry circumvents that
protection.
Signed-off-by: Nuutti Kotivuori <naked@iki.fi>
Prior to this patch, the response of
- GET /images/json
- GET /containers/json
- GET /images/(name)/history
display the Created Time as UNIX format which doesn't make sense.
These should be more readable as CLI command `docker inspect` shows.
Due to the case that an older client with a newer version daemon, we
need the version check for now.
Signed-off-by: Hu Keping <hukeping@huawei.com>
As reported in #11294, the Docker daemon will execute contains it
shouldn't run in the event that a requested tag is present in an image's
ID. This leads to the wrong image being started up silently.
This change reduces the risk of such a collision by using the short ID
iff the actual revOrTag looks like a short ID (not that it necessarily
is).
Signed-off-by: Richard Burnison <rburnison@ebay.com>
This makes `registry.Service` a first class type and does not use jobs
to interact with this type.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Update registry package to use the v2 registry api from distribution. Update interfaces to directly take in digests.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
When buffering to file add support for compressing the tar contents. Since digest should be computed while writing buffer, include digest creation during buffer.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This patch causes `The image you are pulling has been verified` status
message to be produced also when the repository is pulled for the first
time.
Signed-off-by: Michal Minar <miminar@redhat.com>
When pushing to the official registry, v2 push will be attempted first. If the v2 endpoint is unavailable, a push to v1 will occur. If a v2 push is started and failed, the push operation will fail. Pulling non-official images the official v2 registry is also enabled to pull the newly push images. When a pull of a v2 image fails, v1 will always be attempted.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add ability to refer to an image by repository name and digest using the
format repository@digest. Works for pull, push, run, build, and rmi.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
Update pull code to consider any layer download or new tag as an update.
Update hello-world frozen image to be explicitly tagged as frozen, to not interfere with pull tests. The hello-world is used by pull tests because of its small size and there is no other official image with such a size.
fixes#11383
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Manifest is now generated during a v2 push, not relying on previously generated hashes. When pushing a layer, the hash is directly calculated from the tar contents which will be pushed. Computing the hash on push ensures that the hash contents always match what is seen by the registry. This also mitigates issues with tarsum differences and permits using pure SHA digests.
Additionally the new manifest function is moved to the unit tests since it is no longer called outside the tests.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The call to io.Copy was not being checked for errors at a critical point in the
pull logic. This patch will log the error value if not nil.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
pushV2Image() calls TarLayer() which returns an archive. One needs to
Close() the archive once done otherwise it will leave mounted devices
if devicemapper graph driver is being used.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
manifest code calls TarLayer() and gets archive. This archive needs to
be closed once caller is done using it to release the resrouces held
by archive. For the devicemapper graphdriver, archive keeps a device
mounted (device which is backing the layer). If archive.Close() is not
called, that device remains mounted and later deletion of device fails
leading to various other issues.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
The trust graph currently pulls the keys needed to verify official images. For non-official images this graph check is useless. To avoid hitting the key statement url when pulling from private v2 registries, restrict the check to official image pulls.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Pushing by image ID is not allowed in the Docker CLI and not supported by the registry. An unnamed image also cannot be pushed to a private registry, since no endpoint is specified and it will default to the hub. The hub also does not support this use case, therefore removing the code path is the best solution.
The ability to push a layer without a name is unsupported by the v2 registry.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Only show the verification message if all the tarsum checks pass and the image manifest is verified.
No longer return an error when a tarsum verification fails, just reset the verification flag.
Tarsum verification is less meaningful without a verified manifest and therefore it should not cause an error.
Updated the verified image test to pull an image which expected to have a verified manifest and contents.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
It didn't work on v2 anyways. And an image with a lot of aliases was slow to
fetch.
Docker-DCO-1.1-Signed-off-by: Jessie Frazelle <princess@docker.com> (github: jfrazelle)
On pull treat an unavailable v2 registry as a non-error fallback.
On push only show v2 error message in debug mode.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
No longer push to the official v2 registry when it is available. This allows pulling images from the v2 registry without defaulting push. Only pull official images from the v2 official registry.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
When requesting a token, the basic auth header is always being set even
if there is no username value. This patch corrects this and does not set
the basic auth header if the username is empty.
Also fixes an issue where pulling all tags from a v2 registry succeeds
when the image does not actually exist on the registry.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
The checksum is now being stored in a separate file beside the image
JSON file.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Before the V2 registry changes, images with no config could be pushed.
This change fixes a regression that made those images not able to be
pushed to a registry.
Signed-off-by: Euan Kemp <euank@euank.com>
When progress reader closes it overwrites the progress line with the full progress bar, replaces the completed message.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The v2 session code will no longer update the indexEndpoint value, therefore it is not necessary to save and restore the value for use with v1.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.
Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This refactors the starting work by the prior commits to make this safe
for access. A maximum of 5 worker go routines are started to lookup
images on the endpoint. Another go routine consumes the images that are
required to be pushed into a map for quick lookups. The map is required
because the pushing of the image json and layer have to be done in the
correct order or the registry will explode in fire.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Passing RepositoryInfo to ResolveAuthConfig, pullRepository, and pushRepository
Moving --registry-mirror configuration to registry config
Created resolve_repository job
Repo names with 'index.docker.io' or 'docker.io' are now synonymous with omitting an index name.
Adding test for RepositoryInfo
Adding tests for opts.StringSetOpts and registry.ValidateMirror
Fixing search term use of repoInfo
Adding integration tests for registry mirror configuration
Normalizing LookupImage image name to match LocalName parsing rules
Normalizing repository LocalName to avoid multiple references to an official image
Removing errorOut use in tests
Removing TODO comment
gofmt changes
golint comments cleanup. renaming RegistryOptions => registry.Options, and RegistryServiceConfig => registry.ServiceConfig
Splitting out builtins.Registry and registry.NewService calls
Stray whitespace cleanup
Moving integration tests for Mirrors and InsecureRegistries into TestNewIndexInfo unit test
Factoring out ValidateRepositoryName from NewRepositoryInfo
Removing unused IndexServerURL
Allowing json marshaling of ServiceConfig. Exposing ServiceConfig in /info
Switching to CamelCase for json marshaling
PR cleanup; removing 'Is' prefix from boolean members. Removing unneeded json tags.
Removing non-cleanup related fix for 'localhost:[port]' in splitReposName
Merge fixes for gh9735
Fixing integration test
Reapplying #9754
Adding comment on config.IndexConfigs use from isSecureIndex
Remove unused error return value from isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Adding back comment in isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
If .dockerignore mentions either then the client will send them to the
daemon but the daemon will erase them after the Dockerfile has been parsed
to simulate them never being sent in the first place.
an events test kept failing for me so I tried to fix that too
Closes#8330
Signed-off-by: Doug Davis <dug@us.ibm.com>
There has been a lot of discussion (issues 4242 and 5262) about making
`FROM scratch` either a special case or making `FROM` optional, implying
starting from an empty file system.
This patch makes the build command `FROM scratch` special cased from now on
and if used does not pull/set the the initial layer of the build to the ancient
image ID (511136ea..) but instead marks the build as having no base image. The
next command in the dockerfile will create an image with a parent image ID of "".
This means every image ever can now use one fewer layer!
This also makes the image name `scratch` a reserved name by the TagStore. You
will not be able to tag an image with this name from now on. If any users
currently have an image tagged as `scratch`, they will still be able to use that
image, but will not be able to tag a new image with that name.
Goodbye '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158',
it was nice knowing you.
Fixes#4242
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This commit is patch for following comment
// TODO: This method should return the errors instead of masking them and returning false
Signed-off-by: Daehyeok Mun <daehyeok@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
When we use the engine/env object we can run into a situation where
a string is passed in as the value but later on when we json serialize
the name/value pairs, because the string is made up of just numbers
it appears as an integer and not a string - meaning no quotes. This
can cause parsing issues for clients.
I tried to find all spots where we call env.Set() and the type of the
name being set might end up having a value that could look like an int
(like author). In those cases I switched it to use env.SetJson() instead
because that will wrap it in quotes.
One interesting thing to note about the testcase that I modified is that
the escaped quotes should have been there all along and we were incorrectly
letting it thru. If you look at the metadata stored for that resource you
can see the quotes were escaped and we lost them during the serialization
steps because of the env.Set() stuff. The use of env is probably not the
best way to do all of this.
Closes: #9602
Signed-off-by: Doug Davis <dug@us.ibm.com>
I was confused earlier when I did:
```
docker push localhost.localdomain:1234/foo
```
Because docker told me:
```
No such id: localhost.localdomain:1234/foo
```
I actually had buried in my mind the solution to this, but the error
message
confused me because I had recently had some fun trying to get the
registry
working and therefore thought it was telling me that I didn't have an
account
on the registry.
This pull request makes it unambiguous that the error is that the
specified
image is unknown.
/cc @cpuguy83
Docker-DCO-1.1-Signed-off-by: Peter Waller <p@pwaller.net> (github: pwaller)
Roman Strashkin
Derek McGowan
Vincent Demeester
Stephen J Day
Antonio Murdaca
Phil Estes
David Calavera
Vincent Batts
Don Kjer
Vivek Goyal
Arnaud Porterie
John Howard
Matt Moore
yuchangchun
Richard
Tibor Vass
Tibor Vass
Nuutti Kotivuori
Victor Vieux
Hu Keping
Antonio Murdaca
Tianon Gravi
Jessie Frazelle
Burke Libbey
Alexander Morozov
Brian Goff
Ankush Agarwal
Josh Hawn
Simei He
Doug Davis
Alexander Morozov
Srini Brahmaroutu
Doug Davis
Ma Shimiao
Richard Burnison
David Young
Shishir Mahajan
José Tomás Albornoz
Michael Crosby
Lei Jitang
Peter Choi
Michal Minar
bobby abbott
Andy Goldstein
Darren Shepherd
Dan Walsh
Srini Brahmaroutu
Doug MacEachern
Steve Francia
Michael Crosby
Jessica Frazelle
Tibor Vass
guoxiuyan
Euan
Ian Babrou
Victor Vieux
Ian Calvert
Daehyeok Mun
unclejack
Peter Waller