The practice of buffering to a tempfile during a pushing contributes massively
to slow V2 push performance perception. The protocol was actually designed to
avoid precalculation, supporting cut-through data push. This means we can
assemble the layer, calculate its digest and push to the remote endpoint, all
at the same time.
This should increase performance massively on systems with slow disks or IO
bottlenecks.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This fixes two problems:
1. docker run --device /dev/sda:rw ubuntu bash doesn't work
2. --device /dev/zero:/dev/noro:ro doesn't show clear error message,
but fail when writing to cgroup file.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Signed-off-by: Zhang Kun <zkazure@gmail.com>
goformat
Signed-off-by: Zhang Kun <zkazure@gmail.com>
fix small
Signed-off-by: Zhang Kun <zkazure@gmail.com>
change to rm
Signed-off-by: Zhang Kun <zkazure@gmail.com>
handler other error
Signed-off-by: Zhang Kun <zkazure@gmail.com>
unique ERR
Signed-off-by: Zhang Kun <zkazure@gmail.com>
setHostConfig_fail_test
Signed-off-by: Zhang Kun <zkazure@gmail.com>
format
Signed-off-by: Zhang Kun <zkazure@gmail.com>
err handle and modify test
Signed-off-by: Zhang Kun <zkazure@gmail.com>
golint error
Signed-off-by: Zhang Kun <zkazure@gmail.com>
Docker ps default format should not take precedence over cli flags.
This happens effectively for other flags except `-q`.
We need to let the cli to set the format as table to print the
expected output with `-q`.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Changing vendor/src/github.com/docker/libnetwork to match lindenlab/libnetwork custom-host-port-ranges-1.7 branch
The bool logic around setting up the TTY ended up getting flipped
accidentally.
Also added a test for exec with TTY.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Introduce a write denial for files at the root of /proc.
This prohibits root users from performing a chmod of those
files. The rules for denials in proc are also cleaned up,
making the rules better match their targets.
Locally tested on:
- Ubuntu precise (12.04) with AppArmor 2.7
- Ubuntu trusty (14.04) with AppArmor 2.8.95
Signed-off-by: Eric Windisch <eric@windisch.us>
Make command like "docker images ubuntu:14.04" work and filter out the
image with the given tag.
Closes#8048.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This reverts commit 40b71adee3.
Original commit (for which this is effectively a rebased version) is
72a500e9e5 and was provided by Lei Jitang
<leijitang@huawei.com>.
Signed-off-by: Tim Dettrick <t.dettrick@uq.edu.au>
Have network files mounted read-only when mounted using the -v
open and -v parameter has 'ro' passed.
Signed-off-by: Stefan Berger <stefanb@us.ibm.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The process of pulling an image spawns a new goroutine for each layer in the
image manifest. If any of these downloads fail we would stop everything and
return the error, even though other goroutines would still be running and
writing output through a progress reader which is attached to an http response
writer. Since the request handler had already returned from the first error,
the http server panics when one of these download goroutines makes a write to
the response writer buffer.
This patch prevents this crash in the daemon http server by waiting for all of
the download goroutines to complete, even if one of them fails. Only then does
it return, terminating the request handler.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Don't assume that any random image will have 'top' and explicitely use
the busybox image for testing.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
We noticed a regression since the 1.7.1 patch after some refactoring. This
patch corrects the behavior and adds integration tests for modified manifest
and rootfs layer blobs.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
manually stopped
If a container is running with a restart policy of always and it's
manually stopped, then on daemon restart it will be running.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
[pkg/archive] Update archive/copy path handling
- Remove unused TarOptions.Name field.
- Add new TarOptions.RebaseNames field.
- Update some of the logic around path dir/base splitting.
- Update some of the logic behind archive entry name rebasing.
[api/types] Add LinkTarget field to PathStat
[daemon] Fix stat, archive, extract of symlinks
These operations *should* resolve symlinks that are in the path but if the
resource itself is a symlink then it *should not* be resolved. This patch
puts this logic into a common function `resolvePath` which resolves symlinks
of the path's dir in scope of the container rootfs but does not resolve the
final element of the path. Now archive, extract, and stat operations will
return symlinks if the path is indeed a symlink.
[api/client] Update cp path hanling
[docs/reference/api] Update description of stat
Add the linkTarget field to the header of the archive endpoint.
Remove path field.
[integration-cli] Fix/Add cp symlink test cases
Copying a symlink should do just that: copy the symlink NOT
copy the target of the symlink. Also, the resulting file from
the copy should have the name of the symlink NOT the name of
the target file.
Copying to a symlink should copy to the symlink target and not
modify the symlink itself.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[v2: a separate aufs commit is merged into this one]
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
The 'deny ptrace' statement was supposed to only ignore
ptrace failures in the AUDIT log. However, ptrace was implicitly
allowed from unconfined processes (such as the docker daemon and
its integration tests) due to the abstractions/base include.
This rule narrows the definition such that it will only ignore
the failures originating inside of the container and will not
cause denials when the daemon or its tests ptrace inside processes.
Introduces positive and negative tests for ptrace /w apparmor.
Signed-off-by: Eric Windisch <eric@windisch.us>
Integration tests were failing due to proc filter behavior
changes with new apparmor policies.
Also include the missing docker-unconfined policy resolving
potential startup errors. This policy is complain-only so
it should behave identically to the standard unconfined policy,
but will not apply system path-based policies within containers.
Signed-off-by: Eric Windisch <eric@windisch.us>
Currently some notary tests change the system clock to check for expiration.
Skip these tests until the code can be refactored to not rely on updating the system clock.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Builds where the base images have been resolved to trusted digest
references will now be tagged with the original tag reference from
the Dockerfile on a successful build.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Will attempt to load profiles automatically. If loading fails
but the profiles are already loaded, execution will continue.
A hard failure will only occur if Docker cannot load
the profiles *and* they have not already been loaded via
some other means.
Also introduces documentation for AppArmor.
Signed-off-by: Eric Windisch <eric@windisch.us>
Return an error when the container is stopped only in api versions
equal or greater than 1.20 (docker 1.8).
Signed-off-by: David Calavera <david.calavera@gmail.com>
Fix#14915. Add unit test for #14915.
Thanks @runcom for the test case: when the client calls 1.18 api
version w/o hostconfig it results in a nil pointer dereference.
Signed-off-by: Stephen Rust <srust@blockbridge.com>
In 1.6.2 we were decoding inspect API response into interface{}.
time.Time fields were JSON encoded as RFC3339Nano in the response
and when decoded into interface{} they were just strings so the inspect
template treated them as just strings.
From 1.7 we are decoding into types.ContainerJSON and when the template
gets executed it now gets a time.Time and it's formatted as
2015-07-22 05:02:38.091530369 +0000 UTC.
This patch brings back the old behavior by typing time.Time fields
as string so they gets formatted as they were encoded in JSON -- RCF3339Nano
Signed-off-by: Antonio Murdaca <runcom@linux.com>
Update help line to allow 90 characters instead of 80
The trust flag pushes out the help description column wider, requiring more room to display help messages.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Clean up tests to remove duplicate code
Add tests which run pull and create in an isolated configuration directory.
Add build test for untrusted tag
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Added notary server to docker base image.
Created trust suite which runs trust server for running trusted commands.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add a trusted flag to force the cli to resolve a tag into a digest via the notary trust library and pull by digest.
On push the flag the trust flag will indicate the digest and size of a manifest should be signed and push to a notary server.
If a tag is given, the cli will resolve the tag into a digest and pull by digest.
After pulling, if a tag is given the cli makes a request to tag the image.
Use certificate directory for notary requests
Read certificates using same logic used by daemon for registry requests.
Catch JSON syntax errors from Notary client
When an uncaught error occurs in Notary it may show up in Docker as a JSON syntax error, causing a confusing error message to the user.
Provide a generic error when a JSON syntax error occurs.
Catch expiration errors and wrap in additional context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This patch creates a new cli package that allows to combine both client
and daemon commands (there is only one daemon command: docker daemon).
The `-d` and `--daemon` top-level flags are deprecated and a special
message is added to prompt the user to use `docker daemon`.
Providing top-level daemon-specific flags for client commands result
in an error message prompting the user to use `docker daemon`.
This patch does not break any old but correct usages.
This also makes `-d` and `--daemon` flags, as well as the `daemon`
command illegal in client-only binaries.
Signed-off-by: Tibor Vass <tibor@docker.com>
The header row was not being printed when "docker ps" was invoked without containers thanks to the new format support, and we instead received a single blank line.
Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
pushV2Tag already deduplicates layers, but the scope of this
deduplication is only for a particular tag. If we are pushing all tags
in a repository, we may check layers several times. Fix this by moving
the layersSeen map from the pushV2Tag function to the v2Pusher struct.
In addition to avoiding some useless round-trips, this makes the "docker
push" output less confusing. It formerly could contain many repeated
lines like:
124e2127157f: Image already exists
124e2127157f: Image already exists
...
Add test coverage based on the "docker push" output: a hash should not
appear multiple times when pushing multiple tags.
Fixes#14873
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Prevent the docker daemon from mounting the created network files over
those provided by the user via -v command line option. This would otherwise
hide the one provide by the user.
The benefit of this is that a user can provide these network files using the
-v command line option and place them in a size-limited filesystem.
Signed-off-by: Stefan Berger <stefanb@us.ibm.com>
Re-add the docs from @calavera's PR to the moved cli cmd reference docs.
Fix gofmt and vet issues from carried commits
Add integration test for using format with --no-trunc and multi-names
Fix custom_test map order dependency on expected value check
Add docs to reference/commandline/ps.md
Remove "-F" flag option from original carried PR content
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
Adds several integration tests for `docker cp` behavior with over a dozen
tests for each of:
container -> local
local -> container
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
After merging mounting cgroups in container this tests doing wrong
checks. Cgroup paths could be prepended by other cgroups from host.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
The automatic installation of AppArmor policies prevents the
management of custom, site-specific apparmor policies for the
default container profile. Furthermore, this change will allow
a future policy for the engine itself to be written without demanding
the engine be able to arbitrarily create and manage AppArmor policies.
- Add deb package suggests for apparmor.
- Ubuntu postinst use aa-status & fix policy path
- Add the policies to the debian packages.
- Add apparmor tests for writing proc files
Additional restrictions against modifying files in proc
are enforced by AppArmor. Ensure that AppArmor is preventing
access to these files, not simply Docker's configuration of proc.
- Remove /proc/k?mem from AA policy
The path to mem and kmem are in /dev, not /proc
and cannot be restricted successfully through AppArmor.
The device cgroup will need to be sufficient here.
- Load contrib/apparmor during integration tests
Note that this is somewhat dirty because we
cannot restore the host to its original configuration.
However, it should be noted that prior to this patch
series, the Docker daemon itself was loading apparmor
policy from within the tests, so this is no dirtier or
uglier than the status-quo.
Signed-off-by: Eric Windisch <eric@windisch.us>
As suggested in https://github.com/docker/docker/pull/14004/files#r34022527
The concern there is we can't differentiate whether user explicitly
asked for an invalid value of -1 or he did not specify anything.
I don't think this would be a problem, because:
- like all other default values like zero, we can't differentiate
user specify it or not, most of which, zeros are also invalid, so
default is default, we show these default values in help info,
so users would know if they set value as default, it'll be like
they set nothing.
- we can't do this kind of string check in REST api request, so
it'll make the behave different from docker command and RESTapi.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Closes#14621
This one grew to be much more than I expected so here's the story... :-)
- when a bad port string (e.g. xxx80) is passed into container.create()
via the API it wasn't being checked until we tried to start the container.
- While starting the container we trid to parse 'xxx80' in nat.Int()
and would panic on the strconv.ParseUint(). We should (almost) never panic.
- In trying to remove the panic I decided to make it so that we, instead,
checked the string during the NewPort() constructor. This means that
I had to change all casts from 'string' to 'Port' to use NewPort() instead.
Which is a good thing anyway, people shouldn't assume they know the
internal format of types like that, in general.
- This meant I had to go and add error checks on all calls to NewPort().
To avoid changing the testcases too much I create newPortNoError() **JUST**
for the testcase uses where we know the port string is ok.
- After all of that I then went back and added a check during container.create()
to check the port string so we'll report the error as soon as we get the
data.
- If, somehow, the bad string does get into the metadata we will generate
an error during container.start() but I can't test for that because
the container.create() catches it now. But I did add a testcase for that.
Signed-off-by: Doug Davis <dug@us.ibm.com>
The check for the end of the loop was off by one which is why we saw
errors on the following inpsect() call instead of a timeout
Signed-off-by: Doug Davis <dug@us.ibm.com>
Replaced github.com/docker/libcontainer with
github.com/opencontainers/runc/libcontaier.
Also I moved AppArmor profile generation to docker.
Main idea of this update is to fix mounting cgroups inside containers.
After updating docker on CI we can even remove dind.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Check if there is a plugin socket first under `/run/docker/plugins/NAME.sock`.
If there is no socket for a plugin, check `/etc/docker/plugins/NAME.spec` and
`/usr/lib/docker/plugins/NAME.spec` for spec files.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Stephen J Day
Jessie Frazelle
Alexander Morozov
Brian Goff
Qiang Huang
David Calavera
Lei Jitang
Travis Thieman
Shijiang Wei
Clinton Kitson
Mrunal Patel
Doug Davis
Sebastiaan van Stijn
Soshi Katsuta
Tonis Tiigi
Vincent Demeester
Zhang Kun
Alexander Morozov
Arnaud Porterie
Don Kjer
Tibor Vass
John Howard
Eric Windisch
Tim Dettrick
Veres Lajos
Zhang Wei
Stefan Berger
Antonio Murdaca
Josh Hawn
Doug Davis
Antonio Murdaca
Mohammed Aaqib Ansari
Derek McGowan
Yuan Sun
Jessica Frazelle
Alessandro Boch
Kir Kolyshkin
Michael Crosby
Chris Seto
Vincent Batts
Stephen Rust
Diogo Monica
Nathan McCauley
Jessica Frazelle
Shishir Mahajan
Tibor Vass
Tianon Gravi
Phil Estes
Aaron Lehmann
Phil Estes
Ben Firshman
Hu Keping
Dave Tucker