Before this patch libcontainer badly errored out with `invalid
argument` or `numerical result out of range` while trying to write
to cpuset.cpus or cpuset.mems with an invalid value provided.
This patch adds validation to --cpuset-cpus and --cpuset-mems flag along with
validation based on system's available cpus/mems before starting a container.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
Add a daemon flag to control this behaviour. Add a warning message when pulling
an image from a v1 registry. The default order of pull is slightly altered
with this changset.
Previously it was:
https v2, https v1, http v2, http v1
now it is:
https v2, http v2, https v1, http v1
Prevent login to v1 registries by explicitly setting the version before ping to
prevent fallback to v1.
Add unit tests for v2 only mode. Create a mock server that can register
handlers for various endpoints. Assert no v1 endpoints are hit with legacy
registries disabled for the following commands: pull, push, build, run and
login. Assert the opposite when legacy registries are not disabled.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Use `pkg/discovery` to provide nodes discovery between daemon instances.
The functionality is driven by two different command-line flags: the
experimental `--cluster-store` (previously `--kv-store`) and
`--cluster-advertise`. It can be used in two ways by interested
components:
1. Externally by calling the `/info` API and examining the cluster store
field. The `pkg/discovery` package can then be used to hit the same
endpoint and watch for appearing or disappearing nodes. That is the
method that will for example be used by Swarm.
2. Internally by using the `Daemon.discoveryWatcher` instance. That is
the method that will for example be used by libnetwork.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
This PR adds a "request ID" to each event generated, the 'docker events'
stream now looks like this:
```
2015-09-10T15:02:50.000000000-07:00 [reqid: c01e3534ddca] de7c5d4ca927253cf4e978ee9c4545161e406e9b5a14617efb52c658b249174a: (from ubuntu) create
```
Note the `[reqID: c01e3534ddca]` part, that's new.
Each HTTP request will generate its own unique ID. So, if you do a
`docker build` you'll see a series of events all with the same reqID.
This allow for log processing tools to determine which events are all related
to the same http request.
I didn't propigate the context to all possible funcs in the daemon,
I decided to just do the ones that needed it in order to get the reqID
into the events. I'd like to have people review this direction first, and
if we're ok with it then I'll make sure we're consistent about when
we pass around the context - IOW, make sure that all funcs at the same level
have a context passed in even if they don't call the log funcs - this will
ensure we're consistent w/o passing it around for all calls unnecessarily.
ping @icecrime @calavera @crosbymichael
Signed-off-by: Doug Davis <dug@us.ibm.com>
- Print the mount table as in /proc/self/mountinfo
- Do not exit prematurely when one of the ipc mounts doesn't exist.
- Do not exit prematurely when one of the ipc mounts cannot be unmounted.
- Add a unit test to see if the cleanup really works.
- Use syscall.MNT_DETACH to cleanup mounts after a crash.
- Unmount IPC mounts when the daemon unregisters an old running container.
Signed-off-by: David Calavera <david.calavera@gmail.com>
If an invalid logger address is provided on daemon start it will
silently fail. As syslog driver is doing, this check should be done on
daemon start and prevent it from starting even in other drivers.
This patch also adds integration tests for this behavior.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
If you don't have cgroup swap memory support, `dockerCmd`'s output in
these tests will be polluted by a warning from the daemon and will fail
the tests.
No need to have memory swap support for these tests to run as it will
be reset to -1 and everything will continue correctly.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
`docker rename foo ''` would result in:
```
usage: docker rename OLD_NAME NEW_NAME
```
which is the old engine's way of return errors - yes that's in the
daemon code. So I fixed that error msg to just be normal.
While doing that I noticed that using an empty string for the
source container name failed but didn't print any error message at all.
This is because we would generate a URL like: ../containers//rename/..
which would cause a 301 redirect to ../containers/rename/..
however the CLI code doesn't actually deal with 301's - it just ignores
them and returns back to the CLI code/caller.
Rather than changing the CLI to deal with 3xx error codes, which would
probably be a good thing to do in a follow-on PR, for this immediate
issue I just added a cli-side check for empty strings for both old and
new names. This way we catch it even before we hit the daemon.
API callers will get a 404, assuming they follow the 301, for the
case of the src being empty, and the new error msg when the destination
is empty - so we should be good now.
Add tests for both cases too.
Signed-off-by: Doug Davis <dug@us.ibm.com>
Fixes#11957Fixes#12319
Also removes check for Darwin when the stdin reader is closed as it
doesn't appear to block any more.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
GET /containers/json route used to reply with and empty array `[]` when no
containers where available. Daemon containers list refactor introduced
this bug by declaring an empty slice istead of initializing it as well
and it was now replying with `null`.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
- refactor to make it easier to split the api in the future
- addition to check the existing test case and make sure it contains
some expected output
Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
- The build-time variables are passed as environment-context for command(s)
run as part of the RUN primitve. These variables are not persisted in environment of
intermediate and final images when passed as context for RUN. The build environment
is prepended to the intermediate continer's command string for aiding cache lookups.
It also helps with build traceability. But this also makes the feature less secure from
point of view of passing build time secrets.
- The build-time variables also get used to expand the symbols used in certain
Dockerfile primitves like ADD, COPY, USER etc, without an explicit prior definiton using a
ENV primitive. These variables get persisted in the intermediate and final images
whenever they are expanded.
- The build-time variables are only expanded or passed to the RUN primtive if they
are defined in Dockerfile using the ARG primitive or belong to list of built-in variables.
HTTP_PROXY, HTTPS_PROXY, http_proxy, https_proxy, FTP_PROXY and NO_PROXY are built-in
variables that needn't be explicitly defined in Dockerfile to use this feature.
Signed-off-by: Madhav Puri <madhav.puri@gmail.com>
For now docker stats will sum the rxbytes, txbytes, etc. of all
the interfaces.
It is OK for the output of CLI `docker stats` but not good for
the API response, especially when the container is in sereval
subnets.
It's better to leave these origianl data to user.
Signed-off-by: Hu Keping <hukeping@huawei.com>
Notary is capable of acting in offline mode, making use of cache TUF data.
When ping is not successful, notary should still be attempted without error.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
If a logdriver doesn't register a callback function to validate log
options, it won't be usable. Fix the journald driver by adding a dummy
validator.
Teach the client and the daemon's "logs" logic that the server can also
supply "logs" data via the "journald" driver. Update documentation and
tests that depend on error messages.
Add support for reading log data from the systemd journal to the
journald log driver. The internal logic uses a goroutine to scan the
journal for matching entries after any specified cutoff time, formats
the messages from those entries as JSONLog messages, and stuffs the
results down a pipe whose reading end we hand back to the caller.
If we are missing any of the 'linux', 'cgo', or 'journald' build tags,
however, we don't implement a reader, so the 'logs' endpoint will still
return an error.
Make the necessary changes to the build setup to ensure that support for
reading container logs from the systemd journal is built.
Rename the Jmap member of the journald logdriver's struct to "vars" to
make it non-public, and to make it easier to tell that it's just there
to hold additional variable values that we want journald to record along
with log data that we're sending to it.
In the client, don't assume that we know which logdrivers the server
implements, and remove the check that looks at the server. It's
redundant because the server already knows, and the check also makes
using older clients with newer servers (which may have new logdrivers in
them) unnecessarily hard.
When we try to "logs" and have to report that the container's logdriver
doesn't support reading, send the error message through the
might-be-a-multiplexer so that clients which are expecting multiplexed
data will be able to properly display the error, instead of tripping
over the data and printing a less helpful "Unrecognized input header"
error.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
This PR makes a user visible behavior change with userland
proxy disabled by default and rely on hairpin NAT to be enabled
by default. This may not work in older (unsupported) kernels
where the user will be forced to enable userlandproxy if needed.
- Updated the Docs
- Changed the integration-cli to start with userlandproxy
desiabled by default.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
Before, this only waited for the download to complete. There was no
guarantee that the layer had been registered in the graph and was ready
use. This is especially problematic with v2 pulls, which wait for all
downloads before extracting layers.
Change Broadcaster to allow an error value to be propagated from Close
to the waiters.
Make the wait stop when the extraction is finished, rather than just the
download.
This also fixes v2 layer downloads to prefix the pool key with "layer:"
instead of "img:". "img:" is the wrong prefix, because this is what v1
uses for entire images. A v1 pull waiting for one of these operations to
finish would only wait for that particular layer, not all its
dependencies.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Allow to set the signal to stop a container in `docker run`:
- Use `--stop-signal` with docker-run to set the default signal the container will use to exit.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Allows people to create out-of-process graphdrivers that can be used
with Docker.
Extensions must be started before Docker otherwise Docker will fail to
start.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- utils_test.go and docker_utils_test.go
- Moved docker related function to docker_utils.go
- add a test for integration-cli/checker
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Closes#16146
While in there, modified the testing infrastructure for the help text
so that we can get commands with nested commands - like "volume".
Signed-off-by: Doug Davis <dug@us.ibm.com>
The docker volume ls -f dangling=true filter was
inverted; the filtered results actually returned all
non-dangling volumes.
This fixes the filter and adds some integration tests
to test the correct behavior.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixed issues related to network subcommand tests
- "network" in exempted list of short help check
- Condition for exact test modified to meet experimental commands
- Sorting of commands done in flags_experimental
Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>
Update and migrate existing tests to the `DockerHubPullSuite`. Most
tests were preserved, but refactored and made more exhaustive. One test
was deliberately removed (`TestPullVerified`) as it is unreliable and
that the feature was obsoleted by content trust.
Move all trust related tests to `docker_cli_pull_trusted_test.go`.
Move tests depending on a local registry to `docker_cli_pull_local_test.go`.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
Add a `checker` package that adds some utility Checker implementation,
the first one being `checker.Contains`, as well as brining all go-check
provided Checker implementations in scope as a commodity.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
Introduce the `DockerHubPullSuite` that interacts with its own dedicated
daemon, thus allowing to start from a clean environment and finely test
against the impact of isolated push and pull operations.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
When using a named volume without --volume-driver, the driver was
hardcoded to "local".
Even when the volume was already created by some other driver (and
visible in `docker volume ls`), the container would store in it's own
config that it was the `local` driver.
The external driver would work perfecly fine until the daemon is
restarted, at which point the `local` driver was assumed because that is
as it was set in the container config.
Set the bind driver to the driver returned by createVolume.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Separate container iteration, filtering and reducing.
This will make easier in the future to improve the implementation of
docker ps as we know it.
The end goal is to unify the objects returned by the api for docker ps
and docker inspect, leaving all docker ps transformations to the client.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This file was not well documented and had very high cyclomatic complexity.
This patch completely rearranges this file and the ImageDelete method to
be easier to follow and more maintainable in the future.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Makes it possible to filter containers by image, using
--filter=ancestor=busybox and get all the container running busybox
image and image based on busybox (to the bottom).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
The practice of buffering to a tempfile during a pushing contributes massively
to slow V2 push performance perception. The protocol was actually designed to
avoid precalculation, supporting cut-through data push. This means we can
assemble the layer, calculate its digest and push to the remote endpoint, all
at the same time.
This should increase performance massively on systems with slow disks or IO
bottlenecks.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This fixes two problems:
1. docker run --device /dev/sda:rw ubuntu bash doesn't work
2. --device /dev/zero:/dev/noro:ro doesn't show clear error message,
but fail when writing to cgroup file.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Signed-off-by: Zhang Kun <zkazure@gmail.com>
goformat
Signed-off-by: Zhang Kun <zkazure@gmail.com>
fix small
Signed-off-by: Zhang Kun <zkazure@gmail.com>
change to rm
Signed-off-by: Zhang Kun <zkazure@gmail.com>
handler other error
Signed-off-by: Zhang Kun <zkazure@gmail.com>
unique ERR
Signed-off-by: Zhang Kun <zkazure@gmail.com>
setHostConfig_fail_test
Signed-off-by: Zhang Kun <zkazure@gmail.com>
format
Signed-off-by: Zhang Kun <zkazure@gmail.com>
err handle and modify test
Signed-off-by: Zhang Kun <zkazure@gmail.com>
golint error
Signed-off-by: Zhang Kun <zkazure@gmail.com>
Docker ps default format should not take precedence over cli flags.
This happens effectively for other flags except `-q`.
We need to let the cli to set the format as table to print the
expected output with `-q`.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Changing vendor/src/github.com/docker/libnetwork to match lindenlab/libnetwork custom-host-port-ranges-1.7 branch
The bool logic around setting up the TTY ended up getting flipped
accidentally.
Also added a test for exec with TTY.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Introduce a write denial for files at the root of /proc.
This prohibits root users from performing a chmod of those
files. The rules for denials in proc are also cleaned up,
making the rules better match their targets.
Locally tested on:
- Ubuntu precise (12.04) with AppArmor 2.7
- Ubuntu trusty (14.04) with AppArmor 2.8.95
Signed-off-by: Eric Windisch <eric@windisch.us>
Make command like "docker images ubuntu:14.04" work and filter out the
image with the given tag.
Closes#8048.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This reverts commit 40b71adee3.
Original commit (for which this is effectively a rebased version) is
72a500e9e5 and was provided by Lei Jitang
<leijitang@huawei.com>.
Signed-off-by: Tim Dettrick <t.dettrick@uq.edu.au>
Have network files mounted read-only when mounted using the -v
open and -v parameter has 'ro' passed.
Signed-off-by: Stefan Berger <stefanb@us.ibm.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The process of pulling an image spawns a new goroutine for each layer in the
image manifest. If any of these downloads fail we would stop everything and
return the error, even though other goroutines would still be running and
writing output through a progress reader which is attached to an http response
writer. Since the request handler had already returned from the first error,
the http server panics when one of these download goroutines makes a write to
the response writer buffer.
This patch prevents this crash in the daemon http server by waiting for all of
the download goroutines to complete, even if one of them fails. Only then does
it return, terminating the request handler.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Don't assume that any random image will have 'top' and explicitely use
the busybox image for testing.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
We noticed a regression since the 1.7.1 patch after some refactoring. This
patch corrects the behavior and adds integration tests for modified manifest
and rootfs layer blobs.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
manually stopped
If a container is running with a restart policy of always and it's
manually stopped, then on daemon restart it will be running.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
[pkg/archive] Update archive/copy path handling
- Remove unused TarOptions.Name field.
- Add new TarOptions.RebaseNames field.
- Update some of the logic around path dir/base splitting.
- Update some of the logic behind archive entry name rebasing.
[api/types] Add LinkTarget field to PathStat
[daemon] Fix stat, archive, extract of symlinks
These operations *should* resolve symlinks that are in the path but if the
resource itself is a symlink then it *should not* be resolved. This patch
puts this logic into a common function `resolvePath` which resolves symlinks
of the path's dir in scope of the container rootfs but does not resolve the
final element of the path. Now archive, extract, and stat operations will
return symlinks if the path is indeed a symlink.
[api/client] Update cp path hanling
[docs/reference/api] Update description of stat
Add the linkTarget field to the header of the archive endpoint.
Remove path field.
[integration-cli] Fix/Add cp symlink test cases
Copying a symlink should do just that: copy the symlink NOT
copy the target of the symlink. Also, the resulting file from
the copy should have the name of the symlink NOT the name of
the target file.
Copying to a symlink should copy to the symlink target and not
modify the symlink itself.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[v2: a separate aufs commit is merged into this one]
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
The 'deny ptrace' statement was supposed to only ignore
ptrace failures in the AUDIT log. However, ptrace was implicitly
allowed from unconfined processes (such as the docker daemon and
its integration tests) due to the abstractions/base include.
This rule narrows the definition such that it will only ignore
the failures originating inside of the container and will not
cause denials when the daemon or its tests ptrace inside processes.
Introduces positive and negative tests for ptrace /w apparmor.
Signed-off-by: Eric Windisch <eric@windisch.us>
Integration tests were failing due to proc filter behavior
changes with new apparmor policies.
Also include the missing docker-unconfined policy resolving
potential startup errors. This policy is complain-only so
it should behave identically to the standard unconfined policy,
but will not apply system path-based policies within containers.
Signed-off-by: Eric Windisch <eric@windisch.us>
Currently some notary tests change the system clock to check for expiration.
Skip these tests until the code can be refactored to not rely on updating the system clock.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Builds where the base images have been resolved to trusted digest
references will now be tagged with the original tag reference from
the Dockerfile on a successful build.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Will attempt to load profiles automatically. If loading fails
but the profiles are already loaded, execution will continue.
A hard failure will only occur if Docker cannot load
the profiles *and* they have not already been loaded via
some other means.
Also introduces documentation for AppArmor.
Signed-off-by: Eric Windisch <eric@windisch.us>
Return an error when the container is stopped only in api versions
equal or greater than 1.20 (docker 1.8).
Signed-off-by: David Calavera <david.calavera@gmail.com>
Miloslav Trmač
Alexander Morozov
John Howard
Yuan Sun
Antonio Murdaca
Richard Scothern
Arnaud Porterie
Lei Jitang
David Calavera
Jess Frazelle
Doug Davis
Hu Keping
Kun Zhang
Darren Stahl
qhuang
Brian Goff
Evan Hazlett
Tibor Vass
Alexandre Beslic
Morgan Bauer
Antonio Murdaca
Tim Hockin
Srini Brahmaroutu
Sebastiaan van Stijn
Jessica Frazelle
Madhav Puri
Derek McGowan
Vincent Demeester
Nalin Dahyabhai
Jana Radhakrishnan
Mrunal Patel
Aaron Lehmann
Rozhnov Alexandr
Sebastiaan van Stijn
Kunal Kushwaha
Michael Crosby
Alessandro Boch
Stephen Rust
Josh Hawn
Tonis Tiigi
Taylor Jones
Stephen J Day
Qiang Huang
Travis Thieman
Shijiang Wei
Clinton Kitson
Doug Davis
Soshi Katsuta
Alexander Morozov
Don Kjer
Eric Windisch
Tim Dettrick
Darren Shepherd
Veres Lajos
Zhang Wei
Stefan Berger
Mohammed Aaqib Ansari
Kir Kolyshkin