These settings need to be in the HostConfig so that they are not
committed to an image and cannot introduce a security issue.
We can safely move this field from the Config to the HostConfig
without any regressions because these settings are consumed at container
created and used to populate fields on the Container struct. Because of
this, existing settings will be honored for containers already created
on a daemon with custom security settings and prevent values being
consumed via an Image.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Conflicts:
daemon/create.go
changing config to hostConfig was required to fix the
build
Adds pertitent information about what is expected in the json payload
and comments out unsupported (exec) features in runConfig.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Next steps, in another PR, would be:
- make all logging go through the logrus stuff
- I'd like to see if we can remove the env var stuff (like DEBUG) but we'll see
Closes#5198
Signed-off-by: Doug Davis <dug@us.ibm.com>
TreeSize uses syscall.Stat_t which is not available on Windows.
It's called only on daemon path, therefore extracting it to daemon
with build tag 'daemon'
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Fixes#8942
Current behavior is that volumes aren't initialized until start.
Volumes still need to be initialized on start since VolumesFrom and
Binds can be passed in as part of HostConfig on start, however anything
that's already been initialized will just be skipped as is the current
behavior.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Fixes#1171Fixes#6465
Data passed to mount(2) is clipped to PAGE_SIZE if its bigger. Previous
implementation checked if error was returned and then started to append layers
one by one. But if the PAGE_SIZE clipping appeared in between the paths, in the
permission sections or in xino definition the call would not error and
remaining layers would just be skipped(or some other unknown situation).
This also optimizes system calls as it tries to mount as much as possible with
the first mount.
Signed-off-by: Tõnis Tiigi <tonistiigi@gmail.com> (github: tonistiigi)
Ideally lvm2 would be used to create/manage the thin-pool volume that is
then handed to docker to exclusively create/manage the thin and thin
snapshot volumes needed for it's containers. Managing the thin-pool
outside of docker makes for the most feature-rich method of having
docker utilize device mapper thin provisioning as the backing storage
for docker's containers. lvm2-based thin-pool management feature
highlights include: automatic or interactive thin-pool resize support,
dynamically change thin-pool features, automatic thinp metadata checking
when lvm2 activates the thin-pool, etc.
Docker will not activate/deactivate the specified thin-pool device but
it will exclusively manage/create thin and thin snapshot volumes in it.
Docker will not take ownership of the specified thin-pool device unless
it has 0 data blocks used and a transaction id of 0. This should help
guard against using a thin-pool that is already in use.
Also fix typos in setupBaseImage() relative to the thin volume type of
the base image.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Some workloads rely on IPC for communications with other processes. We
would like to split workloads between two container but still allow them
to communicate though shared IPC.
This patch mimics the --net code to allow --ipc=host to not split off
the IPC Namespace. ipc=container:CONTAINERID to share ipc between containers
If you share IPC between containers, then you need to make sure SELinux labels
match.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
Took care of some review comments from crosbymichael.
v2:
- Return "err = nil" if file deviceset-metadata file does not exist.
- Use json.Decoder() interface for loading deviceset metadata.
v3:
- Reverted back to json marshal interface in loadDeviceSetMetaData().
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
This passed the --net=container:CONTINER_ID to lxc-start as --share-net
Docker-DCO-1.1-Signed-off-by: Abin Shahab <ashahab@altiscale.com> (github: ashahab-altiscale)
Running parseVolumesFromSpec on all VolumesFrom specs before initialize
any mounts endures that we don't leave container.Volumes in an
inconsistent (partially initialized) if one of out mount groups is not
available (e.g. the container we're trying to mount from does not
exist).
Keeping container.Volumes in a consistent state ensures that next time
we Start() the container, it'll run prepareVolumes() again.
The attached test demonstrates that when a container fails to start due
to a missing container specified in VolumesFrom, it "remembers" a Volume
that worked.
Fixes: #8726
Signed-off-by: Thomas Orozco <thomas@orozco.fr>
Since the containers can handle the out of memory kernel kills gracefully, docker
will only provide out of memory information as an additional metadata as part of
container status.
Docker-DCO-1.1-Signed-off-by: Vishnu Kannan <vishnuk@google.com> (github: vishh)
In previous patch I had introduce json:"-" tags to be on safer side to make
sure certain fields are not marshalled/unmarshalled. But struct fields
starting with small letter are not exported so they will not be marshalled
anyway. So remove json:"-" tags from there.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Lxc driver was throwing errors for mounts where the mount point does not exist in the container.
This adds a create=dir/file mount option to the lxc template, to alleviate this issue.
Docker-DCO-1.1-Signed-off-by: Abin Shahab <ashahab@altiscale.com> (github: ashahab-altiscale)
We removed the syncpipe package and replaced it with specific calls to
create a new *os.File from a specified fd passed to the process. This
reduced code and an extra object to manage the container's init
lifecycle.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
My pull request failed the build due to gofmat issues. I have run gofmt
on specified files and this commit fixes it.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
The way thin-pool right now is designed, user space is supposed to keep
track of what device ids have already been used. If user space tries to
create a new thin/snap device and device id has already been used, thin
pool retuns -EEXIST.
Upon receiving -EEXIST, current docker implementation simply tries the
NextDeviceId++ and keeps on doing this till it finds a free device id.
This approach has two issues.
- It is little suboptimal.
- If device id already exists, current kenrel implementation spits out
a messsage on console.
[17991.140135] device-mapper: thin: Creation of new snapshot 33 of device 3 failed.
Here kenrel is trying to tell user that device id 33 has already been used.
And this shows up for every device id docker tries till it reaches a point
where device ids are not used. So if there are thousands of container and
one is trying to create a new container after fresh docker start, expect
thousands of such warnings to flood console.
This patch saves the NextDeviceId in a file in
/var/lib/docker/devmapper/metadata/deviceset-metadata and reads it back
when docker starts. This way we don't retry lots of device ids which
have already been used.
There might be some device ids which are free but we will get back to them
once device numbers wrap around (24bit limit on device ids).
This patch should cut down on number of kernel warnings.
Notice that I am creating a deviceset metadata file which is a global file
for this pool. So down the line if we need to save more data we should be
able to do that.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
I was trying to save nextDeviceId to a file but it would not work and
json.Marshal() will do nothing. Then some search showed that I need to
make first letter of struct field capital, exporting this field and
now json.Marshal() works.
This is a preparatory patch for the next one.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Currently we save device metadata and have a helper function saveMetadata()
which converts data in json format as well as saves it to file. For
converting data in json format, one needs to know what is being saved.
Break this function down in two functions. One function only has file
write capability and takes in argument about byte array of json data.
Now this function does not have to know what data is being saved. It
only knows about a stream of json data is being saved to a file.
This allows me to reuse this function to save a different type of
metadata. In this case I am planning to save NextDeviceId so that
docker can use this device Id upon next restart. Otherwise docker
starts from 0 which is suboptimal.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Fixed the following errors:
1. Request(0) causes a dead loop when the map is full and map.last == BEGIN.
2. When map.last is the only available port (or ip), Request(0) returns ErrAllPortsAllocated (or ErrNoAvailableIPs). Exception is when map.last == BEGIN.
Signed-off-by: shuai-z <zs.broccoli@gmail.com>
Fixes#8832
All stdio streams need to finish writing before the
connection can be closed.
Signed-off-by: Tõnis Tiigi <tonistiigi@gmail.com> (github: tonistiigi)
If we need to raise an error, make sure the internal state is clean, because
a successful driver.Get() may have its internal state changed (eg. counting,
or mounts), while callers will only do that after a succussful Mount().
Signed-off-by: shuai-z <zs.broccoli@gmail.com>
In an effort to make layer content 'stable' between import
and export from two different graph drivers, we must resolve
an issue where AUFS produces metadata files in its layers
which other drivers explicitly ignore when importing.
The issue presents itself like this:
- Generate a layer using AUFS
- On commit of that container, the new stored layer contains
AUFS metadata files/dirs. The stored layer content has some
tarsum value: '1234567'
- `docker save` that image to a USB drive and `docker load`
into another docker engine instance which uses another
graph driver, say 'btrfs'
- On load, this graph driver explicitly ignores any AUFS metadata
that it encounters. The stored layer content now has some
different tarsum value: 'abcdefg'.
The only (apparent) useful aufs metadata to keep are the psuedo link
files located at `/.wh..wh.plink/`. Thes files hold information at the
RW layer about hard linked files between this layer and another layer.
The other graph drivers make sure to copy up these psuedo linked files
but I've tested out a few different situations and it seems that this
is unnecessary (In my test, AUFS already copies up the other hard linked
files to the RW layer).
This changeset adds explicit exclusion of the AUFS metadata files and
directories (NOTE: not the whiteout files!) on commit of a container
using the AUFS storage driver.
Also included is a change to the archive package. It now explicitly
ignores the root directory from being included in the resulting tar archive
for 2 reasons: 1) it's unnecessary. 2) It's another difference between
what other graph drivers produce when exporting a layer to a tar archive.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This backend uses the overlayfs union filesystem for containers
plus hard link file sharing for images.
Each container/image can have a "root" subdirectory which is a plain
filesystem hierarchy, or they can use overlayfs.
If they use overlayfs there is a "upper" directory and a "lower-id"
file, as well as "merged" and "work" directories. The "upper"
directory has the upper layer of the overlay, and "lower-id" contains
the id of the parent whose "root" directory shall be used as the lower
layer in the overlay. The overlay itself is mounted in the "merged"
directory, and the "work" dir is needed for overlayfs to work.
When a overlay layer is created there are two cases, either the
parent has a "root" dir, then we start out with a empty "upper"
directory overlaid on the parents root. This is typically the
case with the init layer of a container which is based on an image.
If there is no "root" in the parent, we inherit the lower-id from
the parent and start by making a copy if the parents "upper" dir.
This is typically the case for a container layer which copies
its parent -init upper layer.
Additionally we also have a custom implementation of ApplyLayer
which makes a recursive copy of the parent "root" layer using
hardlinks to share file data, and then applies the layer on top
of that. This means all chile images share file (but not directory)
data with the parent.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
The vfs storage driver currently shells out to the `cp` binary on the host
system to perform an 'archive' copy of the base image to a new directory.
The archive option preserves the modified time of the files which are created
but there was an issue where it was unable to preserve the modified time of
copied symbolic links on some host systems with an outdated version of `cp`.
This change no longer relies on the host system implementation and instead
utilizes the `CopyWithTar` function found in `pkg/archive` which is used
to copy from source to destination directory using a Tar archive, which
should correctly preserve file attributes.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
If we first request port 49153 (BeginPortRange) explicitly, and later some time request the next free port (of same ip/proto) by calling RequestPort() with port number 0, we will again get 49153 returned, even if it's currently in use. Because findPort() blindly retured BeginPortRange the first run, without checking if it has already been taken.
Signed-off-by: shuai-z <zs.broccoli@gmail.com>
when a container failed to start, saves the error message into State.Error so
that it can be retrieved when calling `docker inspect` instead of having to
look at the log
Docker-DCO-1.1-Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com> (github: dqminh)
Never close attached stream before both stdout and stderr have written
all their buffered contents. Remove stdinCloser because it is not needed
any more as the stream is closed anyway after attach has finished.
Fixes#3631
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
Fixes#1992
Right now when you `docker cp` a path which is in a volume, the cp
itself works, however you end up getting files that are in the
container's fs rather than the files in the volume (which is not in the
container's fs).
This makes it so when you `docker cp` a path that is in a volume it
follows the volume to the real path on the host.
archive.go has been modified so that when you do `docker cp mydata:/foo
.`, and /foo is the volume, the outputed folder is called "foo" instead
of the volume ID (because we are telling it to tar up
`/var/lib/docker/vfs/dir/<some id>` and not "foo", but the user would be
expecting "foo", not the ID
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Stable IPs causes some regressions in the way people use Docker, see GH#8493.
Reverting it for 1.3, we'll enable it back for the next release.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
When a container is restarted all the volume configs are parsed again.
Even if the volume was already handled in a previous start it was still
calling "FindOrCreateVolume" on the volume repo causing a new volume to
be created.
This wasn't being detected because as part of the mount initialization
it checks to see if the the _mount_ was already initialized, but this
happens after the parsing of the configs.
So a check is added during parsing to skip a volume which was already
created for that container.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Volume refs were not being restored on daemon restart.
This made it possible to remove a volume being used by other containers
after a daemon restart.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When stdout/stderr is closed prematurely, the proxy's writes to stdout/stderr
(i.e. `log.Errorf/log.Printf`) will returns with EPIPE error, and go runtime
will terminate the proxy when stdout/stderr writes trigger 10 EPIPE errors.
instead of using stdout/stderr as the status handler, we pass an extra file to
the child process and write `0\n` or `1\nerror message` to it and close it
after. This allow the child process to handle stdout/stderr as normal.
Docker-DCO-1.1-Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com> (github: dqminh)
Currently if you start the docker -d on a system with 127.0.0.1 in /etc/resolv.conf
It will set the default dns to 8.8.8.8 8.8.4.4 permanently.
This causes a problem at boot on Fedora machines where NetworkManager has not
populated /etc/resolv.conf before docker gets started.
This fix checks /etc/resolv.conf on every docker run. And only populates
daemon.config.Dns if the user specified it on the command line.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
The defer logic was a little tricky and was hiding one bug: `err` was
being redefined (with `:=`) and thus it escaped the defer error checking
logic.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Prior to the volumes re-factor, data was not being copied on
volumes-from or host-mounted volumes.
After the re-factor, data was being copied for volumes-from.
This reverts this unintentional change in behavior.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Needed to check if the mode was invalid and return error, not valid and
return error.
This didn't get picked up because the existing integration-cli tests
were all either expecting errors when a valid mode was passed in (e.g.
"ro" passed in, we expected an error because it was testing write). So
modified a test which was testing for "rw" to actually pass in "rw"
instead of assuming the "rw"
Docker-DCO-1.1-Signed-off-by: Brian Goff <bgoff@cpuguy83-mbp.home> (github: cpuguy83)
This change will allocate network settings (IP and public ports) at
container creation rather than start and keep them throughout the
lifetime of the container (i.e. until it gets destroyed) instead of
discarding them when the container is stopped.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Since we are moving network allocation outside of container scope (it
will be managed by create/destroy), those functions need to be
accessible from the outside.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Right now, MAC addresses are randomly generated by the kernel when
creating the veth interfaces.
This causes different issues related to ARP, such as #4581, #5737 and #8269.
This change adds support for consistent MAC addresses, guaranteeing that
an IP address will always end up with the same MAC address, no matter
what.
Since IP addresses are already guaranteed to be unique by the
IPAllocator, MAC addresses will inherit this property as well for free.
Consistent mac addresses is also a requirement for stable networking (#8297)
since re-using the same IP address on a different MAC address triggers the ARP
issue.
Finally, this change makes the MAC address accessible through docker
inspect, which fixes#4033.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Add support for pulling signed images from a version 2 registry.
Only official images within the library namespace will be pull from the
new registry and check the build signature.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
This is the first of two steps to break the archive package's dependence
on utils so that archive may be moved into pkg. Also, the `Go()`
function is small, concise, and not specific to the docker internals, so
it is a good candidate for pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
security-opts will allow you to customise the security subsystem.
For example the labeling system like SELinux will run on a container.
--security-opt="label:user:USER" : Set the label user for the container
--security-opt="label:role:ROLE" : Set the label role for the container
--security-opt="label:type:TYPE" : Set the label type for the container
--security-opt="label:level:LEVEL" : Set the label level for the container
--security-opt="label:disabled" : Turn off label confinement for the container
Since we are passing a list of string options instead of a space separated
string of options, I will change function calls to use InitLabels instead of
GenLabels. Genlabels interface is Depracated.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
This also removes dead code in the native driver for a past feature that
was never fully implemented.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Since RemoveLocalDns patch will remove all localhost entries
from resolv.conf we no longer need anything more then
!bytes.Contains(resolvConf, []byte("nameserver")
To check for no nameserver entry in dns config.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
We have a bug report complaining about docker dumping the contents of the
hosts resolv.conf if it container 127.0.0.1. They asked that instead
of dropping the file altogether, that we just remove the line.
This patch removes the 127.0.0.1 lines, if they exist and then
checks if any nameserver lines exist.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
Currently, the HostConfig is only passed from the CLI to Docker only
when issuing a docker create, but not when doing a docker run.
In the near future, in order to allocate ports at creation time rather
than start time, we will need to have the HostConfig readily available
at container creation.
This PR makes the client always pass the HostConfig when creating a
container (regardless of whether it's for a run or create).
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
Since it is possible to request a specific IP, IPAllocator has to verify
that the request is within boundaries.
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
On Fedora and RHEL we ship selinux-enabled flag in the docker.service config,
but if people setup the /var/lib/docker as btrfs and disable SELinux,
we should not block the daemon from running.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
If `--storage-opt dm.datadev=/dev/loop0 --storage-opt
dm.metadatadev=/dev/loop1 ` were provided, the information was not
reflected in the information output.
Closes: #7137
Signed-off-by: Vincent Batts <vbatts@redhat.com>
unclejack
Michael Crosby
Jessie Frazelle
Mike Snitzer
Brian Goff
Michael Crosby
Vincent Batts
Victor Vieux
Alexander Morozov
Jessica Frazelle
Tibor Vass
Alexandr Morozov
Tonis Tiigi
Arnaud Porterie
Doug Davis
Ahmet Alp Balkan
unclejack
Vincent Batts
Sami Wagiaalla
Vishnu Kannan
Dan Walsh
Vivek Goyal
Daniel, Dao Quang Minh
Abin Shahab
Thomas Orozco
Erik Hollensbe
Blake Geno
shuai-z
Malte Janduda
Srini Brahmaroutu
Lei Jitang
Victor Vieux
Josh Hawn
Alexander Larsson
Andrea Luzzardi
Phil Estes
Andy Goldstein
Albert Zhang
Brian Goff
Derek McGowan
ArikaChen
Rafe Colton
Tianon Gravi