Commit Graph

441 Commits

Author SHA1 Message Date
Brian Goff 37a1fadae6 Merge pull request #21097 from thaJeztah/dont-run-without-udev-sync
Fail when devicemapper doesn't support udev-sync
2016-03-14 21:18:01 -04:00
Vincent Batts bfed97b688 Merge pull request #20786 from rhvgoyal/min-free-space
devmapper: Add a new option dm.min_free_space_percent
2016-03-14 20:10:43 -04:00
David Calavera ed6e33eede Make sure we call every graph init with the same root path.
Remove O(n^2) check for several prior configured drivers.

Signed-off-by: David Calavera <david.calavera@gmail.com>
2016-03-11 17:15:55 -05:00
Sebastiaan van Stijn de64171510 Fail when devicemapper doesn't support udev-sync
Now what we provide dynamic binaries for all plaforms,
we shouldn't try to run docker without udev sync support.

This change changes the previous warning to an Error,
unless the user explicitly overrides the warning, in
which case they're at their own risk.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2016-03-10 19:13:44 +01:00
Vivek Goyal 2e222f69b3 devmapper: Add a new option dm.min_free_space
Once thin pool gets full, bad things can happen. Especially in case of xfs
it is possible that xfs keeps on retrying IO infinitely (for certain kind
of IO) and container hangs. 

One way to mitigate the problem is that once thin pool is about to get full,
start failing some of the docker operations like pulling new images or
creation of new containers. That way user will get warning ahead of time
and can try to rectify it by creating more free space in thin pool. This
can be done either by deleting existing images/containers or by adding more
free space to thin pool.

This patch adds a new option dm.min_free_space to devicemapper graph
driver. Say one specifies dm.min_free_space=10%. This means atleast
10% of data and metadata blocks should be free in pool before new device
creation is allowed, otherwise operation will fail.

By default min_free_space is 10%. User can change it by specifying
dm.min_free_space=X% on command line. A value of 0% will disable the
check.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
2016-03-07 20:27:39 +00:00
Tatsushi Inagaki e8513675a2 Aufs: reduce redundant parsing of mountinfo
Check whether or not the file system type of a mountpoint is aufs
by calling statfs() instead of parsing mountinfo. This assumes
that aufs graph driver does not allow aufs as a backing file
system.

Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>
2016-03-04 11:39:59 +09:00
John Starks 5649030e25 Write Windows layer diffs to tar in standard format
Previously, Windows layer diffs were written using a Windows-internal
format based on the BackupRead/BackupWrite Win32 APIs. This caused
problems with tar-split and tarsum and led to performance problems
in implementing methods such as DiffPath. It also was just an
unnecessary differentiation point between Windows and Linux.

With this change, Windows layer diffs look much more like their
Linux counterparts. They use AUFS-style whiteout files for files
that have been removed, and they encode all metadata directly in
the tar file.

This change only affects Windows post-TP4, since changes to the Windows
container storage APIs were necessary to make this possible.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-03-02 16:13:40 -08:00
David Calavera 8f109829e2 Merge pull request #20475 from Microsoft/jstarks/filegetter
graphdriver: Replace DiffPath with DiffGetter
2016-03-02 08:36:36 -08:00
John Starks 58bec40d16 graphdriver: Replace DiffPath with DiffGetter
This allows a graph driver to provide a custom FileGetter for tar-split
to use. Windows will use this to provide a more efficient implementation
in a follow-up change.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-03-01 10:25:33 -08:00
hsinko 772f5495b7 folders->directories
Signed-off-by: hsinko <21551195@zju.edu.cn>
2016-02-29 21:32:30 -08:00
Brian Goff e386dfc33f fix double-lock
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2016-02-27 09:49:21 -05:00
Brian Goff c2f7777603 Revert "Add finer-grained locking for aufs"
This reverts commit f31014197c.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2016-02-27 08:01:19 -05:00
Brian Goff f31014197c Add finer-grained locking for aufs
```
benchmark                       old ns/op       new ns/op     delta
BenchmarkConcurrentAccess-8     10269529748     26834747      -99.74%

benchmark                       old allocs     new allocs     delta
BenchmarkConcurrentAccess-8     309948         7232           -97.67%

benchmark                       old bytes     new bytes     delta
BenchmarkConcurrentAccess-8     23943576      1578441       -93.41%
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2016-02-25 18:06:41 -05:00
Brian Goff 55c91f2ab9 Fix some issues with concurrency in aufs.
Adds a benchmark to measure performance under concurrent actions.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2016-02-25 14:32:13 -05:00
Stefan Weil 2eee613326 Fix some typos in comments and strings
Most of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-22 20:27:15 +01:00
Kai Qiang Wu(Kennan) c33cdf9ee3 Fix the typo
Signed-off-by: Kai Qiang Wu(Kennan) <wkqwu@cn.ibm.com>
2016-02-16 07:00:01 +00:00
Phil Estes 922986b76e Add proper refcounting to zfs graphdriver
Fixes issues with layer remounting (e.g. a running container which then
has `docker cp` used to copy files in or out) by applying the same
refcounting implementation that exists in other graphdrivers like
overlay and aufs.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
2016-02-11 01:00:54 -05:00
Stefan J. Wernli 041a9510c6 Fixing 'docker save' on Windows.
Save was failing file integrity checksums due to bugs in both
Windows and Docker. This commit includes fixes to file time handling
in tarexport and system.chtimes that are necessary along with
the Windows platform fixes to correctly support save. With this
change, sysfile_backups for windowsfilter driver are no longer
needed, so that code is removed.

Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
2016-02-08 18:08:49 -08:00
David Calavera a93cb2b856 Merge pull request #20045 from estesp/zfs-userns-permissions-fix
Fix ZFS permissions bug with user namespaces
2016-02-05 15:32:43 -08:00
Phil Estes aef0995b02 Fix ZFS permissions bug with user namespaces
Fix root directory of the mountpoint being owned by real root. This is
unique to ZFS because of the way file mountpoints are created using the
ZFS tooling, and the remapping that happens at layer unpack doesn't
impact this root (already created) holding directory for the layer.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
2016-02-05 14:37:33 -05:00
Liu Bo b2e27fee53 Graphdriver/btrfs: Avoid using single d.Get()
For btrfs driver, in d.Create(), Get() of parentDir is called but not followed
by Put().

If we apply SElinux mount label, we need to mount btrfs subvolumes in d.Get(),
without a Put() would end up with a later Remove() failure on
"Device resourse is busy".

This calls the subvolume helper function directly in d.Create().

Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
2016-02-04 10:25:24 -08:00
Tonis Tiigi b9a395c85d Remove case sensitive duplicate dir in vendor
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2016-02-03 13:06:32 -08:00
Kai Qiang Wu(Kennan) feda5d7684 Make btrfs call same interface as others
Most storage drivers call graphdriver.GetFSMagic(home),
it is more clean to easy to maintain. So btrfs need to
adopt such change.

Signed-off-by: Kai Qiang Wu(Kennan) <wkqwu@cn.ibm.com>
2016-02-01 07:50:21 +00:00
Phil Estes 67d86d10d8 Merge pull request #19216 from scaleoutsean/master
Add GPFS filesystem IDs
2016-01-22 21:27:03 -05:00
scaleoutsean bdc8241da9 Add GPFS
Signed-off-by: Sean Lee <seanlee@tw.ibm.com>
2016-01-22 21:12:47 +08:00
Tibor Vass d2448a0c20 Merge pull request #19524 from LK4D4/iocopy_writeto
Use bufio.Reader in io.Copy source for overlay.copyRegular
2016-01-21 15:37:12 -05:00
Alexander Morozov 3f5e1c69b3 Use pools.Copy instead of io.Copy for overlay.copyRegular
That function is pretty heavy used on container start. Autoallocating
buffer can be painful.

Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2016-01-21 08:53:37 -08:00
Tonis Tiigi 98d0997894 Revert "Copy aufs hardlinks to top layer"
This reverts commit ef05b83417.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2016-01-21 07:52:50 -08:00
Sebastiaan van Stijn 661d75f398 Merge pull request #19123 from shishir-a412ed/rootfs_size_configurable
daemon option (--storage-opt dm.basesize) for increasing the base device size on daemon restart
2016-01-13 13:22:08 -08:00
Shishir Mahajan e47112d3e8 daemon option (--storage-opt dm.basesize) for increasing the base device size on daemon restart
Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com>
2016-01-13 13:57:31 -05:00
Vivek Goyal 2dccb562df Mark device ID free only if device actually got deleted
Right now if somebody has enabled deferred device deletion, then
deleteTransaction() returns success even if device could not be deleted. It
has been marked for deferred deletion. Right now we will mark device ID free
and potentially use it again when somebody tries to create new container. And
that's wrong. Device ID is not free yet. It will become free once devices
has actually been deleted by the goroutine later.

So move the location of call to markDeviceIDFree() to a place where we know
device actually got deleted and was not marked for deferred deletion.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
2016-01-11 18:57:37 +00:00
Phil Estes 72e65e8793 Fix btrfs subvolume snapshot dir perms for user namespaces
Make sure btrfs mounted subvolumes are owned properly when a remapped
root exists (user namespaces are enabled, for example)

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
2016-01-07 23:05:28 -05:00
Tõnis Tiigi fe550a1f17 Merge pull request #19093 from estesp/userns-root-rework
Allow root non-userns metadata backwards compatibility
2016-01-06 09:48:37 -08:00
Phil Estes e8532023f2 Allow root non-userns metadata backwards compatibility
Instead of creating a "0.0" subdirectory and migrating graphroot
metadata into it when user namespaces are available in the daemon
(currently only in experimental), change the graphroot dir permissions
to only include the execute bit for "other" users.

This allows easy migration to and from user namespaces and will allow
easier integration of user namespace support into the master build.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
2016-01-05 11:51:14 -05:00
Tonis Tiigi 0641429ad8 Use direct filesystem access for tar-split on aufs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2016-01-04 09:41:02 -08:00
Daniel Nephin 91154e9235 Move graph driver registration out of the daemon package.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2015-12-28 13:02:24 -05:00
Daniel Nephin f5916b10ae Remove the graph driver from the daemon, move it into the layer store.
Support restoreCustomImage for windows with a new interface to extract
the graph driver from the LayerStore.

Signed-off-by: Daniel Nephin <dnephin@docker.com>
2015-12-28 12:55:48 -05:00
Shijiang Wei de7f6cf16b ingnore the NotExist error when removing inexistent files
Signed-off-by: Shijiang Wei <mountkin@gmail.com>
2015-12-25 15:19:48 +08:00
Daniel Nephin 0673361ef6 Remove migrateIfDownlevel and aufs migration from docker pre-0.7
Signed-off-by: Daniel Nephin <dnephin@docker.com>
2015-12-22 16:45:42 -05:00
Vincent Batts af59752712 loopback: separate loop logic from devicemapper
The loopback logic is not technically exclusive to the devicemapper
driver. This reorganizes the code such that the loopback code is usable
outside of the devicemapper package and driver.

Signed-off-by: Vincent Batts <vbatts@redhat.com>
2015-12-18 10:57:43 -05:00
Vincent Batts f57d56350e Merge pull request #18686 from cpuguy83/fix_btrfs_subvol_delete_panic
Fix btrfs recursive btrfs subvol delete
2015-12-16 14:26:40 -05:00
David Calavera b44b5bbc8b Merge pull request #18682 from calavera/replace_units_package
Replace pkg/units with docker/go-units.
2015-12-16 10:48:59 -08:00
David Calavera 4fef42ba20 Replace pkg/units with docker/go-units.
Signed-off-by: David Calavera <david.calavera@gmail.com>
2015-12-16 12:26:49 -05:00
Antonio Murdaca baba1a8493 reorder imports with goimports
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2015-12-16 16:50:25 +01:00
Brian Goff f9befce2d3 Fix btrfs recursive btrfs subvol delete
Really fixing 2 things:

1. Panic when any error is detected while walking the btrfs graph dir on
removal due to no error check.
2. Nested subvolumes weren't actually being removed due to passing in
the wrong path

On point 2, for a path detected as a nested subvolume, we were calling
`subvolDelete("/path/to/subvol", "subvol")`, where the last part of the
path was duplicated due to a logic error, and as such actually causing
point #1 since `subvolDelete` joins the two arguemtns, and
`/path/to/subvol/subvol` (the joined version) doesn't exist.

Also adds a test for nested subvol delete.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2015-12-15 18:12:40 -05:00
Antonio Murdaca f22ee02c6d devmapper: store base device fs type
After the very first init of the graph `docker info` correctly shows the
base fs type under `Backing Filesystem`. This information isn't stored
anywhere. After a restart (w/o erasing `/var/lib/docker`) `docker info`
shows an empty string under `Backing Filesystem`.
This patch records the base fs type after the first run in the metadata
or, to fix old devices that don't have this info in the metadata, just
probe the fs type of the base device at graph startup.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2015-12-15 09:33:19 +01:00
Chris Dituri 0aa6ace6e6 Make daemon/graphdriver/devmapper log messages with a common, consistent prefix.
Closes #16667

Uses the prefix "devmapper:" for all the fmt and logrus error, debug, and info messages.

Signed-off-by: Chris Dituri <csdituri@gmail.com>
2015-12-14 21:35:13 -06:00
Justas Brazauskas 927b334ebf Fix typos found across repository
Signed-off-by: Justas Brazauskas <brazauskasjustas@gmail.com>
2015-12-13 18:04:12 +02:00
Christopher Jones 7c077c2c34 Fixed typo change deivce to device.
This changes deivce to device in daemon, test and docs.

Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
2015-12-10 15:23:05 -06:00
Antonio Murdaca 037cbcec98 devmapper: remove unused var
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2015-12-10 08:28:02 +01:00