In an effort to make layer content 'stable' between import
and export from two different graph drivers, we must resolve
an issue where AUFS produces metadata files in its layers
which other drivers explicitly ignore when importing.
The issue presents itself like this:
- Generate a layer using AUFS
- On commit of that container, the new stored layer contains
AUFS metadata files/dirs. The stored layer content has some
tarsum value: '1234567'
- `docker save` that image to a USB drive and `docker load`
into another docker engine instance which uses another
graph driver, say 'btrfs'
- On load, this graph driver explicitly ignores any AUFS metadata
that it encounters. The stored layer content now has some
different tarsum value: 'abcdefg'.
The only (apparent) useful aufs metadata to keep are the psuedo link
files located at `/.wh..wh.plink/`. Thes files hold information at the
RW layer about hard linked files between this layer and another layer.
The other graph drivers make sure to copy up these psuedo linked files
but I've tested out a few different situations and it seems that this
is unnecessary (In my test, AUFS already copies up the other hard linked
files to the RW layer).
This changeset adds explicit exclusion of the AUFS metadata files and
directories (NOTE: not the whiteout files!) on commit of a container
using the AUFS storage driver.
Also included is a change to the archive package. It now explicitly
ignores the root directory from being included in the resulting tar archive
for 2 reasons: 1) it's unnecessary. 2) It's another difference between
what other graph drivers produce when exporting a layer to a tar archive.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Fixes#1992
Right now when you `docker cp` a path which is in a volume, the cp
itself works, however you end up getting files that are in the
container's fs rather than the files in the volume (which is not in the
container's fs).
This makes it so when you `docker cp` a path that is in a volume it
follows the volume to the real path on the host.
archive.go has been modified so that when you do `docker cp mydata:/foo
.`, and /foo is the volume, the outputed folder is called "foo" instead
of the volume ID (because we are telling it to tar up
`/var/lib/docker/vfs/dir/<some id>` and not "foo", but the user would be
expecting "foo", not the ID
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
This is the second of two steps to break the archive package's
dependence on utils so that archive may be moved into pkg. `Matches()`
is also a good candidate pkg in that it is small, concise, and not
specific to docker internals
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
This is the first of two steps to break the archive package's dependence
on utils so that archive may be moved into pkg. Also, the `Go()`
function is small, concise, and not specific to the docker internals, so
it is a good candidate for pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
When read is called on a tarsum with a two different read sizes, specifically the second call larger than the first, the dynamic buffer does not get reallocated causing a slice read error.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Not breaking the default cipher and algorithm for calculating checksums,
but allow for using alternate block ciphers during the checksum
calculation.
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
If a tar were constructed with duplicate file names, then depending on
the order, it could result in same tarsum.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Tarsum now correctly closes the internal TarWriter which appends
a block of 1024 zeros to the end of the returned archive.
Signed-off-by: Josh Hawn <josh.hawn@docker.com>
this is to enhance the tarsum algorithm, but _MUST_ be done in lock step
with the same for docker-registry. (PR will be cited)
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
This introduces Versions for TarSum checksums.
Fixes: https://github.com/docker/docker/issues/7526
It preserves current functionality and abstracts the interface for
future flexibility of hashing algorithms. As a POC, the VersionDev
Tarsum does not include the mtime in the checksum calculation, and would
solve https://github.com/docker/docker/issues/7387 though this is not a
settled Version is subject to change until a version number is assigned.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
If iptables version is < 1.4.11, try to delete the rule vs. checking if it exists. Fixes#6831.
Docker-DCO-1.1-Signed-off-by: Jessica Frazelle <jfrazelle@users.noreply.github.com> (github: jfrazelle)
This commit makes tarsum buffer allocation dynamic. This change
is required to avoid allocating memory excessively after the archive
buffering changes.
Docker-DCO-1.1-Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com> (github: unclejack)
--help and help are successful commands so output should not go to error.
QE teams have requested this change, also users doing docker help | less
or docker run --help | less would expect this to work.
Usage statement should only be printed when the user asks for it.
Errors should print error message and then suggest the docker COMMAND --help
command to see usage information.
The current behaviour causes the user to have to search for the error message
and sometimes scrolls right off the screen. For example a error on a
"docker run" command is very difficult to diagnose.
Finally erros should always exit with a non 0 exit code, if the user
makes a CLI error.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
This code is vital to the security of the project and it is important
we assure it is well-maintained and guarded.
I am vested in assuring this code maintains security and
provides as much compatibility as possible between releases.
Signed-off-by: Eric Windisch <eric@windisch.us>
The graphtest package is only imported in the test files of other
packages therefore we do not leak testing flags.
Signed-off-by: Michael Crosby <michael@docker.com>
This prevents the testing package flags from leaking into the flagsets
of binaries that import docker. I left integration-cli alone.
Docker-DCO-1.1-Signed-off-by: Peter Bourgon <peter@bourgon.org> (github: peterbourgon)
functions to pkg/parsers/kernel, and parsing filters to
pkg/parsers/filter. Adjust imports and package references.
Docker-DCO-1.1-Signed-off-by: Erik Hollensbe <github@hollensbe.org> (github: erikh)
Both functions perform the same logic and they just vary on the base
multiplication units. We can refactor the common code into a single
place.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to initialize every time the function executes since it works as
a catalog.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to have two lines. The type is even explicit when type casting
to `float64(size)`
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to recompile a fixed regular expression each time the function
executes. Abstracting it to the `init()` method.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
May make sense that both `FromHumanSize()` and `RAMInBytes()` support
the same units. Added 'PB' to the RAMInBytes regex.
Also updated tests.
Note: int64 is overflowed on quantities >= EB
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Better to not use `error` as var name (might eclipse the error type) for
clarity and to prevent subtle bugs.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Remove named returns since not used in function body. Might prevent
potential subtle bugs.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Increased coverage:
* Added test cases to size_test.go
* Added coverage for duration.go
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
In that case /etc/resolv.conf will be generated with no search
option. Usage: --dns-search=.
Docker-DCO-1.1-Signed-off-by: Fabio Falci <fabiofalci@gmail.com> (github: fabiofalci)
Without this line of code, if a volume is present in /proc/mounts,
it cannot be remounted with new mount options.
Docker-DCO-1.1-Signed-off-by: Peter Waller <p@pwaller.net> (github: pwaller)
Device Mapper needs device sizes in binary (1024) multiples. Otherwise
kernel checks can find that the specified thin-pool device sizes aren't
a multiple of the specified thin-pool blocksize.
The name for "RAMInBytes" is likely too narrow given the new consumers
but... Also add "tebibyte" support to RAMInBytes.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
as a maintainer.
Best of luck on your e-commerce business Guillaume, and thanks for all
the great contributions!
Docker-DCO-1.1-Signed-off-by: Solomon Hykes <solomon@docker.com> (github: shykes)
This does the "reverse" of HumanSize, i.e. maps a string to an int64
using SI prefixes for the extension.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
These PR does a few things. It ensures that the freezer cgroup is
joined in the systemd driver. It also provides a public api for setting
the freezer state via the cgroups package.
Docker-DCO-1.1-Signed-off-by: Michael Crosby <michael@crosbymichael.com> (github: crosbymichael)
The systemd support for the devices cgroup lacks two required features:
* Support for wildcards to allow mknod on any device
* Support for wildcards to allow /dev/pts support
The second is available in more recent systemd as "char-pts", but not in e.g. v208 which is in wide use.
Additionally, the current approach of letting systemd set up the devices cgroup and then adding
some devices to it doesn't work, because some times systemd (at least v208) re-initializes
the devices cgroup, overwriting our custom devices. See https://github.com/dotcloud/docker/issues/6009
for the details.
When wildcarded mknod support is available in systemd we should implement a pure systemd version,
but we need to keep the old one around for backwards compat.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
This patch adds some tests to ensure that quoted flags are properly
handled by the mflag package.
Docker-DCO-1.1-Signed-off-by: Aleksa Sarai <cyphar@cyphar.com> (github: cyphar)
This patch improves the mflag package to ensure that things arguments
to mflag such as `-flag="var"` or `-flag='var'` have the quotes
stripped from the value (to mirror the getopt functionality for similar
flags).
Docker-DCO-1.1-Signed-off-by: Aleksa Sarai <cyphar@cyphar.com> (github: cyphar)
This also makes sure that devices are pointers to avoid copies
Docker-DCO-1.1-Signed-off-by: Michael Crosby <michael@crosbymichael.com> (github: crosbymichael)
Josh Hawn
Victor Vieux
Jessie Frazelle
Tibor Vass
shuai-z
Alexandr Morozov
Vincent Batts
Andrea Luzzardi
Zach Borboa
Brian Goff
Srini Brahmaroutu
Alexandr Morozov
Michael Crosby
unclejack
Brian Goff
Erik Hollensbe
Rafe Colton
Jessica Frazelle
Derek McGowan
Victor Vieux
Guillaume Dufour
unclejack
Vincent Batts
Phil Estes
Sven Dowideit
SvenDowideit
Dan Walsh
Eric Windisch
Josiah Kiehl
Michael Crosby
lalyos
Solomon Hykes
Peter Bourgon
Tianon Gravi
Victor Vieux
Francisco Carriedo
s-ko
Fabio Falci
Jonathan Boulle
Peter Waller
Mike Snitzer
Mrunal Patel
Michael Crosby
Alexander Larsson
Victor Marmol
Vishnu Kannan
cyphar