To avoid an expensive call to archive.ChangesDirs() which walks two directory
trees and compares every entry, archive.ApplyLayer() has been extended to
also return the size of the layer changes.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
When the user is not using the full has to retrieve a container it's
possible that we find conflicts with the ids of other containers.
At the moment it's just failing saying that it can not find a container,
but it doesn't say why. Adding a small log saying that duplicates where
found is going to help the user.
Closes#8098
Signed-off-by: Alex Gonzalez <agonzalezro@gmail.com>
This makes sure that we don't buffer in memory and that we also flush
stdin from diff as well as untar.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
If two interrupts were fired really quickly interruptCount could have been incremented twice before the LoadUint32 making cleanup not being called at all.
Signed-off-by: Erik Dubbelboer <erik@dubbelboer.com>
Another update to TarSum tests, this patch fixes an issue where
the benchmarks were generating archives incorrectly by not closing
the tarWriter.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
With 32ba6ab from #9261, TempArchive now closes the underlying file and
cleans it up as soon as the file's contents have been read. When pushing
an image, PushImageLayerRegistry attempts to call Close() on the layer,
which is a TempArchive that has already been closed. In this situation,
Close() returns an "invalid argument" error.
Add a Close method to TempArchive that does a no-op if the underlying
file has already been closed.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
These two cases did not actually read the same content with each iteration
of the benchmark. After the first read, the buffer was consumed. This patch
corrects this by using a bytes.Reader and seeking to the beginning of the
buffer at the beginning of each iteration.
Unfortunately, this benchmark was not actually as fast as we believed. But
the new results do bring its results closer to those of the other benchmarks.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Currently devicemapper CreateDevice and CreateSnapDevice keep on retrying
device creation till a suitable device id is found.
With new transaction mechanism we need to store device id in transaction
before it has been created.
So change the logic in such a way that caller decides the devices Id to
use. If that device Id is not available, caller bumps up the device Id
and retries.
That way caller can update transaciton too when it tries a new Id. Transaction
related patches will come later in the series.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
I noticed that 3 of the tarsum test cases had expected a tarsum with
a sha256 hash of
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
As I've been working with sha256 quite a bit lately, it struck me that
this is the initial digest value for sha256, which means that no data
was processed. However, these tests *do* process data. It turns out that
there was a bug in the test handling code which did not wait for tarsum
to end completely. This patch corrects these test cases.
I'm unaware of anywhere else in the code base where this would be an issue,
though we definitily need to look out in the future to ensure we are
completing tarsum reads (waiting for EOF).
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Current implementation is comingling things that ought not be together.
There are _some_ similarities between parsing for the different proto
types, but they are more different than alike, making the code extremely
difficult to reason about.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This reverts commit 967a42f116.
Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>
Roll back the change to fix the parameter of HumanSize from int64 to float64
This moves the IsGIT and IsURL functions out of the generic `utils`
package and into their own `urlutil` pkg.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Signed-off-by: Tibor Vass <teabee89@gmail.com>
Conflicts:
pkg/archive/archive.go
fixed conflict which git couldn't fix with the added BreakoutError
Conflicts:
pkg/archive/archive_test.go
fixed conflict in imports
I also needed to add a mflag.IsSet() function that allows you to check
to see if a certain flag was actually specified on the cmd line.
Per #9221 - also tweaked the docs to fix a typo.
Closes#9221
Signed-off-by: Doug Davis <dug@us.ibm.com>
This fixes the removal of TempArchives which can read with only one
read. Such archives weren't getting removed because EOF wasn't being
triggered.
Docker-DCO-1.1-Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com> (github: unclejack)
Now filter name is trimmed and lowercased before evaluation for case
insensitive and whitespace trimemd check.
Signed-off-by: Oh Jinkyun <tintypemolly@gmail.com>
Currently we set up a cookie and upon failure not call UdevWait(). This
does not cleanup the cookie and associated semaphore and system will
soon max out on total number of semaphores.
To avoid this, call UdevWait() even in failure path which in turn will
cleanup associated semaphore.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Signed-off-by: Vincent Batts <vbatts@redhat.com>
pkg/archive contains code both invoked from cli (cross platform) and
daemon (linux only) and Unix-specific dependencies break compilation on
Windows. We extracted those stat-related funcs into platform specific
implementations at pkg/system and added unit tests.
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Some parts of pkg/archive is called on both client/daemon code. To get
it compiling on Windows, these funcs are extracted into files with
build tags.
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
SIGCHLD and SIGWINCH used in api/client (cli code) are not
available on Windows. Extracting into separate files with build
tags.
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Otherwise udev can unecessarily execute various rules (and issue
scanning IO, etc) against the thin-pool -- which can never be a
top-level device.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
The current Dev version of TarSum includes hashing of extended
file attributes and omits inclusion of modified time headers.
I refactored the logic around the version differences to make it
more clear that the difference between versions is in how tar
headers are selected and ordered.
TarSum Version 1 is now declared with the new Dev version continuing
to track it.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This re-applies commit b39d02b with additional iptables rules to solve the issue with containers routing back into themselves.
The previous issue with this attempt was that the DNAT rule would send traffic back into the container it came from. When this happens you have 2 issues.
1) reverse path filtering. The container is going to see the traffic coming in from the outside and it's going to have a source address of itself. So reverse path filtering will kick in and drop the packet.
2) direct return mismatch. Assuming you turned reverse path filtering off, when the packet comes back in, it's goign to have a source address of itself, thus when the reply traffic is sent, it's going to have a source address of itself. But the original packet was sent to the host IP address, so the traffic will be dropped because it's coming from an address which the original traffic was not sent to (and likely with an incorrect port as well).
The solution to this is to masquerade the traffic when it gets routed back into the origin container. However for this to work you need to enable hairpin mode on the bridge port, otherwise the kernel will just drop the traffic.
The hairpin mode set is part of libcontainer, while the MASQ change is part of docker.
This reverts commit 63c303eecd.
Docker-DCO-1.1-Signed-off-by: Patrick Hemmer <patrick.hemmer@gmail.com> (github: phemmer)
By default is a demo of file differences, but can be used to create a
tar of changes between an old and new path.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
In an effort to make layer content 'stable' between import
and export from two different graph drivers, we must resolve
an issue where AUFS produces metadata files in its layers
which other drivers explicitly ignore when importing.
The issue presents itself like this:
- Generate a layer using AUFS
- On commit of that container, the new stored layer contains
AUFS metadata files/dirs. The stored layer content has some
tarsum value: '1234567'
- `docker save` that image to a USB drive and `docker load`
into another docker engine instance which uses another
graph driver, say 'btrfs'
- On load, this graph driver explicitly ignores any AUFS metadata
that it encounters. The stored layer content now has some
different tarsum value: 'abcdefg'.
The only (apparent) useful aufs metadata to keep are the psuedo link
files located at `/.wh..wh.plink/`. Thes files hold information at the
RW layer about hard linked files between this layer and another layer.
The other graph drivers make sure to copy up these psuedo linked files
but I've tested out a few different situations and it seems that this
is unnecessary (In my test, AUFS already copies up the other hard linked
files to the RW layer).
This changeset adds explicit exclusion of the AUFS metadata files and
directories (NOTE: not the whiteout files!) on commit of a container
using the AUFS storage driver.
Also included is a change to the archive package. It now explicitly
ignores the root directory from being included in the resulting tar archive
for 2 reasons: 1) it's unnecessary. 2) It's another difference between
what other graph drivers produce when exporting a layer to a tar archive.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Fixes#1992
Right now when you `docker cp` a path which is in a volume, the cp
itself works, however you end up getting files that are in the
container's fs rather than the files in the volume (which is not in the
container's fs).
This makes it so when you `docker cp` a path that is in a volume it
follows the volume to the real path on the host.
archive.go has been modified so that when you do `docker cp mydata:/foo
.`, and /foo is the volume, the outputed folder is called "foo" instead
of the volume ID (because we are telling it to tar up
`/var/lib/docker/vfs/dir/<some id>` and not "foo", but the user would be
expecting "foo", not the ID
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
This is the second of two steps to break the archive package's
dependence on utils so that archive may be moved into pkg. `Matches()`
is also a good candidate pkg in that it is small, concise, and not
specific to docker internals
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
This is the first of two steps to break the archive package's dependence
on utils so that archive may be moved into pkg. Also, the `Go()`
function is small, concise, and not specific to the docker internals, so
it is a good candidate for pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
When read is called on a tarsum with a two different read sizes, specifically the second call larger than the first, the dynamic buffer does not get reallocated causing a slice read error.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Not breaking the default cipher and algorithm for calculating checksums,
but allow for using alternate block ciphers during the checksum
calculation.
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
If a tar were constructed with duplicate file names, then depending on
the order, it could result in same tarsum.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Tarsum now correctly closes the internal TarWriter which appends
a block of 1024 zeros to the end of the returned archive.
Signed-off-by: Josh Hawn <josh.hawn@docker.com>
this is to enhance the tarsum algorithm, but _MUST_ be done in lock step
with the same for docker-registry. (PR will be cited)
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
This introduces Versions for TarSum checksums.
Fixes: https://github.com/docker/docker/issues/7526
It preserves current functionality and abstracts the interface for
future flexibility of hashing algorithms. As a POC, the VersionDev
Tarsum does not include the mtime in the checksum calculation, and would
solve https://github.com/docker/docker/issues/7387 though this is not a
settled Version is subject to change until a version number is assigned.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
If iptables version is < 1.4.11, try to delete the rule vs. checking if it exists. Fixes#6831.
Docker-DCO-1.1-Signed-off-by: Jessica Frazelle <jfrazelle@users.noreply.github.com> (github: jfrazelle)
This commit makes tarsum buffer allocation dynamic. This change
is required to avoid allocating memory excessively after the archive
buffering changes.
Docker-DCO-1.1-Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com> (github: unclejack)
--help and help are successful commands so output should not go to error.
QE teams have requested this change, also users doing docker help | less
or docker run --help | less would expect this to work.
Usage statement should only be printed when the user asks for it.
Errors should print error message and then suggest the docker COMMAND --help
command to see usage information.
The current behaviour causes the user to have to search for the error message
and sometimes scrolls right off the screen. For example a error on a
"docker run" command is very difficult to diagnose.
Finally erros should always exit with a non 0 exit code, if the user
makes a CLI error.
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
This code is vital to the security of the project and it is important
we assure it is well-maintained and guarded.
I am vested in assuring this code maintains security and
provides as much compatibility as possible between releases.
Signed-off-by: Eric Windisch <eric@windisch.us>
The graphtest package is only imported in the test files of other
packages therefore we do not leak testing flags.
Signed-off-by: Michael Crosby <michael@docker.com>
This prevents the testing package flags from leaking into the flagsets
of binaries that import docker. I left integration-cli alone.
Docker-DCO-1.1-Signed-off-by: Peter Bourgon <peter@bourgon.org> (github: peterbourgon)
functions to pkg/parsers/kernel, and parsing filters to
pkg/parsers/filter. Adjust imports and package references.
Docker-DCO-1.1-Signed-off-by: Erik Hollensbe <github@hollensbe.org> (github: erikh)
Both functions perform the same logic and they just vary on the base
multiplication units. We can refactor the common code into a single
place.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to initialize every time the function executes since it works as
a catalog.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to have two lines. The type is even explicit when type casting
to `float64(size)`
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
No need to recompile a fixed regular expression each time the function
executes. Abstracting it to the `init()` method.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
May make sense that both `FromHumanSize()` and `RAMInBytes()` support
the same units. Added 'PB' to the RAMInBytes regex.
Also updated tests.
Note: int64 is overflowed on quantities >= EB
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Better to not use `error` as var name (might eclipse the error type) for
clarity and to prevent subtle bugs.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Remove named returns since not used in function body. Might prevent
potential subtle bugs.
Docker-DCO-1.1-Signed-off-by: Francisco Carriedo <fcarriedo@gmail.com> (github: fcarriedo)
Alexander Morozov
Ankush Agarwal
Tomasz Nurkiewicz
Ian Bishop
Porjo
unclejack
Tibor Vass
Michael Crosby
Josh Hawn
Srini Brahmaroutu
Álex González
Alexander Morozov
Michael Crosby
Tianon Gravi
Vincent Batts
Erik Dubbelboer
Lewis Marshall
Jessica Frazelle
Jessie Frazelle
Andy Goldstein
Vivek Goyal
Erik Hollensbe
Qiang Huang
Brian Goff
Yohei Ueda
Tatsushi Inagaki
Vincent Batts
Doug Davis
Oh Jinkyun
Ahmet Alp Balkan
John Gossman
unclejack
Mike Snitzer
Steven Burgess
Arnaud Porterie
Patrick Hemmer
Victor Vieux
Solomon Hykes
Victor Vieux
shuai-z
Andrea Luzzardi
Zach Borboa
Srini Brahmaroutu
Brian Goff
Rafe Colton
Derek McGowan
Guillaume Dufour
Phil Estes
Sven Dowideit
SvenDowideit
Dan Walsh
Eric Windisch
Josiah Kiehl
Michael Crosby
lalyos
Solomon Hykes
Peter Bourgon
Victor Vieux
Francisco Carriedo