avoid creating a new buffer for each file in the archive but
preallocate a 1M buffer. Another advantage of having a bigger buffer
is that we reduce the number of read and write syscalls.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This should avoid the following panic in Lgetxattr:
panic: runtime error: makeslice: len out of range
Fixes: 0da4bc60b3 ("pkg/archive: strictly handle errors reading xattrs")
See: https://travis-ci.org/containers/buildah/jobs/617652963
Signed-off-by: Zac Medico <zmedico@gmail.com>
This API caused build regressions as golang.org/x dependencies
introduced breaking changes. We don't have any callers in our
projects and I consider it a bug that those APIs were there,
which might prevent us from causing a major bump.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Preservation of user.* xattrs is supported by most (if not all) other
drivers, and it's especially useful for containers that run under a
PaX kernel where "user.pax.flags" is used to store PaX flags.
Signed-off-by: Zac Medico <zmedico@gmail.com>
We now add the information how many exclusions a match has, which is
needed for more fine-granular filepath matching.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
The `pure` indicator for a match now returns `true` if it's the only
one. This additional information will be used by consumers to decide
more fine-granular if a match can be excluded completely or not.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Modified patch of Kevin Pelzel.
Also changed ApplyDiff to take new ApplyDiffOpts Struct.
Signed-off-by: Kevin Pelzel <kevinpelzel22@gmail.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
zstd is a compression algorithm that has a very fast decoder, while
providing also good compression ratios. The fast decoder makes it
suitable for container images, as decompressing the tarballs is a very
expensive operation. This is a first step at supporting zstd as we We
don't yet generate zstd layers.
In my testing, copying the Fedora image from a local dir: repository,
the wall clock time passed from ~8s with gzip compression to ~4.5s
with zstd.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
if os.[Lc,C]hown are failing with EINVAL, it might be related to an
UID/GID not mapped in the user namespace we are currently using.
It could be possible to detect this issue by inspecting
/proc/self/uid_map or /proc/self/gid_map, but that won't be possible
when we are pulling a new image and extracting it from a chroot where
/proc is not mounted.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
from my tests, I've seen a net improvement of around 30% on the wall
clock time in decompressing layers.
These additional packages will need to be re-vendored:
github.com/klauspost/pgzip v1.2.1
github.com/klauspost/compress v1.4.1
github.com/klauspost/cpuid v1.2.0
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
old kernels, such as 4.9 used by Debian 9 do not support extended
attributes in a user namespace. In this case, ignore the error and
print a warning. This should be fine, as we are already doing it for
file systems that do not support xattrs.
Closes: https://github.com/containers/libpod/issues/1941
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
When writing a header with no format explicitly specified, the first
thing archive/tar 1.10 does is round the header's ModTime field,
possibly up, which confuses our change-detection logic when we later go
to check if the result of untarring the archive matches the source
content. Truncate the timestamp before that can happen.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we know we're going to extract the contents of a tar archive as
part of a copying process, set the desired format in the tar header to
PAX so that we get subsecond precision in timestamps, so that the
timestamps on the copies that we create aren't rounded off.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we call createTarFile with a non-nil IDPair indicating that the
newly-created file should have its ownership changed, pass in IDs that
we've mapped back to host-level IDs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Expose reading and writing ID mapping in the archive and chrootarchive
packages, and in the driver interface. Generally this means that
when computing or applying diffs, we need to have ID mappings passed in
that are specific to the layers we're using.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If the overlay filesystem implements an optimization that landed in
97c684cc911060ba7f97c0925eaf842f159a39e8, and in the mainline kernel in
4.10, directories created in merged directories are marked as opaque by
the kernel to let the kernel know that it needn't bother looking at
other layers when reading the contents of that directory.
This means that, when generating a diff for an upper directory, we can't
treat the presence of an opaque attribute as enough of an indication
that a layer diff needs to include whiteout for a directory of the same
name from a lower layer.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add back in these interfaces since containers/image is using.
archive.UntartPath
archive.IsArchive
Also add back this constant for containers/image.
archive.HeaderSize
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Had to vendor in a new version of golang.org/x/net to build
Also had to make some changes to drivers to handle
archive.Reader -> io.Reader
archive.Archive -> io.ReadCloser
Also update .gitingore to ignore emacs files, containers-storage.*
and generated man pages.
Also no longer test travis against golang 1.7, cri-o, moby have also
done this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Also updates github.com/Microsoft/hcsshim
and github.com/docker/go-connections so they work
with the newer version of logrus.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Rename the library module and CLI wrapper.
Rename daemon/graphdriver to drivers.
Catch up vendoring to match modules we've pruned.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Currently when overlay creates a whiteout file then the overlay2 layer is archived,
the correct tar header will be created for the whiteout file, but the tar logic will then attempt to open the file causing a failure.
When tar encounters such failures the file is skipped and excluded for the archive, causing the whiteout to be ignored.
By skipping the copy of empty files, no open attempt will be made on whiteout files.
Fixes#23863
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
This fix fixes#23459.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
There might be other (valid) reasons for setxattr(2) to fail, so only
ignore it when it's a not supported error (ENOTSUP). Otherwise, bail.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Since certain filesystems don't support extended attributes, ignore
errors produced (emitting a warning) when attempting to apply extended
attributes to file.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
If the destination does not exist, it needs to be created with ownership
mapping to the remapped uid/gid ranges if user namespaces are enabled.
This fixes ADD operations, similar to the prior fixes for COPY and WORKDIR.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
Closes#20470
Before this PR we used to scan the entire build context when there were
exclusions in the .dockerignore file (paths that started with !). Now we
only traverse into subdirs when one of the exclusions starts with that dir
path.
Signed-off-by: Doug Davis <dug@us.ibm.com>
During "COPY" or other tar unpack operations, a target/destination
parent dir might not exist and should be created with ownership of the
root in the right context (including remapped root when user namespaces
are enabled)
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
When execute `docker export -o path xxx` and path is a directory docker
has no privilege to write to, daemon will print lots of error logs that
most of them are duplicated and redundant.
This will remove unnecessary error logs and print only once.
Signed-off-by: Zhang Wei <zhangwei555@huawei.com>
zhangguanzhang