if the "convert_images" option is set in the configuration file, then
convert traditional images to the chunked format on the fly.
This is very expensive at the moment since the entire zstd:chunked
file is created and then processed.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
change the file format to store the tar-split as part of the
zstd:chunked image. This will allow clients to rebuild the entire
tarball without having to download it fully.
also store the uncompressed digest for the tarball, so that it can be
stored into the storage database.
Needs: https://github.com/containers/image/pull/1976
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
gofumpt is a superset of gofmt, enabling some more code formatting
rules.
This commit is brought to you by
gofumpt -w .
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Two error messages suggest that podman-system-migrate is a binary that
can be run, when the command is "podman system migrate".
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of the
deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
drop host deduplication by just looking at the file path. It could be
useful in very specific use cases, but it is too expensive for generic
images. If the need arises, we first need to create an index of the
files that we can deduplicate so there is no need to calculate the
checksum on the fly.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
after the missing parts are merged, it is necessary to recalculate the
chunks to ask to the server.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Based on a conversation on the Podman mailing list:
Mentioning podman-system-migrate in the error message may help users
resolve their issues faster.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded
at 1kb for now) and add this information to the manifest file.
The receiver will create a hole (using unix.Seek and unix.Ftruncate)
instead of writing the actual zeros.
Closes: https://github.com/containers/storage/issues/1091
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
avoid parsing each json TOC file for the layers in the local storage,
but attempt to create a lookaside cache in a custom format faster to
load (and potentially be mmap'able).
The same cache is used to lookup files, chunks and candidates for
deduplication with hard links.
There are 3 kind of digests stored:
- digest(file.payload))
- digest(digest(file.payload) + file.UID + file.GID + file.mode + file.xattrs)
- digest(i) for each i in chunks(file payload)
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
try to reuse an existing cache object, instead of creating it for
every layer.
Set a time limit on how long it can be reused so to clean up stale
references.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
it solves a problem where the discard could be performed before the
compression handler was closed (through a deferred call).
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Giuseppe Scrivano
Kir Kolyshkin
Daniel J Walsh
Nalin Dahyabhai
Miloslav Trmač
Sascha Grunert
Valentin Rothberg