if the "convert_images" option is set in the configuration file, then
convert traditional images to the chunked format on the fly.
This is very expensive at the moment since the entire zstd:chunked
file is created and then processed.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
change the file format to store the tar-split as part of the
zstd:chunked image. This will allow clients to rebuild the entire
tarball without having to download it fully.
also store the uncompressed digest for the tarball, so that it can be
stored into the storage database.
Needs: https://github.com/containers/image/pull/1976
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
gofumpt is a superset of gofmt, enabling some more code formatting
rules.
This commit is brought to you by
gofumpt -w .
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
in addition to zstd:chunked, add support for the estargz format.
estargz is maintained at github.com/containerd/stargz-snapshotter
Images using estargz can be used on old clients and registries that
have no support for the zstd compression algorithm.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
containers/image/pkg/compression depends on that function, so it's
highly undesirable for it to drag in all of containers/storage.
So, move it into a separate subpackage, along with its exclusive
dependency tree.
Code that is called both from the new pkg/chunked/compressor
and the original pkg/chunked has been moved into pkg/chunked/internal ,
so that we don't expose any new public API. That move was made
purely mechanically without any concern for conceptual consistency,
to minimize the size of the dependency (and not to spend time on
that until the concept is proven to be useful).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
add a new custom variant of the zstd compression that permits to
retrieve each file separately.
The idea is based on CRFS and its stargz format for having seekable
and indexable tarballs.
One disadvantage of the stargz format is that a custom file is added
to the tarball to store the layer metadata, and the metadata file is
part of the image itself. Clients that are not aware of the stargz
format will propagate the metadata file inside of the containers.
The zstd compression supports embeddeding additional data as part of
the stream that the zstd decompressor will ignore (skippable frame),
so the issue above with CRFS can be solved directly within the zstd
compression format.
Beside this minor advantage, zstd is much faster and compresses better
than gzip, so take this opportunity to push the zstd format further.
The zstd compression is supported by the OCI image specs since August
2019: https://github.com/opencontainers/image-spec/pull/788 and has
been supported by containers/image since then.
Clients that are not aware of the zstd:chunked format, won't notice
any difference when handling a blob that uses the variant.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Giuseppe Scrivano