in addition to zstd:chunked, add support for the estargz format.
estargz is maintained at github.com/containerd/stargz-snapshotter
Images using estargz can be used on old clients and registries that
have no support for the zstd compression algorithm.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
if the option use_hard_links is set:
[storage.options]
pull_options = {use_hard_links = "true"}
then attempt to deduplicate files using hard links first.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
enable partial pulls only when it is explicitely configured in the
storage.conf file:
[storage.options]
pull_options = {enable_partial_images = "true"}
This is to prevent the experimental feature to leak into CRI-O.
The default value will change in future once the feature is stable.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
containers/image/pkg/compression depends on that function, so it's
highly undesirable for it to drag in all of containers/storage.
So, move it into a separate subpackage, along with its exclusive
dependency tree.
Code that is called both from the new pkg/chunked/compressor
and the original pkg/chunked has been moved into pkg/chunked/internal ,
so that we don't expose any new public API. That move was made
purely mechanically without any concern for conceptual consistency,
to minimize the size of the dependency (and not to spend time on
that until the concept is proven to be useful).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
add a new custom variant of the zstd compression that permits to
retrieve each file separately.
The idea is based on CRFS and its stargz format for having seekable
and indexable tarballs.
One disadvantage of the stargz format is that a custom file is added
to the tarball to store the layer metadata, and the metadata file is
part of the image itself. Clients that are not aware of the stargz
format will propagate the metadata file inside of the containers.
The zstd compression supports embeddeding additional data as part of
the stream that the zstd decompressor will ignore (skippable frame),
so the issue above with CRFS can be solved directly within the zstd
compression format.
Beside this minor advantage, zstd is much faster and compresses better
than gzip, so take this opportunity to push the zstd format further.
The zstd compression is supported by the OCI image specs since August
2019: https://github.com/opencontainers/image-spec/pull/788 and has
been supported by containers/image since then.
Clients that are not aware of the zstd:chunked format, won't notice
any difference when handling a blob that uses the variant.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Giuseppe Scrivano