Commit Graph

20 Commits

Author SHA1 Message Date
Miloslav Trmač 053ac6105d Remove ChecksumAnntation from ZstdChunkedFooterData
Manage the value directly to simplify.

This happens to fix the ReadFooterDataFromBlob code path,
which was not setting ChecksumAnntation at all.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-04-13 16:57:14 +02:00
Miloslav Trmač 1f47b38c09 Only obtain the zstd:chunked TOC digest once
Make it structually clear that the code is all using the same value,
making it less likely for the verifier and other uses to get out of sync.

Also avoids some redundant parsing and error paths.
The conversion path looks longer, but that's just moving the parsing
from the called function (which is redundant for other callers).

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-04-13 16:57:07 +02:00
Giuseppe Scrivano f388a77afb
chunked: fix unmarshaling of file names
The getString() function was used to extract string values, but it
doesn't handle escaped characters.  Replace it with iter.ReadString()
that is slower but handles escaped characters correctly.

Closes: https://github.com/containers/storage/issues/1878

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-04-09 22:03:13 +02:00
Giuseppe Scrivano f6356d6ccd
chunked: refactor private fields to internal struct
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-03-20 15:47:37 +01:00
Giuseppe Scrivano f989a95707
chunked: fix generating footer
commit 7bbf6ed448 introduces an error
where the generated footer is not correct.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-09-05 08:55:29 +02:00
Giuseppe Scrivano eedd976e5b
chunked: refactor code in new functions
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-09-05 08:55:29 +02:00
Giuseppe Scrivano 7bbf6ed448
chunked: generate tar-split as part of zstd:chunked
change the file format to store the tar-split as part of the
zstd:chunked image.  This will allow clients to rebuild the entire
tarball without having to download it fully.

also store the uncompressed digest for the tarball, so that it can be
stored into the storage database.

Needs: https://github.com/containers/image/pull/1976

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-06-17 00:31:39 +02:00
Kir Kolyshkin a4d8f720a2 Format sources with gofumpt
gofumpt is a superset of gofmt, enabling some more code formatting
rules.

This commit is brought to you by

	gofumpt -w .

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2023-05-26 16:17:31 -07:00
Giuseppe Scrivano f98fa3967d
chunked: use io.github.containers
we do not own containers.io so let's use io.github.containers, since
the project is part of the containers organization under github.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-11-03 12:48:45 +01:00
Miloslav Trmač 5e410ef763 Misc individual warning fixes
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-10-14 17:17:53 +02:00
Giuseppe Scrivano 33f973c732
Revert "chunked: use valyala/gozstd"
This reverts commit 5bb6d8e65e.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-17 16:16:46 +01:00
Giuseppe Scrivano 198820877c
pkg/chunked: add support for sparse files
automatically detect holes in sparse files (the threshold is hardcoded
at 1kb for now) and add this information to the manifest file.

The receiver will create a hole (using unix.Seek and unix.Ftruncate)
instead of writing the actual zeros.

Closes: https://github.com/containers/storage/issues/1091

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-13 13:32:13 +01:00
Giuseppe Scrivano 5bb6d8e65e
chunked: use valyala/gozstd
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:28:15 +01:00
Giuseppe Scrivano 0621da79cc
chunked: improve json parsing
reduce the number of allocations done by the parser by reading into a
bytes.Buffer.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 048f7c08ad
chunked: use json-iterator
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 96fc5c872a
chunked: omit empty fields in json manifest
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-07 21:28:16 +01:00
Giuseppe Scrivano bfd9c8046e
chunked: chunk deduplication
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-07 21:28:15 +01:00
Giuseppe Scrivano 22ba9b01ef
chunked: add infra to dedup from partial file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-12-24 13:28:25 +01:00
Giuseppe Scrivano 2855d17137
chunked: estargz support
in addition to zstd:chunked, add support for the estargz format.

estargz is maintained at github.com/containerd/stargz-snapshotter

Images using estargz can be used on old clients and registries that
have no support for the zstd compression algorithm.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-08-25 16:16:20 +02:00
Miloslav Trmač 92b63ead41 Split pkg/chunked.ZstdCompressor into a separate subpackage
containers/image/pkg/compression depends on that function, so it's
highly undesirable for it to drag in all of containers/storage.

So, move it into a separate subpackage, along with its exclusive
dependency tree.

Code that is called both from the new pkg/chunked/compressor
and the original pkg/chunked has been moved into pkg/chunked/internal ,
so that we don't expose any new public API. That move was made
purely mechanically without any concern for conceptual consistency,
to minimize the size of the dependency (and not to spend time on
that until the concept is proven to be useful).

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2021-07-23 21:43:52 +02:00