Replace the in-memory buffer with a O_TMPFILE file. This reduces the
memory requirements for a partial pull since the tar-split data can be
written to disk.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Those are the cases where the value being converted is already of that
type (checked to be that way for all os/arch combinations).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We have seen an image with:
- total size 1.43 GB
- uncompressed zstd:chunked manifest size of 91.7 MB
- uncompressed tar-split size (not constrained by maxTocSize) 310 MB
Without more infrastructure, we are just guessing about what
the system we are running on can support, so, for now, *shrug*, bump
the number.
Eventually we should stream the data from/to disk, making this
much less relevant; that makes building the infrastructure to
estimate available memory unattractive.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
That's a logically better place, it pairs the getBlobAt
calls with the ErrBadRequest types specific to those call sites.
We will, also, add more fallback reasons.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We now have several internal subpackages of pkg/chunked, so delineate
more explicitly the parts that should be kept as small as possible
because the c/image compression package depends on them.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
rewrite how the result from GetBlobAt is used, to make sure 1) that
the streams are always closed, and 2) that any error is processed.
Closes: https://issues.redhat.com/browse/OCPBUGS-43968
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
... instead of our version which makes assumptions on the
internal decisions of the tar-split project, and needs heuristics
to guess where file padding ends.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Don't ignore the tar-split when the TOC requires one,
otherwise we could deduplicate a layer without tar-split
with a layer with tar-split.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
always consume the tar-split data when present to avoid blocking the
producer. Previously, the tar-split data was only read when the digest
was specified.
commit 6875c9fbcf introduced the
regression.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
if a digest was not specified in the TOC, ignore completely the
tar-split data.
Otherwise the clients fail to pull images created before commit
b5413c2bd6.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Other TOC formats don't fill the data in.
For now, this only increases memory usage, but we will
need the data soon.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We have the ImageSourceChunk data type, and we already
construct these values, so scan into them directly instead
of having three separate variables for the two bits of data.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Replace it by individual variables.
Then formally deprecate the ChecksumAnnotationTarSplit field.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Again, decrease the size of the compression code for c/image.
We will simplify this further immediately.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This code path is usually never triggered because
the annotations are present; and it was broken until recently.
Remove it to simplify the code and analysis.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Manage the value directly to simplify.
This happens to fix the ReadFooterDataFromBlob code path,
which was not setting ChecksumAnntation at all.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Make it structually clear that the code is all using the same value,
making it less likely for the verifier and other uses to get out of sync.
Also avoids some redundant parsing and error paths.
The conversion path looks longer, but that's just moving the parsing
from the called function (which is redundant for other callers).
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Make it structually clear that the code is all using the same value,
making it less likely for the verifier and other uses to get out of sync.
Also avoids some redundant parsing and error paths.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
if the "convert_images" option is set in the configuration file, then
convert traditional images to the chunked format on the fly.
This is very expensive at the moment since the entire zstd:chunked
file is created and then processed.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
change the file format to store the tar-split as part of the
zstd:chunked image. This will allow clients to rebuild the entire
tarball without having to download it fully.
also store the uncompressed digest for the tarball, so that it can be
stored into the storage database.
Needs: https://github.com/containers/image/pull/1976
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
we do not own containers.io so let's use io.github.containers, since
the project is part of the containers organization under github.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Giuseppe Scrivano