This commit introduces the capability to convert tar layers
to the zstd:chunked format, without performing any zstd compression.
This allows using zstd:chunked without the cost of compression and
decompression.
Closes: https://issues.redhat.com/browse/RUN-3056
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This ultimately amounts to the same thing (and os.File.Seek is
a tiny bit more costly, with some atomic operations) - using
os.File.Seek is more consistent with the surrounding code
relying on uses through os.File, as well as a tiny bit
shorter.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
it is an explicit API breaking change, so that cannot be used by old
users (e.g. an older containers/image version) that are not ported to
support the new semantic that only one ApplyDiffWithDiffer call is
supported for one differ object.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The chunkedDiffer object holds state and resources that are managed
within a single ApplyDiff call. Reusing the same differ instance for
multiple ApplyDiff calls could lead to incorrect state or errors related
to already-closed resources.
Add a flag and check to ensure ApplyDiff cannot be called more than
once on the same chunkedDiffer instance, making its usage pattern explicit
and preventing potential misuse.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Ensure temporary file descriptors for tar-split data
are closed properly in error paths within zstd:chunked
handling.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Transfer ownership of the `tocDigest` file handle upon
successful completion of `ApplyDiff` in the chunked
differ. This prevents the handle from being closed by
the differ if the operation succeeds, as ownership is
effectively passed along.
Explicitly close the `TarSplit` reader during
`CleanupStagedLayer` and `ApplyDiffWithDiffer`. This ensures
the associated file handle is released even if the layer
application process is aborted or fails before the reader
is naturally consumed and closed elsewhere.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The ApplyDiffWithDiffer function was marked as deprecated,
with PrepareStagedLayer being the recommended replacement.
Its implementation was just a wrapper around PrepareStagedLayer.
Remove the deprecated function from the Store and LayerStore
interfaces and its implementation.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Replace the direct call to unix.Open with the O_TMPFILE flag
with the dedicated openTmpFile helper function.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Replace the in-memory buffer with a O_TMPFILE file. This reduces the
memory requirements for a partial pull since the tar-split data can be
written to disk.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Those are the cases where the value being converted is already of that
type (checked to be that way for all os/arch combinations).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
That's a logically better place, it pairs the getBlobAt
calls with the ErrBadRequest types specific to those call sites.
We will, also, add more fallback reasons.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Try to split fields by purpose, and document some of the context.
For consistency, use the same order in the struct literals.
Only reorders existing fields, should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will allow c/image to validate the uncompressed digest against the config's
RootFS.DiffID value (ensuring that the layer's contents are the same when pulled
via TOC and traditionally); and the uncompressed digest will be used as a layer ID,
ensuring users see the traditional layer and image IDs they are used to.
This doesn't work for layers without a tar-split (all estargz, and old zstd:chunked
layers); for those, we fall back to traditional pulls.
Alternatively, for EXTREMELY restricted use cases, add an
"insecure_allow_unpredictable_image_contents" option to storage.conf. This option
allows partial pulls of estargz and old zstd:chunked layers, and skips the costly
uncompressed digest computation. It is then up to the user to worry about
images where the tar representation and the TOC representation don't match,
and about unpredictable image IDs.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... so that it happens
- before we start doing anything destructive
- only once (with no risk of defaults getting out of sync)
- in a single place
Ideally this should happen along with the initial parsing
of the config file; this is not that, but it is a minor step
in that direction.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Use a map of struct{} to save on storage
- Track the items by path, not by the digest; that's one byte more
per entry, but it will allow us to abstract the path logic.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We now have several internal subpackages of pkg/chunked, so delineate
more explicitly the parts that should be kept as small as possible
because the c/image compression package depends on them.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
rewrite how the result from GetBlobAt is used, to make sure 1) that
the streams are always closed, and 2) that any error is processed.
Closes: https://issues.redhat.com/browse/OCPBUGS-43968
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Disable the storage.options.pull_options.enable_partial_images option by
default, so that it will have to be explicitly enabled in order to be
used.
Update the apply-diff-from-staging-directory integration test to call
the test helper binary directly, so that the configuration file the test
writes won't have its settings overridden by command line options that
the storage() test helper function adds.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Instead of sharing the badRequestErr logic, duplicate it.
That's a bit ugly, but we get better debug messages and a more
traditional control flow.
Should not change behavior, except for debug messages.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... to centralize the fallback allowed / required logic.
Should not change behavior, apart from maybe some error text.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The current value obtained by summing the sizes of regular file contents
does not match the size of the uncompressed layer tarball.
We don't have a convenient source to compute the correct size
for estargz without pulling the full layer and defeating the point;
so we must allow for the size being unknown.
For recent zstd:chunked images, we have the full tar-split,
so we can compute the correct size; that will happen in
the following commits.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
define a new error type so that the caller can determine whether it is
safe to ignore the error and retrieve the resource fully.
Closes: https://github.com/containers/storage/issues/2115
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Avoid handling cases where the server doesn't support at least 64
ranges in a request, in order to prevent falling back to the
traditional pull mechanism.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Conceptually, these read from the network and _should_ be interruptible.
But we could, at best, interrupt the wait on the channels returned by
GetBlobAt; we would then still get a ReadCloser where we can call
Read() but not concurrently notice a cancellation.
The cancellation needs to happen at the HTTP client side, i.e. inside
the c/image private.ImageSourceInternalOnly.GetBlobAt call (not ImageSourceSeekable),
and, although it seems not to be quite documented, the http stack's http.NewRequestWithContext
does cause the HTTP body transport to be terminated on a context cancellation.
I.e. it is out of control of this codebase anyway; so don't pretend to handle it.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
if the compressed digest was validated, as it happens when
'pull_options = {convert_images = "true"}' is set, then store it as
well so that reusing the blob by its compressed digest works.
Previously, when an image converted to zstd:chunked was pulled a
second time, it would not be recognized by its compressed digest,
resulting in the need to re-pull the image again.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Giuseppe Scrivano