The ApplyDiffWithDiffer function was marked as deprecated,
with PrepareStagedLayer being the recommended replacement.
Its implementation was just a wrapper around PrepareStagedLayer.
Remove the deprecated function from the Store and LayerStore
interfaces and its implementation.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Replace the direct call to unix.Open with the O_TMPFILE flag
with the dedicated openTmpFile helper function.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Replace the in-memory buffer with a O_TMPFILE file. This reduces the
memory requirements for a partial pull since the tar-split data can be
written to disk.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
For struct Stat_t in syscall pkg:
- Ino is always uint64;
- Dev/Rdev can be uint64, uint32, or int32;
- Nlink might be uint64, uint32, or uint16.
Fix the code accordingly, adding or removing typecasts where needed,
and annotating those with //nolint:unconvert to calm down the unconvert
linter.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Those are the cases where the value being converted is already of that
type (checked to be that way for all os/arch combinations).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This form is available since Go 1.22 (see
https://tip.golang.org/ref/spec#For_range) and will probably be seen
more and more in the new code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We have seen an image with:
- total size 1.43 GB
- uncompressed zstd:chunked manifest size of 91.7 MB
- uncompressed tar-split size (not constrained by maxTocSize) 310 MB
Without more infrastructure, we are just guessing about what
the system we are running on can support, so, for now, *shrug*, bump
the number.
Eventually we should stream the data from/to disk, making this
much less relevant; that makes building the infrastructure to
estimate available memory unattractive.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
That's a logically better place, it pairs the getBlobAt
calls with the ErrBadRequest types specific to those call sites.
We will, also, add more fallback reasons.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Try to split fields by purpose, and document some of the context.
For consistency, use the same order in the struct literals.
Only reorders existing fields, should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This will allow c/image to validate the uncompressed digest against the config's
RootFS.DiffID value (ensuring that the layer's contents are the same when pulled
via TOC and traditionally); and the uncompressed digest will be used as a layer ID,
ensuring users see the traditional layer and image IDs they are used to.
This doesn't work for layers without a tar-split (all estargz, and old zstd:chunked
layers); for those, we fall back to traditional pulls.
Alternatively, for EXTREMELY restricted use cases, add an
"insecure_allow_unpredictable_image_contents" option to storage.conf. This option
allows partial pulls of estargz and old zstd:chunked layers, and skips the costly
uncompressed digest computation. It is then up to the user to worry about
images where the tar representation and the TOC representation don't match,
and about unpredictable image IDs.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... so that it happens
- before we start doing anything destructive
- only once (with no risk of defaults getting out of sync)
- in a single place
Ideally this should happen along with the initial parsing
of the config file; this is not that, but it is a minor step
in that direction.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
- Use a map of struct{} to save on storage
- Track the items by path, not by the digest; that's one byte more
per entry, but it will allow us to abstract the path logic.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We now have several internal subpackages of pkg/chunked, so delineate
more explicitly the parts that should be kept as small as possible
because the c/image compression package depends on them.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
ihen the file name is the root directory, avoid using an empty string
or the ".." name to open the file. The latter does not cause any
security issues or unexpected behavior, it is logically incorrect and
should be avoided.
Closes: https://github.com/containers/storage/issues/2191
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
rewrite how the result from GetBlobAt is used, to make sure 1) that
the streams are always closed, and 2) that any error is processed.
Closes: https://issues.redhat.com/browse/OCPBUGS-43968
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Disable the storage.options.pull_options.enable_partial_images option by
default, so that it will have to be explicitly enabled in order to be
used.
Update the apply-diff-from-staging-directory integration test to call
the test helper binary directly, so that the configuration file the test
writes won't have its settings overridden by command line options that
the storage() test helper function adds.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Instead of sharing the badRequestErr logic, duplicate it.
That's a bit ugly, but we get better debug messages and a more
traditional control flow.
Should not change behavior, except for debug messages.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... to centralize the fallback allowed / required logic.
Should not change behavior, apart from maybe some error text.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
1. Move codespell config out of Makefile, simplify (remove unused
stuff).
2. Fix found issues (using codespell -w).
3. Add a codespell CI job.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The current value obtained by summing the sizes of regular file contents
does not match the size of the uncompressed layer tarball.
We don't have a convenient source to compute the correct size
for estargz without pulling the full layer and defeating the point;
so we must allow for the size being unknown.
For recent zstd:chunked images, we have the full tar-split,
so we can compute the correct size; that will happen in
the following commits.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Giuseppe Scrivano