This will allow c/image to validate the uncompressed digest against the config's
RootFS.DiffID value (ensuring that the layer's contents are the same when pulled
via TOC and traditionally); and the uncompressed digest will be used as a layer ID,
ensuring users see the traditional layer and image IDs they are used to.
This doesn't work for layers without a tar-split (all estargz, and old zstd:chunked
layers); for those, we fall back to traditional pulls.
Alternatively, for EXTREMELY restricted use cases, add an
"insecure_allow_unpredictable_image_contents" option to storage.conf. This option
allows partial pulls of estargz and old zstd:chunked layers, and skips the costly
uncompressed digest computation. It is then up to the user to worry about
images where the tar representation and the TOC representation don't match,
and about unpredictable image IDs.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
introduce a new `dedup` command to the `containers-storage` tool to
deduplicate similar files in image layers. Reflinks support from the
underlying file system is needed.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Disable the storage.options.pull_options.enable_partial_images option by
default, so that it will have to be explicitly enabled in order to be
used.
Update the apply-diff-from-staging-directory integration test to call
the test helper binary directly, so that the configuration file the test
writes won't have its settings overridden by command line options that
the storage() test helper function adds.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
xref https://github.com/containers/storage/pull/2065#issuecomment-2313002081
This does the same for the man page that I did for the default
storage.conf previously; instead of trying to squash all the
`pull_options` into a single option that we document "ad-hoc", make
it its own table just like the other sections.
- Add notes about "string bools" around relevant values.
- Avoid redundantly specifying the default; the default is the first
thing we show after the `=`.
Synchronize the text between the man page and the default
`storage.conf`.
Signed-off-by: Colin Walters <walters@verbum.org>
The default storage.conf we ship is inconsistent in a few ways;
there's a lot of fixes rolled up into this single commit.
First: we were using a toml "inline table" for the pull options,
and then documenting each key in that table in one blob above.
It simply looks much nicer to use a non-inline table - then
we can move the docs next to each individual value. This is also
more consistent with other sections of the config.
I also thinned out a bit the doc comments; I think instead of trying
to have a longer explanation of zstd:chunked in the comments here
we should refer to the man page, which is a better place to have
details (and that we should fill out more).
Per another PR, I also stumbled across the fact that we have a
lot of "string bool" values and cannot be native TOML booleans. Document
that clearly next to each type.
We already have default values in the *code* for all
of these, so comment them all out to be consistent with other
values. (We're then getting closer to having the config file
be entirely comments, but that's a distinct project)
Finally, update the recent man pages I added to match
these changes.
Signed-off-by: Colin Walters <walters@verbum.org>
These aren't installed yet so they'll just live in the git repository
but hopefully we can expand them and use them as a central reference
point for "how do I do (zstd:chunked)|(composefs)". We have copies
of that kind of instruction in other downstream places but
I think here makes the most sense.
Signed-off-by: Colin Walters <walters@verbum.org>
drop the possibility to configure a remapping for all the layers in
the storage.
The feature dates back to the initial fork from Docker, that supported
a single user namespace where all the images were pulled. It was never
used by the container tools since we have a finer control of the user
namespaces.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Synchronise documentation around the configuration file to accurately
reflect new `convert_images` flag.
Signed-off-by: Ed Baunton <edbaunton@gmail.com>
by default enable pulling a partial image, it is still possible to
disable the feature through the configuration file.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add ImageDirectory() and ImageRunDirectory(), which return the paths of
directories which the caller can use to store image-specific data which
will be cleaned up automatically when the image is removed or the system
is restarted, respectively.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Remap-User/Group setting is always override by
Remap-UIDs/GIDs setting and ignored.
This commit enables Remap-User/Group setting.
Signed-off-by: Toshiki Sonoda <sonoda.toshiki@fujitsu.com>
Allow storage users to split the filesystem of containers vs image
store, `imagestore` if configured will pull images in image storage
instead of the `graphRoot` while keeping the other parts still in the
originally configured `graphRoot`.
overlay: set workdir and upperdir according to splitstore
If splitstore is set `workdir` and `upperdir` must go into the
splitstore i.e `graphRoot`.
Signed-off-by: Aditya R <arajan@redhat.com>
Add initial Check() and Repair() methods to Stores.
Check() checks for inconsistencies between the layers which the
lower-level storage driver claims to know about and the ones which we
know we're managing. It checks that layers referenced by layers,
images, and containers are known to us and that images referenced by
containers are known to us. It checks that data which we store
alongside layers, images, and containers is still present, and to the
extent which we store other information about that data (frequenly just
the size of the data), verifies that it matches recorded expectations.
Lastly, it checks that layers which are part of images (and which we
therefore know what they should have in them) have the expected content,
and nothing else.
Repair() removes any containers, images, and layers which have any
errors associated with them. This is destructive, so its use should be
considered and deliberate.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add "unshare", "create-storage-layer", and "storage-layers" commands to
the test helper, along with a "-U" flag to have it unshare when handling
a given command.
Add "-o" as an alias for the "--owner" flag to "copy".
Add "-r" as an alias for the "--ro" flag to "mount".
Add a "-q" flag to "layers" so that we can list just the IDs.
Drop mention of a couple of not-implemented options from
docs/containers-storage-create-layer.md.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When updateNames() copies an image's record from a read-only store into
the read-write store, copy the accompanying data as well.
Add fields for setting data items at creation-time to LayerOptions,
ImageOptions, and ContainerOptions to make this easier for us and our
consumers.
Replace the store-specific Create() (and the one CreateWithFlags() and
Put()) with private create() and put() methods, since they're not
intended for consumption outside of this package, and add Flags to the
options structures we pass into those methods. In create() methods,
make copies of those passed-in options structures before modifying any
of their contents.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Insisting on “DCO” imposes formalities, that serve self-purpose. One cannot
assume that the submitter has time or will to read texts about symbolism in
software contributions. If the system wants to see the text
nrEAUIEUAIe eanuitdnuae EAIUEAUIAIE »ℓ§444.3.72b)°»°ℓ§euaieauuae
in each commit, people will write this, or any other text, that the system wants to
see. All such text, which presence is mandated by the system, has the same value.
Signed-off-by: Дилян Палаузов <git-dpa@aegee.org>
drop host deduplication by just looking at the file path. It could be
useful in very specific use cases, but it is too expensive for generic
images. If the need arises, we first need to create an index of the
files that we can deduplicate so there is no need to calculate the
checksum on the fly.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We have had cases where users assume that create a storage.conf file in
their home directory, will still inherit fields from the system
storage.conf files. Need to clear this up in the storage.conf file and
man page.
Fixes: https://github.com/containers/podman/issues/11778
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This patch fixes the handling of inodes and sizes, currently if
user sets indoes and sizes together, quota is only set on sizes.
Second problem with quota is that we have to have unigue projectids
for each directory. Originally container/storage only did quota on
rootfs, now we want to support it on volumes as well. We need to be
able to get unigue projectids for these two different parent
directories. The added function, attempts to maintain at least 10,000
unigue id's based on the inode of the parent directory. I know that this
is not perfect and we have a potential for overlay. If you have a
better algorythm, I would love to use it.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
quota for overlay also supports setting the maximum number of
inodes. OpenShift would like to be able to set this to control the
number of inodes added to an image or to a volume.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
The storage driver set in `/etc/containers/storage.conf` was ignored for rootless configurations.
Rootless configurations would use overlay or vfs by default.
If the STORAGE_DRIVER environment was variable set, this value would be used instead.
This commit changes this behavior to inherit the driver set in `/etc/containers/storage.conf`.
To inherit the driver in rootless configurations, the driver must be valid in a rootless context.
The valid rootless drivers are btrfs, overlay, and vfs at this time.
To remain consistent with previous behavior, the STORAGE_DRIVER environment variable supersedes any driver set in `/etc/containers/storage.conf` and does not need to be a valid rootless driver.
buildah's documentation for the `--storage-driver` command will need to be updated to reflect these changes.
Albin Vass