Commit Graph

80 Commits

Author SHA1 Message Date
Giuseppe Scrivano c9260b973a
chunked: rename GetDiffer to NewDiffer
it is an explicit API breaking change, so that cannot be used by old
users (e.g. an older containers/image version) that are not ported to
support the new semantic that only one ApplyDiffWithDiffer call is
supported for one differ object.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2025-05-07 08:52:19 +02:00
Kir Kolyshkin b7fb12e894 Remove unneeded conversion
Those are the cases where the value being converted is already of that
type (checked to be that way for all os/arch combinations).

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2025-04-01 16:18:43 -07:00
openshift-merge-bot[bot] f1c4bdf1d7
Merge pull request #2306 from kolyshkin/modernize
Use `slices` and `maps` more where appropriate
2025-04-01 22:00:08 +00:00
Kir Kolyshkin c3ff7f58df Use any instead of interface{}
It's available since Go 1.18 (see https://pkg.go.dev/builtin#any).

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2025-03-31 15:37:25 -07:00
Kir Kolyshkin ffb65e75e8 cmd/containers-storage: simplify
Simplify the code that finds the command.

While at it, rename cmd to name to avoid using cmd and command together.

Best reviewed with --ignore-all-space.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2025-03-31 15:09:03 -07:00
Giuseppe Scrivano 7082298e07
cmd: drop args to MountImage to use with vfs
the vfs driver does not create any mount, so it doesn't accept any
argument passed to MountImage.  Drop the arguments so mount image can
be used in the tests suite.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2025-03-25 15:49:02 +01:00
Giuseppe Scrivano fb7bf395e3
containers-storage: new command 'dedup'
introduce a new `dedup` command to the `containers-storage` tool to
deduplicate similar files in image layers.  Reflinks support from the
underlying file system is needed.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-12-09 16:41:59 +01:00
Nalin Dahyabhai 435aa93e14 Disable partial pulls (zstd:chunked) by default
Disable the storage.options.pull_options.enable_partial_images option by
default, so that it will have to be explicitly enabled in order to be
used.

Update the apply-diff-from-staging-directory integration test to call
the test helper binary directly, so that the configuration file the test
writes won't have its settings overridden by command line options that
the storage() test helper function adds.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2024-11-04 13:52:48 -05:00
Kir Kolyshkin 67e40f6286 cmd/containers-storage: use strings.Cut
In some places, using strings.Cut (available since Go 1.18) instead of
strings.SplitN allows for simpler more readable code.

While at it, simplify UID/GID error message (as strconv.ParseUint error
already contains the source string).

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2024-09-19 17:03:22 -07:00
Giuseppe Scrivano 5f68205f21
cmd: use PrepareStagedLayer
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-09-19 09:33:27 +02:00
Jan Rodák 4e1b4704d9
Fix errcheck: error return value of `m.CleanupStagedLayer` is not checked
Signed-off-by: Jan Rodák <hony.com@seznam.cz>
2024-07-09 16:57:57 +02:00
Giuseppe Scrivano e6d23590da
cmd: replace usage of deprecated functions
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-05-10 15:21:00 +02:00
Ed Baunton 86946837a9
Fix missing newline from stdout of create-container command
Signed-off-by: Ed Baunton <edbaunton@gmail.com>
2024-02-16 05:23:28 -05:00
Nalin Dahyabhai 5db18d7353 Store: add ImageDirectory() and ImageRunDirectory()
Add ImageDirectory() and ImageRunDirectory(), which return the paths of
directories which the caller can use to store image-specific data which
will be cleaned up automatically when the image is removed or the system
is restarted, respectively.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2024-01-23 14:35:01 -05:00
Giuseppe Scrivano fa794cb997
chunked: validate converted images
validate that the retrieved data for converted images matches the
expected digest.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-01-08 22:00:35 +01:00
Giuseppe Scrivano f424bfcf9f
store: new struct to pass ApplyDiffWithDiffer opts
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-12-04 16:58:13 +01:00
Giuseppe Scrivano b0885dfba9
store: drop rootless from arguments
drop the rootless argument from DefaultStoreOptions and
UpdateStoreOptions since this can be retrieved internally through the
unshare package.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-11-16 21:13:28 +01:00
Giuseppe Scrivano c40dde3f9e
cmd: add applydiff-using-staging-dir
add a new command to exercise the ApplyDiff from a staging directory.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2023-10-03 09:38:44 +02:00
Kir Kolyshkin a4d8f720a2 Format sources with gofumpt
gofumpt is a superset of gofmt, enabling some more code formatting
rules.

This commit is brought to you by

	gofumpt -w .

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2023-05-26 16:17:31 -07:00
Daniel J Walsh 4428f8b775
Merge pull request #1549 from flouthoc/pull-store
store: add support to split filesystem using `imagestore`
2023-04-17 16:55:03 -04:00
Aditya R 30775d4b2b
store: add support to split ImageStore
Allow storage users to split the filesystem of containers vs image
store, `imagestore` if configured will pull images in image storage
instead of the `graphRoot` while keeping the other parts still in the
originally configured `graphRoot`.

overlay: set workdir and upperdir according to splitstore
If splitstore is set `workdir` and `upperdir` must go into the
splitstore i.e `graphRoot`.

Signed-off-by: Aditya R <arajan@redhat.com>
2023-04-14 17:34:27 +05:30
Nalin Dahyabhai cabf1b9fd0 Initial implementation of consistency checks
Add initial Check() and Repair() methods to Stores.

Check() checks for inconsistencies between the layers which the
lower-level storage driver claims to know about and the ones which we
know we're managing.  It checks that layers referenced by layers,
images, and containers are known to us and that images referenced by
containers are known to us.  It checks that data which we store
alongside layers, images, and containers is still present, and to the
extent which we store other information about that data (frequenly just
the size of the data), verifies that it matches recorded expectations.
Lastly, it checks that layers which are part of images (and which we
therefore know what they should have in them) have the expected content,
and nothing else.

Repair() removes any containers, images, and layers which have any
errors associated with them.  This is destructive, so its use should be
considered and deliberate.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2023-04-13 10:38:24 -04:00
Nalin Dahyabhai 3007ac6efb Test helper: add "unshare"/"create-storage-layer"/"storage-layers"
Add "unshare", "create-storage-layer", and "storage-layers" commands to
the test helper, along with a "-U" flag to have it unshare when handling
a given command.

Add "-o" as an alias for the "--owner" flag to "copy".

Add "-r" as an alias for the "--ro" flag to "mount".

Add a "-q" flag to "layers" so that we can list just the IDs.

Drop mention of a couple of not-implemented options from
docs/containers-storage-create-layer.md.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2023-04-10 10:23:23 -04:00
Nalin Dahyabhai 0f2bccfa56 Complete "pulling up" of images in updateNames()
When updateNames() copies an image's record from a read-only store into
the read-write store, copy the accompanying data as well.

Add fields for setting data items at creation-time to LayerOptions,
ImageOptions, and ContainerOptions to make this easier for us and our
consumers.

Replace the store-specific Create() (and the one CreateWithFlags() and
Put()) with private create() and put() methods, since they're not
intended for consumption outside of this package, and add Flags to the
options structures we pass into those methods.  In create() methods,
make copies of those passed-in options structures before modifying any
of their contents.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2023-03-31 10:36:30 -04:00
Nalin Dahyabhai 87b926f5f2 test helper: maxArgs = 0 is actually a meaningful value
We do have commands that don't take any arguments, and quietly ignoring
arguments when they get them is not a good idea.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2023-01-27 09:21:43 -05:00
Nalin Dahyabhai 6d91bc12f3 cmd: add a CLI wrapper for GarbageCollect
Add "gc" as an action for the CLI wrapper, for running the
GarbageCollect() method.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2023-01-26 16:09:00 -05:00
Alexander Larsson d6cb12d10f Add test for transient mode
If STORAGE_TRANSIENT is set to 1 then transient mode will be enabled
in the tests.

Also adds overlay-transient to the cirrus CI.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
2022-11-14 16:36:30 +01:00
Miloslav Trmač d60159bc30 Add more error handling to cmd/containers-storage
Introduce an outputJSON helper to decrease repetition.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-10-14 17:17:54 +02:00
Miloslav Trmač ace9c41df3 Consolidate error reporting in cmd/containers-storage
Have the action handlers return an error value, and let
main() format that error, if any; this avoids duplicated
error formating code in the action handlers, dropping
89 lines.

This might change the error format in some cases (typically
%v vs. %+v).

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-10-14 17:17:53 +02:00
Miloslav Trmač c497e66e9a Work around a paralleltest crash
> ERRO [runner] Panic: paralleltest: package "main" (isInitialPkg: true, needAnalyzeSource: true): runtime error: index out of range [0] with length 0: goroutine 5859 [running]:
> ...
> github.com/kunwardeep/paralleltest/pkg/paralleltest.isTestFunction(0x1b7d8c0?)
> 	github.com/kunwardeep/paralleltest@v1.0.6/pkg/paralleltest/paralleltest.go:252 +0x165

Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-10-14 17:17:53 +02:00
Miloslav Trmač fbd5eff94d Simplify array concatenation
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:31:43 +02:00
Miloslav Trmač d80b48f97b Remove an unused assignment
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:31:42 +02:00
Miloslav Trmač b0c2fe5f20 Add missing error handling
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:31:41 +02:00
Miloslav Trmač be215b91d7 Fix JSON annotation
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:31:40 +02:00
Miloslav Trmač 7635db182b Use io.ReadAll instead of ioutil.ReadAll
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:30:46 +02:00
Giuseppe Scrivano cf998256f7
store: don't ignore empty mappings for top layer
do not pick the first available image if it was requested to not use
any mapping.

It causes a problem where an image is already present in the local
store with a mapping, then a container that doesn't specify any
mapping would end up to reuse that:

$ podman system reset -y
$ podman run --rm --uidmap 0:10000:1000 centos ls -ld /home
drwxr-xr-x. 2 root root 6 Nov  3  2020 /home

$ podman run --rm centos ls -ld /home
drwxr-xr-x. 2 10000 10000 6 Nov  3  2020 /home

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-09-10 17:43:02 +02:00
Giuseppe Scrivano 62b88ea445
containers-storage: add --volatile to container create
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-03-01 21:11:40 +01:00
Daniel J Walsh 120cc997d2
Move storageOpts structures into types subdir to shrink bindings.
Currently when we build podman bindings we are pulling in the entire
storage libraries, even though we only need a few structures and
functions.

Testing with the following program

```
package main

import (
	"fmt"
	"github.com/containers/storage/types"
)

func main() {
	fmt.Println(types.GetRootlessRuntimeDir(0))
}
```

Removing types above gives me compile size of the the program

du -s t.old t.new
9640	t.before
3232	t.after

Currently these functions are being vendored into
containers/common/pkg/config, which leads to large size in podman-remote
and podman bindings.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2021-02-26 05:34:09 -05:00
Giuseppe Scrivano 9cefbaec5f
containers-storage: support layer bigdata
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-02-02 11:39:03 +01:00
Daniel J Walsh 3fb27a360c
new interface for MountImage added
Interface added to mount image rootfs without any container.
image can be mounted only in ReadOnly mode.

Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2020-07-07 13:45:51 -04:00
yangfeiyu20102011 bb6c40aa2e
new interface Free for deleting Store object
New interface for deleting already-created Store object

When the store is not needed, it is neccessary to remove
it and release the memory

Signed-off-by: yangfeiyu <yangfeiyu20102011@163.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2020-06-29 13:01:33 -04:00
Valentin Rothberg c677b7d069 Move opts to internal/opts and fork docker filters
The dependency on docker's filter package prevents us from using newer
versions of docker/moby where the package's API has changed.  The filter
package is small enough to justify maintaining a small fork of it. Also,
since the opts package is only used for the `containers-storage` binary
and is not part of the actual storage library, move it to the ./internal
directory to make it clearer that this is an internal API.

Fixes: #450
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2019-10-29 13:12:13 +01:00
Nalin Dahyabhai b5a276ea64 ImageSize: don't get tripped up by images with no layers
Images don't have to have layers, so they don't have to have top layers,
and we shouldn't return an error when attempting to determine the size
of such an image.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2019-09-25 10:27:01 -04:00
Nalin Dahyabhai 2bfb2b5b41 Add tests for the ReadOnly field
Check that layers and images in read-only stores are correctly marked as
such.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2019-05-21 11:54:50 -04:00
Nalin Dahyabhai 85c846a2bb tests: test layerless images
Test creation of images without layers, and containers based on them.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2019-05-07 16:21:04 -04:00
Daniel J Walsh ef42340c2e
Revert "Add MountTempFromSource and RemoveTemp interfaces"
This reverts commit e9695564db.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2019-05-02 11:54:55 -04:00
Daniel J Walsh e9695564db
Add MountTempFromSource and RemoveTemp interfaces
These interfaces can be used to setup a graphdriver mountpoint
of the source directory for use within a container.
The RemoveTemp interface umounts the mountpoint and then removes
all of the modified data in the graphdriver for this source directory.

The primary use case of these interfaces is for container engines that
want to mount a directory from the host system into the container. The
source dirctory then can be modified without actually changing the
directory on the host.

Containers will use these interfaces for sharing packaing cache directories
like /var/cache/dnf, to help speed up container builds.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2019-04-29 14:46:40 -04:00
Giuseppe Scrivano 3dd479804f
storage: move the logic to detect rootless into utils.go
add a helper function that automatically detects the UID when running
in a user namespace..

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2019-04-05 15:11:57 +02:00
Daniel J Walsh 877c68cba0
Move pkg/util default storage functions from libpod to containers/storage
In an effort to remove cross vendoring, trying to fix buildah from importing
from libpod.  I beleive these libraries make more sense in containers/storage
then in libpod.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2019-03-19 08:33:42 -04:00
Nalin Dahyabhai b41a2a8b83 Drop our dependency on github.com/containers/image
Drop our dependency on the image library's manifest package by requiring
that callers pass its Digest() function to us as a callback.  This makes
our CLI test/diagnostic tool calculate digests of s1 manifests
incorrectly, but that's not something that we were testing.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2019-03-06 15:06:34 -05:00