Commit Graph

29617 Commits

Author SHA1 Message Date
Giuseppe Scrivano cdf703fa81
chunked: return PathError for fsetxattr
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:25:26 +02:00
Giuseppe Scrivano 145b4f9a52
chunked: return PathError for utimensat
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:25:25 +02:00
Giuseppe Scrivano 7094880624
chunked: return PathError for Linkat errors
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:24:53 +02:00
Giuseppe Scrivano 886d4eedba
chunked: return PathError for fchmodat/fchmod
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:24:27 +02:00
Giuseppe Scrivano d1447ae783
chunked: unify chown calls
refactor all calls to Fchown and Fchownat into a single function, and
provide a better error messages.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:24:27 +02:00
Giuseppe Scrivano 62c4c243a3
chunked: refactor doHardLink to get full destination path
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 17:24:23 +02:00
openshift-merge-bot[bot] 0e780f7616
Merge pull request #1956 from cgwalters/xattrs
chunked: Add helpers for /proc/self/fd accesses
2024-06-07 14:25:37 +00:00
Colin Walters cfde71b024 chunked: Add helpers for /proc/self/fd accesses
It's not hard to grep for /proc/self/fd, but this way is a bit
cleaner and avoids typos, etc.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-07 09:30:55 -04:00
openshift-merge-bot[bot] 3ab2a4bbc0
Merge pull request #1952 from giuseppe/refactor-chunked-package
chunked: refactor file system ops in a new file and add some tests
2024-06-07 13:02:59 +00:00
Giuseppe Scrivano 42801b27de
chunked: add tests for filesystem operations
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 14:38:24 +02:00
Giuseppe Scrivano c4ba01f635
chunked: use filepath.Split() instead of Dir()/Base()
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 14:37:42 +02:00
Giuseppe Scrivano 4c716c8628
chunked: honor mode for mkdirat
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 08:32:24 +02:00
Giuseppe Scrivano 61a2c5ddf3
chunked: refactor args to openOrCreateDirUnderRoot()
follow the same pattern used by other functions

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano 7c61cfaee7
chunked: refactor args to openFileUnderRoot()
follow the same pattern used by other functions.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano c811876d13
chunked: fix opening parent dir
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano 2278cb1e0e
chunked: provide constructor for seekableFile
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 21:37:13 +02:00
Giuseppe Scrivano 37800897ab
chunked: split file operations to a new file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 13:11:46 +02:00
openshift-merge-bot[bot] de5b13bf73
Merge pull request #1951 from giuseppe/dump-handle-duplicates
dump: handle duplicates
2024-06-05 19:24:14 +00:00
openshift-merge-bot[bot] 50855b2ba0
Merge pull request #1949 from cgwalters/overlay-more-comments
graphdriver: Clarify name of fsverity const, tweak docs
2024-06-05 15:33:44 +00:00
Colin Walters 6ec1306ac3 graphdriver: Clarify name of fsverity const, tweak docs
- Add comments that make clear these are internal values
- Rename "Enabled" to "IfAvailable" to clarify what it does
- Also note that `DifferFsVerityRequired` is not actually set by anything
  in this codebase.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-05 10:53:56 -04:00
openshift-merge-bot[bot] 663f6e7c3e
Merge pull request #1947 from cgwalters/chunked-cloexec
chunked: Set O_CLOEXEC
2024-06-05 14:33:28 +00:00
Colin Walters 661531fb0d chunked: Set O_CLOEXEC
I was just reading the code and I have a mental checklist item
for "invoking open without O_CLOEXEC" that triggered here.
(See also e.g.
https://github.com/containers/composefs/pull/185#discussion_r1322925050
)

It has security-relevant properties for us, xref
CVE-2024-21626 for example.

This isn't the only missing variant of this in this codebase,
just using this targeted PR to test the waters for more PRs.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-05 09:20:07 -04:00
Giuseppe Scrivano 1848835383
dump: handle duplicates
Add validation for duplicate entries.  Duplicates are ignored, unless
there is a mismatch in the values.  In that case, an error is returned.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-05 13:59:15 +02:00
Giuseppe Scrivano 588dd2767c
dump: refactor dump_test.go
move all the test cases into a struct instead of declaring each of
them separately.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-05 13:13:42 +02:00
openshift-merge-bot[bot] c923593d5d
Merge pull request #1948 from giuseppe/use-escaped-name-for-check
dump: use the sanitized path for root check
2024-06-04 20:54:10 +00:00
Giuseppe Scrivano 06a03bd48f
dump: use the sanitized path for root check
otherwise if the root is stored as "./", it ends up adding the root
node twice causing mkcomposefs to fail.

Closes: https://github.com/containers/storage/issues/1941

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-04 22:12:18 +02:00
openshift-merge-bot[bot] 7857557d61
Merge pull request #1946 from containers/renovate/golang.org-x-sys-0.x
fix(deps): update module golang.org/x/sys to v0.21.0
2024-06-04 17:04:39 +00:00
renovate[bot] 4d8aba4e00
fix(deps): update module golang.org/x/sys to v0.21.0
Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-06-04 15:57:34 +00:00
openshift-merge-bot[bot] 986e170721
Merge pull request #1940 from giuseppe/fix-escape-for-composefs-dump
dump: replace unicode package with custom functions
2024-06-04 14:44:24 +00:00
Giuseppe Scrivano 12f34d8f6d
dump: replace unicode package with custom functions
To avoid a mismatch with the C composefs library that uses isgraph()
and isprint(), do not use the unicode package but provide the expected
definition for these functions.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-04 16:17:01 +02:00
openshift-merge-bot[bot] 45933193bd
Merge pull request #1942 from giuseppe/always-consume-tar-split-data
chunked: fix deadlock by always consuming tar-split
2024-06-04 13:38:23 +00:00
Giuseppe Scrivano 4595fa2aab
chunked: fix deadlock by always consuming tar-split
always consume the tar-split data when present to avoid blocking the
producer. Previously, the tar-split data was only read when the digest
was specified.

commit 6875c9fbcf introduced the
regression.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 23:01:34 +02:00
openshift-merge-bot[bot] 9661c8f9f8
Merge pull request #1939 from cgwalters/misc-doc-comments
compression: Add some doc comments
2024-06-03 20:30:45 +00:00
openshift-merge-bot[bot] 5310107af1
Merge pull request #1931 from akihikodaki/main
archive: always fix mode for root dir with ForceMask
2024-06-03 20:00:31 +00:00
Colin Walters d78daad6ca compression: Add some doc comments
I'm digging in more to zstd:chunked and I hope these comments are
useful.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-03 14:54:32 -04:00
openshift-merge-bot[bot] 5fe400b7ae
Merge pull request #1935 from coypoop/trivial-netbsd-patches
Trivial patches to minimize patches needed by NetBSD
2024-06-03 17:15:34 +00:00
openshift-merge-bot[bot] 44daeaa690
Merge pull request #1937 from giuseppe/chunked-fine-tune-threshold
chunked: change auto merge threshold to 1024
2024-06-03 15:53:01 +00:00
openshift-merge-bot[bot] 2312b28b2e
Merge pull request #1936 from giuseppe/storage-ignore-tar-split-without-digest
chunked: ignore the tar-split data if digest is empty
2024-06-03 14:25:00 +00:00
Giuseppe Scrivano 617a808a63
chunked: change auto merge threshold to 1024
Increase the threshold for auto-merging parts from 128 to 1024. This change
aims to reduce the number of parts in an HTTP multi-range request, thus
increasing the likelihood that the server will accept the request.

The previous threshold of 128 often resulted in a large number of small
ranges, which could lead to HTTP multi-range requests being rejected by
servers due to the excessive number of parts.

It partially addresses the reported issue.

Reported-by: https://github.com/containers/storage/issues/1928

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 15:00:35 +02:00
Giuseppe Scrivano 6875c9fbcf
chunked: ignore the tar-split data if digest is empty
if a digest was not specified in the TOC, ignore completely the
tar-split data.

Otherwise the clients fail to pull images created before commit
b5413c2bd6.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 13:48:49 +02:00
Maya Rashish 730997e4dd Share the options_freebsd.go between netbsd and freebsd
For packaging for NetBSD, we will patch this file anyway (we don't
have a hard-coded install directory for packages)

Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish be551d5171 Add netbsd to the list of bsd-like OSes
Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish 3c2e4b8da5 !windows instead of a long list of OSes that aren't windows
Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish 20f07857d5 Remove unused file
Between the _windows file and _unix file being used by non-windows,
we catch all cases.

The _others file is probably a maintenance burden and every new OS
needs to be excluded from building it.

Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
openshift-merge-bot[bot] 7ad4028f65
Merge pull request #1934 from giuseppe/remove-unreachable-code
overlay: remove unreachable code
2024-05-31 12:46:50 +00:00
Giuseppe Scrivano 0c883a5eeb
overlay: remove unreachable code
the removed code is not reachable since permsKnown must be true to be
in this code branch.

Closes: https://github.com/containers/storage/issues/1900

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-05-30 23:28:13 +02:00
openshift-merge-bot[bot] a4090c44cf
Merge pull request #1932 from containers/renovate/major-ci-vm-image
chore(deps): update dependency containers/automation_images to v20240529
2024-05-30 20:13:37 +00:00
renovate[bot] d93e44423c
chore(deps): update dependency containers/automation_images to v20240529
Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-05-29 23:09:09 +00:00
Akihiko Odaki 96545b8496 archive: always fix mode for root dir with ForceMask
We used to override the root directory permission only when the record
for the root directory exists in the archive. However,
docker.io/ubuntu:jammy-20240427 lacks such a record and not overriding
the root directory permission makes all other permission overrides
invalid.

Signed-off-by: Akihiko Odaki <odaki@rsg.ci.i.u-tokyo.ac.jp>
2024-05-29 08:42:06 +09:00
openshift-merge-bot[bot] 989e69d83a
Merge pull request #1929 from giuseppe/compose-fs-add-missing-dir-parents
composefs: add parent directory if missing
2024-05-28 13:16:15 +00:00