Commit Graph

29610 Commits

Author SHA1 Message Date
Colin Walters cfde71b024 chunked: Add helpers for /proc/self/fd accesses
It's not hard to grep for /proc/self/fd, but this way is a bit
cleaner and avoids typos, etc.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-07 09:30:55 -04:00
openshift-merge-bot[bot] 3ab2a4bbc0
Merge pull request #1952 from giuseppe/refactor-chunked-package
chunked: refactor file system ops in a new file and add some tests
2024-06-07 13:02:59 +00:00
Giuseppe Scrivano 42801b27de
chunked: add tests for filesystem operations
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 14:38:24 +02:00
Giuseppe Scrivano c4ba01f635
chunked: use filepath.Split() instead of Dir()/Base()
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 14:37:42 +02:00
Giuseppe Scrivano 4c716c8628
chunked: honor mode for mkdirat
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-07 08:32:24 +02:00
Giuseppe Scrivano 61a2c5ddf3
chunked: refactor args to openOrCreateDirUnderRoot()
follow the same pattern used by other functions

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano 7c61cfaee7
chunked: refactor args to openFileUnderRoot()
follow the same pattern used by other functions.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano c811876d13
chunked: fix opening parent dir
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 22:33:43 +02:00
Giuseppe Scrivano 2278cb1e0e
chunked: provide constructor for seekableFile
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 21:37:13 +02:00
Giuseppe Scrivano 37800897ab
chunked: split file operations to a new file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-06 13:11:46 +02:00
openshift-merge-bot[bot] de5b13bf73
Merge pull request #1951 from giuseppe/dump-handle-duplicates
dump: handle duplicates
2024-06-05 19:24:14 +00:00
openshift-merge-bot[bot] 50855b2ba0
Merge pull request #1949 from cgwalters/overlay-more-comments
graphdriver: Clarify name of fsverity const, tweak docs
2024-06-05 15:33:44 +00:00
Colin Walters 6ec1306ac3 graphdriver: Clarify name of fsverity const, tweak docs
- Add comments that make clear these are internal values
- Rename "Enabled" to "IfAvailable" to clarify what it does
- Also note that `DifferFsVerityRequired` is not actually set by anything
  in this codebase.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-05 10:53:56 -04:00
openshift-merge-bot[bot] 663f6e7c3e
Merge pull request #1947 from cgwalters/chunked-cloexec
chunked: Set O_CLOEXEC
2024-06-05 14:33:28 +00:00
Colin Walters 661531fb0d chunked: Set O_CLOEXEC
I was just reading the code and I have a mental checklist item
for "invoking open without O_CLOEXEC" that triggered here.
(See also e.g.
https://github.com/containers/composefs/pull/185#discussion_r1322925050
)

It has security-relevant properties for us, xref
CVE-2024-21626 for example.

This isn't the only missing variant of this in this codebase,
just using this targeted PR to test the waters for more PRs.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-05 09:20:07 -04:00
Giuseppe Scrivano 1848835383
dump: handle duplicates
Add validation for duplicate entries.  Duplicates are ignored, unless
there is a mismatch in the values.  In that case, an error is returned.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-05 13:59:15 +02:00
Giuseppe Scrivano 588dd2767c
dump: refactor dump_test.go
move all the test cases into a struct instead of declaring each of
them separately.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-05 13:13:42 +02:00
openshift-merge-bot[bot] c923593d5d
Merge pull request #1948 from giuseppe/use-escaped-name-for-check
dump: use the sanitized path for root check
2024-06-04 20:54:10 +00:00
Giuseppe Scrivano 06a03bd48f
dump: use the sanitized path for root check
otherwise if the root is stored as "./", it ends up adding the root
node twice causing mkcomposefs to fail.

Closes: https://github.com/containers/storage/issues/1941

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-04 22:12:18 +02:00
openshift-merge-bot[bot] 7857557d61
Merge pull request #1946 from containers/renovate/golang.org-x-sys-0.x
fix(deps): update module golang.org/x/sys to v0.21.0
2024-06-04 17:04:39 +00:00
renovate[bot] 4d8aba4e00
fix(deps): update module golang.org/x/sys to v0.21.0
Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-06-04 15:57:34 +00:00
openshift-merge-bot[bot] 986e170721
Merge pull request #1940 from giuseppe/fix-escape-for-composefs-dump
dump: replace unicode package with custom functions
2024-06-04 14:44:24 +00:00
Giuseppe Scrivano 12f34d8f6d
dump: replace unicode package with custom functions
To avoid a mismatch with the C composefs library that uses isgraph()
and isprint(), do not use the unicode package but provide the expected
definition for these functions.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-04 16:17:01 +02:00
openshift-merge-bot[bot] 45933193bd
Merge pull request #1942 from giuseppe/always-consume-tar-split-data
chunked: fix deadlock by always consuming tar-split
2024-06-04 13:38:23 +00:00
Giuseppe Scrivano 4595fa2aab
chunked: fix deadlock by always consuming tar-split
always consume the tar-split data when present to avoid blocking the
producer. Previously, the tar-split data was only read when the digest
was specified.

commit 6875c9fbcf introduced the
regression.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 23:01:34 +02:00
openshift-merge-bot[bot] 9661c8f9f8
Merge pull request #1939 from cgwalters/misc-doc-comments
compression: Add some doc comments
2024-06-03 20:30:45 +00:00
openshift-merge-bot[bot] 5310107af1
Merge pull request #1931 from akihikodaki/main
archive: always fix mode for root dir with ForceMask
2024-06-03 20:00:31 +00:00
Colin Walters d78daad6ca compression: Add some doc comments
I'm digging in more to zstd:chunked and I hope these comments are
useful.

Signed-off-by: Colin Walters <walters@verbum.org>
2024-06-03 14:54:32 -04:00
openshift-merge-bot[bot] 5fe400b7ae
Merge pull request #1935 from coypoop/trivial-netbsd-patches
Trivial patches to minimize patches needed by NetBSD
2024-06-03 17:15:34 +00:00
openshift-merge-bot[bot] 44daeaa690
Merge pull request #1937 from giuseppe/chunked-fine-tune-threshold
chunked: change auto merge threshold to 1024
2024-06-03 15:53:01 +00:00
openshift-merge-bot[bot] 2312b28b2e
Merge pull request #1936 from giuseppe/storage-ignore-tar-split-without-digest
chunked: ignore the tar-split data if digest is empty
2024-06-03 14:25:00 +00:00
Giuseppe Scrivano 617a808a63
chunked: change auto merge threshold to 1024
Increase the threshold for auto-merging parts from 128 to 1024. This change
aims to reduce the number of parts in an HTTP multi-range request, thus
increasing the likelihood that the server will accept the request.

The previous threshold of 128 often resulted in a large number of small
ranges, which could lead to HTTP multi-range requests being rejected by
servers due to the excessive number of parts.

It partially addresses the reported issue.

Reported-by: https://github.com/containers/storage/issues/1928

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 15:00:35 +02:00
Giuseppe Scrivano 6875c9fbcf
chunked: ignore the tar-split data if digest is empty
if a digest was not specified in the TOC, ignore completely the
tar-split data.

Otherwise the clients fail to pull images created before commit
b5413c2bd6.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-06-03 13:48:49 +02:00
Maya Rashish 730997e4dd Share the options_freebsd.go between netbsd and freebsd
For packaging for NetBSD, we will patch this file anyway (we don't
have a hard-coded install directory for packages)

Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish be551d5171 Add netbsd to the list of bsd-like OSes
Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish 3c2e4b8da5 !windows instead of a long list of OSes that aren't windows
Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
Maya Rashish 20f07857d5 Remove unused file
Between the _windows file and _unix file being used by non-windows,
we catch all cases.

The _others file is probably a maintenance burden and every new OS
needs to be excluded from building it.

Signed-off-by: Maya Rashish <maya@NetBSD.org>
2024-06-01 08:47:34 -04:00
openshift-merge-bot[bot] 7ad4028f65
Merge pull request #1934 from giuseppe/remove-unreachable-code
overlay: remove unreachable code
2024-05-31 12:46:50 +00:00
Giuseppe Scrivano 0c883a5eeb
overlay: remove unreachable code
the removed code is not reachable since permsKnown must be true to be
in this code branch.

Closes: https://github.com/containers/storage/issues/1900

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-05-30 23:28:13 +02:00
openshift-merge-bot[bot] a4090c44cf
Merge pull request #1932 from containers/renovate/major-ci-vm-image
chore(deps): update dependency containers/automation_images to v20240529
2024-05-30 20:13:37 +00:00
renovate[bot] d93e44423c
chore(deps): update dependency containers/automation_images to v20240529
Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-05-29 23:09:09 +00:00
Akihiko Odaki 96545b8496 archive: always fix mode for root dir with ForceMask
We used to override the root directory permission only when the record
for the root directory exists in the archive. However,
docker.io/ubuntu:jammy-20240427 lacks such a record and not overriding
the root directory permission makes all other permission overrides
invalid.

Signed-off-by: Akihiko Odaki <odaki@rsg.ci.i.u-tokyo.ac.jp>
2024-05-29 08:42:06 +09:00
openshift-merge-bot[bot] 989e69d83a
Merge pull request #1929 from giuseppe/compose-fs-add-missing-dir-parents
composefs: add parent directory if missing
2024-05-28 13:16:15 +00:00
Giuseppe Scrivano 6b1c044dc1
composefs: add parent directory if missing
it solves this error with mkcomposefs when the TOC doesn't specify the
parent directory for an entry:

Error: committing the finished image: failed to put layer using a partial pull: failed to convert json to erofs: exit status 1: mkcomposefs: Parent directory missing for /usr/share/locale/ca/LC_MESSAGES/libc.mo

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2024-05-28 11:52:30 +02:00
openshift-merge-bot[bot] c42fcc5cc2
Merge pull request #1930 from containers/renovate/github.com-burntsushi-toml-1.x
fix(deps): update module github.com/burntsushi/toml to v1.4.0
2024-05-23 17:43:39 +00:00
renovate[bot] f672531ba1
fix(deps): update module github.com/burntsushi/toml to v1.4.0
Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-05-23 16:56:18 +00:00
openshift-merge-bot[bot] 5cd00c56a8
Merge pull request #1927 from mtrmac/three-people-review
Fix locking bugs from #1916, and one more
2024-05-22 21:53:25 +00:00
Miloslav Trmač 830c7c6383 Fix lock hierarchy
As documented at the top of the store type,
the roLayerStores must be locked _before_
the container store.

This is a minimal / conservative fix; this
probably could use a read-only lock only for the
duration of the "LayerID == parent" check.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-05-22 19:14:06 +02:00
Miloslav Trmač 7ca2113a63 Move the containerStore locking inside putLayer
There's no need for the callers to deal with this,
and now the API of the function is much less surprising.

Also fix the documentation about locking rlstores.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-05-22 19:13:21 +02:00
Miloslav Trmač 5bf717bb16 Add a comment to help future maintainers
Should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-05-22 19:13:21 +02:00