Commit Graph

18 Commits

Author SHA1 Message Date
Kir Kolyshkin a4d8f720a2 Format sources with gofumpt
gofumpt is a superset of gofmt, enabling some more code formatting
rules.

This commit is brought to you by

	gofumpt -w .

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2023-05-26 16:17:31 -07:00
Daniel J Walsh a3204cf7e8
Move to golang 1.18 and later
Github.com is reporting security issues on older versions of
golang.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2023-04-03 15:26:54 -04:00
Miloslav Trmač a1ccc9d862 Use os.WriteFile instead of ioutil.WriteFile
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:31:34 +02:00
Miloslav Trmač 7635db182b Use io.ReadAll instead of ioutil.ReadAll
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2022-09-12 16:30:46 +02:00
Sascha Grunert 3455d12729
Switch to golang native error wrapping
We now use the golang error wrapping format specifier `%w` instead of the
deprecated github.com/pkg/errors package.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2022-07-07 13:22:46 +02:00
Giuseppe Scrivano 26c561f9a6
chunked: fix read layer cache
if the layer cache doesn't already exist, automatically create it from
the layer TOC.

commit 10697a05a2 introduced this
regression.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-02-14 12:36:00 +01:00
Giuseppe Scrivano 198820877c
pkg/chunked: add support for sparse files
automatically detect holes in sparse files (the threshold is hardcoded
at 1kb for now) and add this information to the manifest file.

The receiver will create a hole (using unix.Seek and unix.Ftruncate)
instead of writing the actual zeros.

Closes: https://github.com/containers/storage/issues/1091

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-13 13:32:13 +01:00
Giuseppe Scrivano fd89b93ef3
chunked: add tests for the cache
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-13 13:32:13 +01:00
Giuseppe Scrivano ab25eafc17
cache: parse the correct field for offset
commit 10697a05a2 introduced the issue.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-13 12:15:08 +01:00
Giuseppe Scrivano 96c0403bb1
cache: store correctly the digestLen field
commit 10697a05a2 introduced the issue.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-13 12:15:08 +01:00
Giuseppe Scrivano 31b28dbedf
chunked: use a RWMutex for the cache
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 0621da79cc
chunked: improve json parsing
reduce the number of allocations done by the parser by reading into a
bytes.Buffer.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 048f7c08ad
chunked: use json-iterator
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 10697a05a2
chunked: implement lookaside cache
avoid parsing each json TOC file for the layers in the local storage,
but attempt to create a lookaside cache in a custom format faster to
load (and potentially be mmap'able).

The same cache is used to lookup files, chunks and candidates for
deduplication with hard links.

There are 3 kind of digests stored:

- digest(file.payload))
- digest(digest(file.payload) + file.UID + file.GID + file.mode + file.xattrs)
- digest(i) for each i in chunks(file payload)

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-10 11:27:42 +01:00
Giuseppe Scrivano 526c57d8b0
chunked: reuse cache
try to reuse an existing cache object, instead of creating it for
every layer.

Set a time limit on how long it can be reused so to clean up stale
references.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-07 21:28:16 +01:00
Giuseppe Scrivano be4e8f622d
chunked: move copy logic to storage_linux.go
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-07 21:28:16 +01:00
Giuseppe Scrivano bfd9c8046e
chunked: chunk deduplication
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2022-01-07 21:28:15 +01:00
Giuseppe Scrivano f18141fa76
chunked: move cache to separate file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2021-12-24 13:28:25 +01:00