For Chunk::External we can theoretically return multiple chunks for
files that have been subdivided (by the rollsum, for example). I think
it's time to lay this idea for rest, at least for the time being, and
improve the ergonomics and efficiency of the API by replacing the
Box<[ContentReference]> with a single ContentReference.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
🦎
Use the oci-client crate to pull a zstd:chunked image using something
approximating a performant and robust downloader.
For now this just creates a "tmp" directory in the current directory and
puts the chunks there. It doesn't actually do anything with the data
once it's downloaded, but at that point it would be easy to reassemble
it into some form.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
I chose HashMap because it would work nicely with oci-spec, but
oci-client uses a BTreeMap instead. Let's just use a getter function.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
Add a 'profiling' profile and make some initial changes based on the
results.
The big win: parsing JSON directly out of a zstd::Decoder turns out to
be pretty slow, even if we add a BufReader, so decompress it in one go
first and use the serde_json slice API on the result.
A smaller win: instead of using an iterator-based JSON parser for the
tar-split, just split on lines and parse each line separately.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
Allison Karlitskaya