The cache implementations are recording both the base and specific compression variant;
CandidateLocations2 all call CandidateTemplateWithCompression to choose the
appropriate variants to return based on CandidateLocations2Options.
This way, neither the BIC implementations nor the transports are not responsible for
converting zstd:chunked entries to zstd entries if the user wants the latter.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
We will want to record more than a single alghoritm name. For now,
just introduce the structure and modify users, we'll add the new fields
later.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... and add CandidateWithLocation and CandidateWithUnknownLocation ,
so that the BIC implementations only need to deal with one value
instead of carrying around three; we will want to add one more,
and encapsulating them all into a single template will make it
transparent to the cache implementations.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The new code is not called, so it should not change behavior
(apart from extending the BoltDB/SQLite schema).
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The implementations (sharing the code in pkg/blobinfocache/internal/prioritize)
already need to look up the algorithm, so just return the value as a part of
a BICReplacementCandidate2, instead of looking it up again in the transport.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
The primary benefit is that we now filter the algorithms _before_ collecting
and trimming the list of candidates, i.e. if the most recent candidates
are not matching the requirements, CandidateLocations2 will return older
candidates instead of returning just some unwanted candidates which will be rejected.
This requires us to look up the actual compression in the BIC code, so this also
changes pkg/blobinfocache/internal/test to use real algorithm names.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... into candidateLocations and appendReplacementCandidates.
We will add more conditions there soon.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
... and move the canSubstitute parameter to it.
We will add more options, and we well want to carry them around
as a unit.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
It seems we try to reuse blobs only for the specified registry, however
we can have valid known compressed digests across registry as well
following pr attempts to use that by doing following steps.
* `CandidateLocations2` now processes all known blobs and appends them
to returned candidates at the lowest priority. As a result when
`TryReusingBlob` tries to process these candidates and if the blobs
filtered by the `Opaque` set by the `transport` fail to match then
attempt is made against all known blobs (ones which do not belong to the
current registry).
* Increase the sample set of potential blob reuse to all known
compressed digests , also involving the one which do not belong to
current registry.
* If a blob is found match it against the registry where we are
attempting to push. If blob is already there consider it a `CACHE
HIT!` and reply skipping blob, since its already there.
How to verify this ?
* Remove all images `buildah rmi --all` // needed so all new blobs can
be tagged again in common bucket
* Remove any previous `blob-info-cache` by
```console
rm /home/<user>/.local/share/containers/cache/blob-info-cache-v1.boltdb
```
```console
$ skopeo copy docker://registry.fedoraproject.org/fedora-minimal docker://quay.io/fl/test:some-tag
$ buildah pull registry.fedoraproject.org/fedora-minimal
$ buildah tag registry.fedoraproject.org/fedora-minimal quay.io/fl/test
$ buildah push quay.io/fl/test
```
```console
Getting image source signatures
Copying blob a3497ca15bbf skipped: already exists
Copying config f7e02de757 done
Writing manifest to image destination
Storing signatures
```
Signed-off-by: Aditya R <arajan@redhat.com>
Cache implementations still must work without Open/Close being called
because the v1 interface is public, but we can call Open/Close in copy.Image
to amortize the overhead cost a bit.
Currently all Open/Close methods are empty, so this should not change
behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Extend the blob info cache to also cache the name of the type of
compression used on a blob that we've seen, or specific values that
indicate that we know the blob was not compressed, or that we don't
know whether or not it was compressed.
New methods for adding known blob-compression pairs and reading
candidate locations including compression information are part of a new
internal BlobInfoCache2 interface which the library's BlobInfoCache
implementors also implement.
When we copy a blob, try to record the state of compression for the
source blob, and if we applied any changes, the blob we produced.
Make sure that when TryReusingBlob successfully uses a blob from the
blob info cache, that it provides compression information in the
BlobInfo that it returns, so that manifests can be updated to describe
layers using the correct MIME types.
When attempting to write a manifest, if a manifest can't be written
because layers were compressed using an algorithm which can't be
expressed using that manifest type, continue on to trying other manifest
formats.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
... so that major-version-aware Go module import
(as opposed to vX.Y.Z+incompatible, which does not allow different
packages to use different versions) works right.
Also requires adding some more GO111MODULE=on options to Makefile.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
This allows consumers who don't need the defaulting logic to pick
their desired implementation without pulling in unrelated
dependencies. For example, it allows you to consume the memory or
NoCache implementations without pulling in github.com/boltdb.
I'd initially included blobinfocache wrappers to preserve backwards
compatibility with the old API while consumers updated to the new
APIs, but Miloslav said we don't have any backwards-compatibility
commitments at the moment [1].
Having single-implementation subpackages also allows for more generic
names (e.g. NewMemoryCache is now just New), because memory.New is
clear enough while memory.NewMemoryCache stutters.
[1]: https://github.com/containers/image/pull/602#issuecomment-471570789
Signed-off-by: W. Trevor King <wking@tremily.us>
Miloslav Trmač