* WIP: first pass at deploy
* WIP: second pass, now with more cowbell
* WIP: refactor coverage handling (cleaner, more consistent, no longer has to clobber top-level bin/)
* WIP: the start of more "jq" tests
* WIP: add a few TODOs
* Add a benchmark for `om.OrderedMap.Set`
I was testing a minor memory-usage improvement to `Set`, but it turns out it doesn't actually matter (and this helped me determine that, so I might as well keep it).
* Add explicit `Reference.StringWithKnownDigest` unit test
* WIP: refactor EnsureManifest loop with more correct handling of child manifests vs blobs
* Update to use the new `ociregistry.HTTPError` for more consistent/correct HTTP error handling
* WIP: remove TODO that was implemented elsewhere (and fix error message text / comment text)
* WIP: also normalize descriptor field ordering
* WIP: assume pre-normalized platform (no reason to normalize more than once)
* WIP: initial "deploy" data munging helpers plus tests
* WIP: update Jenkinsfile.deploy to use new deploy code
* WIP: remove example-commands symlink so Git detects rename better
* WIP: add delay for racy registry startup
* WIP: remove trap once it's no longer necessary
* WIP: typo
* WIP: remove unnecessary TODOs
This adjusts our data model to store/track data from the registry as an image index (with more data fidelity) instead of as a custom data structure. One of the most notable benefits is that we can now track the annotation objects for each architecture, but even more importantly, this allows us to generate the indexes we need to push in deploy directly from data that's on-disk (where the old structure would require querying the registry).
Notably, however, this does *not* change deploy (yet) -- those changes are still in progress, but this was a large enough refactor/rewrite that I figured I should start here.
This also switches us from using containerd's registry client library to using [`cuelabs.dev/go/oci/ociregistry`](https://pkg.go.dev/cuelabs.dev/go/oci/ociregistry), which is much closer to the distribution-spec APIs (modeled/designed after them, in fact), where containerd's (and frankly most others) are a higher-level abstraction. This is important because we're running into raw number of request rate limits, and almost every library for this always starts with a `HEAD` before pushing content, and this will allow us to go directly for a `PUT` (and then only do the "copy child objects" dance if the `PUT` fails).
This also drops our `allTags` data entry (which we only *ever* used the first value of, and then only to identify a `sourceId`/`buildId` in a human-meaningful way, because it's not useful/meaningful for any other use case), and moves `tags` from being arch-specific up to the per-source-object level because it's always identical across all arches for a given `sourceId` so it's silly to copy it N times for every architecture object.
This makes a *very* dramatic difference in the speed of `sources.sh`.
Before:
```console
$ time .scripts/sources.sh $(cat subset.txt) > sources.json
...
real 6m13.444s
user 2m51.334s
sys 3m29.256s
```
After:
```console
$ time .scripts/sources.sh $(cat subset.txt) > sources.json
...
real 0m31.238s
user 0m32.769s
sys 0m1.375s
```
This adds a `--cache` flag to builds which will read from a cache file and write back out to it a set of images that are ~safe to cache (and because it's just a file, we can clear cache entries trivially by deleting the relevant entries from the file).
This also adds better 429 behavior that's twofold:
1. cap our maximum registry requests at ~500/min with an initial burst of 100 unless we hit a 429 (which should, in theory, keep us closer to staying under the Hub abuse rate limits)
2. actually retry requests when they give us a 429 response (capping both new requests and retries at the same ~500/min which is ~8/sec)
As a final bonus, I added/generated an appropriate cache file for our local test suite which brings the total number of actual Hub/registry requests our test suite makes down to one single image lookup.
Tianon Gravi