Commit Graph

171 Commits

Author SHA1 Message Date
Tianon Gravi f7926308bc Switch most "git" operations over to https://github.com/src-d/go-git for a very dramatic speed increase (especially during dependency calculation) 2019-06-12 08:06:45 -07:00
Tianon Gravi c67f06ee5a Remove "apply-constraints" from Travis to make the test more useful (now that the command works properly) 2019-06-12 08:06:45 -07:00
Tianon Gravi 2781cc2621 Apply gofmt 2019-06-12 08:06:45 -07:00
Tianon Gravi 157882047c Improve Dockerfile fetching error reporting and "bashbrew children" / "bashbrew from" handling (especially in the case of no "--apply-constraints" flag) 2019-06-12 08:06:45 -07:00
Tianon Gravi 5bf7a3d9bd Drop "ADD --from=" (which isn't supported by Docker) 2019-06-12 08:06:45 -07:00
Tianon Gravi b939318aa7 Add new "ArchLastStageFrom" function for getting the effective "base" of the given image (needed for docs generation) 2019-06-12 08:06:45 -07:00
Tianon Gravi 62995582c7 Remove "DockerFrom" function entirely to cause scripts to fail and force migration 2019-06-12 08:06:45 -07:00
Tianon Gravi a243c72bda Unexpose "DockerfileMetadata" and add new exposed "DockerFroms" (since that's what's really necessary externally from that internal structure) 2019-06-12 08:06:45 -07:00
Tianon Gravi da87baad5a Add initial "multi-stage" support in bashbrew
This allows bashbrew to properly handle cross-repository and cross-tag dependencies even in the face of multiple `FROM` instructions or `COPY --from=`.

This also provides the scaffolding necessary to implement this in scripts using `bashbrew cat`.

As fallback behavior, the `*DockerFrom` functions should return the `FROM` of the last stage in the `Dockerfile` (which is essentially the `FROM` of the final image).

Also, the output of `bashbrew from` is now a space-separated list.
2019-06-12 08:06:45 -07:00
Tianon Gravi 28a21ad94b Apply 7d4d3222cd ("path.Clean" earlier so "bashbrew cat" shows the cleaned paths appropriately too) 2019-06-05 15:55:17 -07:00
Tianon Gravi 9f26ec45e1 Tag our put-multiarch image and explicitly use itself for cache 2019-05-10 17:31:11 -07:00
Tianon Gravi ef26a55720 Fix "arm32" vs "arm" bug 2019-05-10 17:21:35 -07:00
Tianon Gravi d1cb5996f7 Add optional "--dry-run" flag for pre-testing "heavy" pushes 2019-05-10 16:55:53 -07:00
Tianon Gravi db8b401468 Treat 401 the same as 404 when fetching a manifest 2019-05-10 16:38:41 -07:00
Tianon Gravi af23605e63 Add new "put-multiarch" script
This is a replacement for `bashbrew put-shared` (without `--single-arch`) that takes some shortcuts to get a pretty decent speed gain which should allow us to go back to the naïve solution of "push everything from each arch `:xyz-tag` to the multiarch repo's `:xyz-tag` and still have reasonable performance.

This was tested by pushing all ~3700 tags we currently have to https://hub.docker.com/u/trollin and for a full no-op we were around 8.5 minutes, and for a full forced push of every tag manifest (with no necessary blob mounts) we were around 45 minutes.
2019-05-09 16:53:24 -07:00
Tianon Gravi 1c24c31208 Update bashbrew readme (thanks @7AC) 2019-04-04 13:25:24 -07:00
Tianon Gravi 73d98151b5 First attempt to fix Travis (force Go modules "on") 2019-03-28 21:39:21 -07:00
Tianon Gravi d5559299cf Port bashbrew to use Go modules instead of GB
Something in here also fixes the "missing build output on error" bug we've been seeing on Jenkins (see https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/rabbitmq/184/console for example).
2019-03-28 15:45:12 -07:00
Tianon Gravi 26d2eab54b Update Bashbrew builds to Go 1.12
(and update `manifest-tool` to 1.0.0-rc; bombs away, @estesp!)
2019-03-14 14:52:22 -07:00
Tianon Gravi 079cc45216 Fix bashism 2019-03-12 22:10:05 -07:00
Tianon Gravi ccf4f480a7 Hotfix: "bashbrew build" sorting fix 2019-01-28 17:01:16 -08:00
Tianon Gravi a93f9f538f Improve bashbrew SharedTags handling (explicitly allowed for "bashbrew cat", etc) 2019-01-28 16:16:50 -08:00
Tianon Gravi 1693d9a9e6 Update go-dockerlibrary to include https://github.com/docker-library/go-dockerlibrary/pull/19 2019-01-07 16:22:36 -08:00
Tianon Gravi 2dee8376c4 Update to manifest-tool 0.9.0 2018-11-26 20:22:07 -08:00
Tianon Gravi ed5b868911 Update "gpg" invocations to use "--batch"
See https://bugs.debian.org/913614
2018-11-14 13:08:09 -08:00
Tianon Gravi 52138c6dbe Add "naughty-from.sh" script inspired by the report of the same name
This verifies that the "Architectures" field of each entry in a given pull request is a subset of its parent.
2018-10-29 22:44:56 -07:00
Tianon Gravi 0335845df0 Stop testing "extra commands" on bashbrew changes (so they'll succeed more often) 2018-10-29 21:47:25 -07:00
Tianon Gravi 6a1a2d2c1d Add support for "File:" in manifest files
See https://github.com/docker-library/go-dockerlibrary/pull/16 for format implementation details.
2018-10-22 16:06:21 -07:00
Tianon Gravi 5cbc68680a Add new "archFilter" template function
Example usage:

```
{{- range $i, $e := ($archSpecific | ternary (archFilter arch $.Entries) $.Entries) -}}
    ...
{{- end -}}
```
2018-08-23 11:37:13 -07:00
Tianon Gravi 1e37e9b879 Update bashbrew release binaries to Go 1.10 2018-08-14 12:51:30 -07:00
Tianon Gravi 9911b85e11 Slightly improved error message for failed template executions 2018-08-14 12:48:04 -07:00
Tianon Gravi ae92f4d748 Add a note about pre-built bashbrew binaries 2018-07-20 13:10:57 -07:00
Tianon Gravi afbe0d6f03 Improve put-shared noop behavior and add "--force" flag for pushing operations 2018-05-03 12:21:57 -07:00
Tianon Gravi 15a94e48fa Add "gpgconf --kill all" to bashbrew build Dockerfile (partially as a test for other images)
This helps combat the following race condition:

    + rm -rf /tmp/tmp.FHddOB
    rm: can't remove '/tmp/tmp.FHddOB/S.gpg-agent.browser': No such file or directory
2018-04-06 09:28:44 -07:00
Tianon Gravi b4cc45e64f Adjust wording 2018-03-20 10:11:34 -07:00
Tianon Gravi f3aced2ea4 Fix grep logic 2018-03-20 10:11:34 -07:00
Tianon Gravi 4e74853942 Add simple Travis test for incorrect "latest" tags
```console
$ TRAVIS_BRANCH=master TRAVIS_PULL_REQUEST=false ./bashbrew/travis.sh
Testing master -- BUILD ALL THE THINGS!

Incorrectly 'latest' tags detected:
  geonetwork:latest-postgres

Read https://github.com/docker-library/official-images#tags-and-aliases for more details.

```
2018-03-20 10:11:34 -07:00
Tianon Gravi b84de49388 Add missing newline 2018-01-24 10:12:23 -08:00
Tianon Gravi f36ef9d30b Update "put-shared" to allow some tags to fail per-repo (and collect errors) 2018-01-24 09:57:47 -08:00
Tianon Gravi 90101d3445 Also tag SharedTags (and include them in dependency calculations, etc)
This will include SharedTags directly in "bashbrew list", etc, but explicitly does _not_ include them during "bashbrew push".  This allows for other images to be "FROM" a SharedTags value, and should be reasonably safe since any given SharedTag should only be applicable for a given architecture at most once.
2017-10-24 10:47:17 -07:00
Tianon Gravi 5d0cbc967e Update to manifest-tool v0.7.0
With this release, we get to drop our custom patch for pushing to `library/xxx`, and go back to using the officially released upstream binaries (which now exist for all the same architectures we support).
2017-10-06 16:12:43 -07:00
Tianon Gravi ec5b4e484e Add new "archNamespace" template function for parsing "--arch-namespace" parameters
Example usage: `{{ archNamespace "windows-amd64" }}` => `winamd64`

This is intended for use in https://github.com/docker-library/docs to help create a list of "Supported architectures" for each image which links to each supported architecture repository (and if supported by Bashbrew itself, allows us to keep https://github.com/docker-library/docs isolated from Jenkins assumptions while still allowing Jenkins to contain the single-source-of-truth for arch-to-namespace mappings).
2017-10-02 12:27:10 -07:00
Tianon Gravi 50e9eb1ddd Add a "--single-arch" flag to "put-shared" for pushing SharedTags to architecture-specific namespaces 2017-09-12 16:18:27 -07:00
Tianon Gravi 0da6c3c534 Minor bashbrew de-optimization 2017-09-12 13:32:45 -07:00
Tianon Gravi dca13c025b Build manifest-tool from source to apply a minor patch 2017-09-11 15:52:01 -07:00
Tianon Gravi 9aedfc4915 Add a "--dry-run" flag to "build", "tag", "push", and "put-shared" 2017-09-07 13:53:58 -07:00
Tianon Gravi 1568683373 CUE LOUD GRUMBLING 2017-09-06 13:11:15 -07:00
Tianon Gravi 145add96da Fix minor typo 2017-09-05 07:36:47 -07:00
Tianon Gravi 3b0c4503cb Use the Hub v2 API for a dramatic no-op push speedup
This also has the benefit of only triggering automated builds when there have been actual changes to the image.

This works by comparing the Hub's `last_updated` timestamp to the `Created` timestamp of local images (in the case of `bashbrew push`), or to the `last_updated` timestamp of the images which make up the `SharedTags` and/or multiarch images (in the case of `bashbrew put-shared`).

This is a dramatic speedup because the Hub v2 API is faster (and has no auth for public repos) than the registry API.  If there are any errors in the fetching of `last_updated` from the Hub, we massage the timestamps such that the push simply always happens, as before (which will be the case for pushes to non-Hub registries, for example).
2017-08-28 09:46:47 -07:00
Tianon Gravi c4108a0534 Update go-dockerlibrary to pull in the new "getenv" function
See https://github.com/docker-library/go-dockerlibrary/pull/15.
2017-07-12 16:15:18 -07:00