Commit Graph

1121 Commits

Author SHA1 Message Date
Hidde Beydals 7ff8da432e
Merge pull request #459 from fluxcd/libgit2-semver-flake 2021-10-22 13:43:40 +02:00
Hidde Beydals 9ff5334688 go-git: ensure original tag is used for TS lookup
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 13:25:24 +02:00
Hidde Beydals 4c94f88ebb libgit2: ensure original tag is used for TS lookup
This adds an additional selection candidate, to ensure test flakiness
happens more often.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 13:19:34 +02:00
Hidde Beydals b35d7d88f2
Merge pull request #458 from fluxcd/libgit2-simple-tag 2021-10-22 12:40:45 +02:00
Hidde Beydals 56201f30fe libgit2: Free most objects
This commit ensures most of the `git2go` objects `Free` themselves from
the underlying C object.

Ensuring all objects are freed is not possible yet, due to the way
commits are wired in to facilitate verification later on. In a later
follow up, we should change this and e.g. validate as part of the
checkout process, and move the implementation specific authentication
configuration from `git` into `libgit2`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 12:01:45 +02:00
Hidde Beydals d9473d008c libgit2: add remaining checkout strategy tests
This commit is a follow up on 4dc3185c5f
and adds tests for the remaining checkout strategies, while
consolidating some of the logic.

The consolidated logic ensures that (SemVer) tag and commit checkouts
happen using the same "checkout detached HEAD" logic.
The branch checkout is left unmodified, and simply checks out at the
current HEAD of the given branch.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 12:01:41 +02:00
Hidde Beydals 4dc3185c5f
Merge pull request #457 from fluxcd/libgit2-simple-tag 2021-10-21 17:34:45 +02:00
Hidde Beydals e736493730 libgit2: correctly resolve (annotated) tags
In d0560e5 the SemVer implementations were aligned, and the logic was
simplified a bit (or so I thought). This did however result in the
introduction of a regression, as it failed to take "simple tags" into
account.

This commit ensures both are taken into account again, and ensures it
is now covered by a proper test.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-21 17:10:43 +02:00
Hidde Beydals 79c19adf3f
Merge pull request #453 from dholbach/fix-contributing-sections 2021-10-18 16:52:24 +02:00
Daniel Holbach a7a1d06d83 'installing required deps' should not be under comms
Signed-off-by: Daniel Holbach <daniel@weave.works>
2021-10-18 15:55:45 +02:00
Hidde Beydals c574637c54
Merge pull request #434 from pa250194/gcp-bucket-provider 2021-10-14 23:05:17 +02:00
pa250194 8f0ea2e8af Merge branch 'gcp-bucket-provider' of https://github.com/pa250194/source-controller into gcp-bucket-provider 2021-10-14 14:57:58 -05:00
pa250194 c4e4b3928c Added Logger to closing GCP client
Signed-off-by: pa250194 <pa250194@ncr.com>

Added log for GCP provider auth error

Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:57:48 -05:00
pa250194 7921caf056 Updated docs to include GCP provider instructions
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:26:33 -05:00
pa250194 5077c1f9f6 Added more tests and cleaned up GCP provider logic
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:26:33 -05:00
pa250194 c98130548e Added Tests to GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:21:15 -05:00
pa250194 99c79bffaa Tests for GCP Bucket Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:19:19 -05:00
pa250194 be1ed50ac4 Service Account Key Authentication to GCP Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:16:53 -05:00
pa250194 2baa8a289c Added Comments for reconcileWithGCP and reconcileWithMinio
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:10:29 -05:00
pa250194 39811ed46a Add Support for GCP storage with workload identity
Added Support for Google Cloud Storage with Workload Identity as
Source Provider. This enables the use of GCP without enabling S3
compatible access.

Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:07:11 -05:00
pa250194 38bf4d9859 Fixed spelling and capitalization
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 13:51:02 -05:00
Hidde Beydals e2548cbe57 Update fluxcd/golang-with-libgit2 to 1.1.1-1
This includes a tiny fix for Darwin to ensure the generated `.pc`
file includes the right paths.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:49:52 -05:00
Hidde Beydals c2495ae408 Fix generation of API documentation
The version was accidentally set to an invalid version, causing the
API documentation generation to fail.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:49:52 -05:00
Hidde Beydals 96ab646cd4 Release v0.16.0
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:49:52 -05:00
Dylan Arbour 5e6abae9e8 Add ReconcileStrategy to HelmChart
This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which
allows defining when a new chart should be packaged and/or published if it
originates from a `Bucket` or `GitRepository` resource.

The two available strategies are:

- `ChartVersion`: creates a new artifact when the version of the Helm chart as
  defined in the `Chart.yaml` from the Source is different from the current
  version.
- `Revision`: creates a new artifact when the revision of the Source is
  different from the current revision.

For the `Revision` strategy, the (checksum part of the) revision of the
artifact the chart originatesfrom is added as SemVer metadata.

A chart from a `GitRepository` with Artifact revision
`main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer
`0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

A chart from a `Bucket` with Artifact revision
`f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0`
will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-10-14 13:49:52 -05:00
Stefan Prodan 6fe6f07d5e Update containerd and runc to fix CVEs
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-10-14 13:49:52 -05:00
Hidde Beydals 66fffe103e CONTRIBUTING: include pkg-config as macOS dep
As this isn't available on Darwin by default, unlike on most Linux
distributions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals 610131956e Update Dockerfile used in tests as well
This ensures the Dockerfile used for testing is making use of the
same scratch image to compile `libgit2` as the actual application
image.

In a future iteration we should restructure our GitHub Action
workflows to re-use the application image, saving us an additional
Dockerfile and a duplicate build. Inspiration for this (which makes
use of a local registry for the duration of the build) can be found
at: https://github.com/fluxcd/golang-with-libgit2/blob/main/.github/workflows/build.yaml

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals d04c532461 Switch to scratch based libgit2 container image
This moves the `libgit2` compilation to the image, to ensure it
can be build on builders that aren't backed by AMD64.

The image is structured in such a way that e.g. running nightly
builds targeting a different Go version, or targeting a different
OS vendor would be possible in the future via build arguments.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals 153b122970 Document libgit2 build behavior in CONTRIBUTING.md
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals cc01df231e Detect macOS produced libgit2.dylib on Darwin
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals 1b11e11a90 Allow libgit2 build to be enforced
This can be useful on machines where libgit2 is installed due to other
applications depending on it, but where the composition of this
installation does not properly work with the controller.

Reason the system version is still preferred, is because this lowers the
barrier for drive-by contributors, as a working set of (Git) dependencies
should only really be required if you are going to perform work in that
domain.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals 500d0aeda0 Update base image to version with Darwin detection
To provide a better (contributing) experience to those with Apple
machines, as determining the correct paths there is a bit harder.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals b283e3e9c5 Change image to image under Flux organization
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals c9e3f97470 Add `docker-buildx` target to `Makefile`
To allow building a multi-platform container image using `buildx`.

Various configuration flags allow for fine(r)-grain control over the
build process:

- `BASE_IMG`: FQDN of the base image that should be used, without a
  tag.
- `BASE_TAG: tag of the base image that should be used. Allows checksum
  sum to be included.
- `BUILDX_PLATFORMS`: platforms to target for the final container
  image.
- `BUILDX_ARGS`: additional `docker buildx build` arguments, e.g.
  `--push` to push the result to a (local) image registry.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
Hidde Beydals 869c7960e3 Update github.com/libgit2/git2go to v31.6.1
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.

This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.co/hiddeco/golang-with-libgit2#usage) for more
detailed information).

The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.

Some small changes have been made to the `libgit2` package to address
(future) deprecations.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
pa250194 f797fbfdf0 Added Logger to closing GCP client
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 09:52:11 -05:00
pa250194 f62571bcec Added log for GCP provider auth error
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 09:41:19 -05:00
pa250194 0c0a76d7fa Merge branch 'gcp-bucket-provider' of https://github.com/pa250194/source-controller into gcp-bucket-provider 2021-10-14 09:39:14 -05:00
pa250194 116906cca4 Fixed spelling and capitalization
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:53 +01:00
pa250194 751243ce50 Refactor comments and method names
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 02102de2c7 Removed resumable downloads
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 57ef719f74 Updated docs to include GCP provider instructions
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 b02a7625ea Added more tests and cleaned up GCP provider logic
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 a46b0f54b8 Added Tests to GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 1fae4f61c1 Tests for GCP Bucket Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 57b54c8592 Service Account Key Authentication to GCP Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 2cc48fefb1 Added initial testing for new GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 a600528729 Added Comments for reconcileWithGCP and reconcileWithMinio
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00
pa250194 572eed74e5 Add Support for GCP storage with workload identity
Added Support for Google Cloud Storage with Workload Identity as
Source Provider. This enables the use of GCP without enabling S3
compatible access.

Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 10:16:11 +01:00