Commit Graph

208 Commits

Author SHA1 Message Date
Hidde Beydals e736493730 libgit2: correctly resolve (annotated) tags
In d0560e5 the SemVer implementations were aligned, and the logic was
simplified a bit (or so I thought). This did however result in the
introduction of a regression, as it failed to take "simple tags" into
account.

This commit ensures both are taken into account again, and ensures it
is now covered by a proper test.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-21 17:10:43 +02:00
pa250194 c4e4b3928c Added Logger to closing GCP client
Signed-off-by: pa250194 <pa250194@ncr.com>

Added log for GCP provider auth error

Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:57:48 -05:00
pa250194 c98130548e Added Tests to GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:21:15 -05:00
pa250194 99c79bffaa Tests for GCP Bucket Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:19:19 -05:00
pa250194 be1ed50ac4 Service Account Key Authentication to GCP Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:16:53 -05:00
pa250194 38bf4d9859 Fixed spelling and capitalization
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 13:51:02 -05:00
Hidde Beydals 869c7960e3 Update github.com/libgit2/git2go to v31.6.1
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.

This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.co/hiddeco/golang-with-libgit2#usage) for more
detailed information).

The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.

Some small changes have been made to the `libgit2` package to address
(future) deprecations.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:48:15 -05:00
pa250194 7c0d4c070e Refactor comments and method names
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 13:43:08 -05:00
pa250194 38be5ed111 Cleanup obsolete comments
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 12:40:55 -05:00
pa250194 057c65e939 Removed resumable downloads
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 12:39:00 -05:00
pa250194 6ff5970fe1 Added more tests and cleaned up GCP provider logic
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 c204f6a8ee Added Tests to GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 eeb38bdf2c Tests for GCP Bucket Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 0444c6e16d Service Account Key Authentication to GCP Provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 90395f426a Remove .DS_STORE file
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 78379ddcd5 Added initial testing for new GCP provider
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194 7da9619b58 Feature: Add Support for Google Cloud Storage along with Workload Identity
Added Support for Google Cloud Storage with Workload Identity as Source Provider. This enables the use of GCP without enabling S3 compatible access.

Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
Hidde Beydals d0560e5dbe Use same SemVer logic in both Git implementations
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-08-05 12:40:54 +02:00
Hidde Beydals c4d7e46b90 Drop deprecated `io/ioutil`
The package has been deprecated since Go 1.16, see:
https://golang.org/doc/go1.16#ioutil

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-07-29 09:58:00 +02:00
Blake Burkhart 0df2b0e0f0 Fix tag checkout with libgit2
SetHeadDetached (git_repository_set_head_detached) only changes HEAD,
and does not actually checkout the files on disk. Use CheckoutHead with
the CheckoutForce Strategy to actually check the files out on disk.

Additionally add a test that validates the hash of a checked out file's
contents.

Previously, the hash of the desired tag was being reported as the
checked out revision by the GitRepository. However the wrong files were
checked out and an incorrect revision would be deployed by Flux.

Signed-off-by: Blake Burkhart <blake.burkhart@us.af.mil>
2021-06-28 20:03:11 -05:00
Stefan Prodan 79b180376d
Update go-git to v5.4.1
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-05-26 14:06:41 +03:00
Somtochi Onyekwere ee43942afb Test for Method fails with some private keys
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2021-04-19 15:58:18 +01:00
Hidde Beydals b5004a93bc Make Storage#Archive file filtering configurable
This commit makes the filtering applied during the archiving
configurable by introducing an optional `ArchiveFileFilter`
callback argument and a `SourceIgnoreFilter` implementation.

`SourceIgnoreFilter` filters out files matching
sourceignore.VCSPatterns and any of the provided patterns.
If an empty gitignore.Pattern slice is given, the matcher is set to
sourceignore.NewDefaultMatcher.

The `GitRepository` now loads the ignore patterns before archiving
the repository contents by calling `sourceignore.LoadIgnorePatterns`
and other helpers. The loading behavior is **breaking** as
`.sourceignore` files in the (subdirectories of the) repository are
now still taken into account if `spec.ignore` for a resource is
defined, overwriting is still possible by creating an overwriting
rule in the `spec.ignore` of the resource.

This change also makes it possible for the `BucketReconciler` to not
configure a callback at all and prevent looking for ignore
matches twice. To finalize the bucket refactor, a change to the
reconciler has been made to look for a `.sourceignore` file in
the root of the bucket to provide an additional way of configuring
(global) exclusions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-04-13 15:34:20 +02:00
Hidde Beydals cca2c4a362 Check ignore matches before Bucket item downloads
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-04-13 15:34:20 +02:00
Somtochi Onyekwere d3d1917e5e Add tests for libgit2
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2021-04-12 16:31:42 +01:00
Somtochi Onyekwere e82c8e8b70 Use password in git method for secret
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2021-04-12 15:52:51 +01:00
Stefan Prodan 664a568822
Add support for Git submodules with go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-30 13:00:13 +03:00
Stefan Prodan 97d0326432
Enable git submodules cloning when using go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-29 15:25:33 +03:00
Stefan Prodan f0016cfad1
Enable self-signed certs for go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-29 13:23:32 +03:00
Michael Bridgen f7e08c4738 Tidy git clone errors
In some circumstances (that are rather hard to reproduce), cloning
from a GitLab repo gets a multiline response as described in
https://github.com/fluxcd/image-automation-controller/pull/115.

This uses the same remedy as in that PR, by calling the funcs provided
by fluxcd/pkg/gitutil on any error returned by libgit2 or gogit clone
operations.

Signed-off-by: Michael Bridgen <mikeb@squaremobius.net>
2021-03-03 13:41:53 +00:00
Philip Laine 0a1631dc5a Remove redundant if else
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-12 11:18:29 +01:00
Philip Laine 73301df023 Add md5 test and check priority of hash types
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-12 11:16:16 +01:00
Philip Laine f9ddeb06e1 Fix hash type switch statement
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-12 08:43:43 +01:00
Philip Laine 6b3d96ef3b Check hostkey type when validating hostkey
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-11 23:04:46 +01:00
Hidde Beydals b2f6639c01 libgit2: use provided host to validate public key
The callback from libgit2 only provides a hostname (without the port),
but the `known_hosts` file indexes the public keys based on the full
host (e.g. `[localhost]:123` for a host behind a specific port).

As a result, it was unable to find the correct public key for the
hostname when it was added to the `known_hosts` file with the port.

To work around this, we add the user provided host that includes the
port to the `PublicKeyAuth` strategy, and use this to find the right
entry in the `known_hosts` file, after having validated that the
hostname provided to the callback matches the hostname of the host
provided by the user.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-11 12:52:55 +01:00
Hidde Beydals fac1afa2a8 Move `git/common` to `git`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-08 14:05:05 +01:00
Hidde Beydals 7e63ef841c Rename Git packages to implementations
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-08 13:55:23 +01:00
Philip Laine c063484761 Add custom certificate validation
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-08 12:19:22 +01:00
Hidde Beydals 8f2a920765 Properly forward commit hash for go-git impl
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-11 16:05:02 +01:00
Philip Laine f044ebc253 Rename git2go option to libgit2
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-06 23:19:57 +01:00
Philip Laine 1b4eacc588 Refactor argument name to enable git2go
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-04 10:07:17 +01:00
Philip Laine 6344528e4f Fix smaller issues
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2020-12-02 20:18:29 +01:00
Philip Laine 5ff65d7ae7 Add user support to git2go implementation
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2020-12-02 20:18:29 +01:00
Philip Laine 2cb3754953 Add support for gitv2 servers
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-02 20:18:29 +01:00
Hidde Beydals 67b10aad22 Respect configured user in SSH Git repository URL
We had a hardcoded assumption that the SSH user for a Git repository is
always "git". This is however not true in all scenarios, for example
when one is making use of Gerrit for team code collaboration, as users
there have their own username for (SSH) Git operations.

This commit changes the logic of the auth strategy helpers to:

1. Select the auth strategy based on the protocol of the parsed URL,
   instead of a simple rely on a correct prefix.
2. Use the user information from the parsed URL to configure the user
   for the public key authentication strategy, with a fallback to `git`
   if none is defined.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-30 16:26:23 +01:00
Hidde Beydals c9eb8f03c0 Switch to Masterminds/semver and pkg/version libs
Co-authored-by: Illia Ovchynnikov <illia.ovchynnikov@gmail.com>
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-28 12:21:08 +01:00
Illia Ovchynnikov 394b5c3bd0
Pick the most recent chart/tag for ambiguous semver matches
Signed-off-by: Illia Ovchynnikov <illia.ovchynnikov@gmail.com>
2020-10-27 20:26:28 +01:00
Hidde Beydals 582c2092b6 Change copyright to Flux authors
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-27 17:54:22 +01:00
Stefan Prodan 13874ce552
Add Git URL to clone error message
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-10-16 15:58:09 +03:00
stefanprodan f16e5f041a Promote API to v1beta1 2020-09-30 16:11:00 +03:00
Hidde Beydals f35dcc3656 Upgrade blang/semver from v3.5.0 to v4.0.0
v4.0.0 is fully compatible with Go Modules.
2020-08-31 10:03:47 +02:00
Hidde Beydals 668e0223dd Drop single branch strategy for SemVer checkout
As this will result in a checkout failure when the default branch on the
remote is not `master`. Surfaced due to Contour switching from `master` to
`main` overnight.
2020-08-26 11:39:06 +02:00
Hidde Beydals 9d947b8efa Use testserver package from fluxcd/pkg 2020-08-18 12:11:29 +02:00
stefanprodan 5044d46b81 Fix Git semver checkout 2020-08-18 11:32:42 +03:00
stefanprodan e65e2f0623 Make git package public 2020-07-20 13:10:18 +03:00
stefanprodan cfc3d35cbf Fix artifact test server
path.Join strips // from URLs, using fmt.Sprintf to fix URLForFile
2020-07-10 10:55:52 +03:00
Hidde Beydals 7f118d6222 testserver/artifact: rename to ArtifactFromFiles 2020-06-30 12:18:09 +02:00
Hidde Beydals 0bc28f7a42 testserver: move from internal to pkg 2020-06-30 11:48:14 +02:00