Commit Graph

208 Commits

Author SHA1 Message Date
Sanskar Jaiswal 972d1cac2a fix docs, error handling and managed proxy auth
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal ec45a612b1 enable managed transport for controller tests
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal 5152721ae0 factor out unmanaged checkout into its own functions
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal 94c50fa3a8 remvoe support for sha1 and md5 hashing for public keys
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal 90ec1e230c expand proxy tests to cover managed transport
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Co-authored-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal 7d2bc64f47 fix panics on unmanaged http and proxy on managed http
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal d4beacb6ad Remove dependency on libgit2 credentials callback
Injects transport and auth options at the transport level directly to
bypass the inbuilt credentials callback because of it's several
shortcomings. Moves some of the pre-existing logic from the reconciler
to the checkout implementation.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Paulo Gomes a0d0a6312d
libgit2: enable managed transport by default
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-24 10:46:32 +01:00
Sunny 581695b4d6
gitrepo: Intro contentConfigChecksum & improvement
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.

Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.

Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.

The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Sunny 749068e9c3
pkg/git: introduce concrete and partial commit
Introduce concrete and partial commits. Concrete commits have all the
information from remote including the hash and commit content. Partial
commits are based on locally available copy of a repo, they may only
contain the commit hash and reference.

IsConcreteCommit() can be used to find out if a given commit is based on
local information or full remote repo information.

Update go-git and libgit2 branch/tag clone optimization to return a
partial commit and no error.

Update and simplify the go-git and libgit2 tests for the same.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Paulo Gomes 8b50367849
libgit2: Add support for hashed known_hosts
Hashed known_hosts was previously only supported when using
go-git. Now both Git implementations benefit from this
features, and the code coverage across them can ensure no
future regression.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-16 16:57:22 +01:00
Paulo Gomes 6a407704a3
libgit2: remove redundant test
Both libgit2 and go-git now have tests that confirms
support for the supported authentication algorithms.
This test was created previously and is no longer
needed.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-13 15:00:10 +01:00
Paulo Gomes ce92881013
libgit2: remove connection caching
Connection caching was a feature created to resolve
upstream issues raised from concurrent ssh connections.
Some scenarios were based on multiple key exchange
operations happening at the same time.

This PR removes the connection caching, and instead:
- Services Session.StdoutPipe() as soon as possible,
  as it is a known source of blocking SSH connections.
- Reuse SSH connection within the same subtransport,
  eliminating the need for new handshakes when talking
  with the same server.
- Simplifies the entire transport logic for better
  maintainability.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-13 11:09:02 +01:00
Paulo Gomes 7f315f9807
gogit: use rem.ListContext instead of rem.List
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 16:36:55 +01:00
Paulo Gomes 90ef278797
libgit2: return func to help callers free git2go objects
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 13:02:27 +01:00
Paulo Gomes 44e166e576
libgit2: dispose git2go objects when error != nil
getBlankRepoAndRemote's callers are responsible for the disposal
of the returned objects. However, the caller does not expect to
need to dispose objects when err != nil, which may result to memory
leaks.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 11:40:09 +01:00
Paulo Gomes 54e07d8783
libgit2: recover from panic in short-circuited clones
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 11:40:08 +01:00
Somtochi Onyekwere 2bb3a1fea9
Implements no-op before CheckoutTag in gogit
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-05-11 11:40:07 +01:00
Somtochiama 8a3df9da42
gogit: check if revision changed before cloning in checkout branch (#694)
* Check if revision has changed in gogit CheckoutBranch

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-05-11 11:40:06 +01:00
Sanskar Jaiswal 6a793a55f6
Optimize libgit2 checkout tag strategy through condition no-ops
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-11 11:40:05 +01:00
Paulo Gomes 860d7051da
libgit2: optimise checkout branch clones
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.

This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 11:40:04 +01:00
Paulo Gomes c666f50424
Remove minio region
Twice recently the minio region seemed to have changed
unsetting it seems to work consistently.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-10 14:28:54 +01:00
Paulo Gomes 2b59150fbe
tests: algorithms test coverage for go-git and libgit2
Assures support for:
- Authentication Key Types
  - rsa
  - ecdsa P256
  - ecdsa P384
  - ecdsa P521
  - ed25519
- Key Exchange Algoritms:
  - diffie-hellman-group14-sha1
  - diffie-hellman-group14-sha256
  - curve25519-sha256
  - ecdh-sha2-nistp256
  - ecdh-sha2-nistp384
  - ecdh-sha2-nistp521
  - curve25519-sha256@libssh.org
- HostKey Algoritms:
  - ssh-rsa
  - rsa-sha2-256
  - rsa-sha2-512
  - ecdsa-sha2-nistp256
  - ecdsa-sha2-nistp384
  - ecdsa-sha2-nistp521
  - ssh-ed25519

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-09 13:31:54 +01:00
Paulo Gomes 4e3e62923b
git: Add git.HostKeyAlgos
Enables the setting of HostKey algorithms to be used from
a client perspective. This implementation supports go-git
and libgit2 when in ManagedTransport.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-06 17:58:09 +01:00
Paulo Gomes d86ea25e87
Fix minio test by changing region to us-west-2
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-06 08:59:56 +01:00
Paulo Gomes c0c5046e5f
libgit2: recover from git2go panic
Some specific scenarios may lead libgit2 or git2go to panic.
Adding a recovery logic, ensures a predictable execution path
for callers, and safeguards the controller's stability.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-06 08:39:00 +01:00
Eng Zer Jun 10b92da433
test: use `T.TempDir` to create temporary test directory
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-29 23:01:39 +08:00
Hidde Beydals 18b59351c1 Update dependencies
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.

- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 16:41:42 +02:00
Peter Gundel 37551f1964
Reduce file permission
As suggested by @pjbgf

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-13 17:14:45 +02:00
Peter Gundel 0949d0adb9
Change all file permissions to octal format
This better represent permissions as Linux handles such information in
octal format, meaning that the left-most 0 has an important meaning
and is not to be ignored as normally integers would.

See https://github.com/fluxcd/source-controller/issues/603

Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-12 21:24:34 +02:00
Paulo Gomes 68eece48fb
libgit2: optimise mutex on cached connections
Previously the mutex.Lock was acquired before creating
a new connection. The lock would then hold until the
process was finished, and all network latency would be
absorbed by other goroutines trying to establish a new
connection.

Now the lock is acquired after the connection has been
created. The downside of this approach is that concurrent
goroutine may be trying to open a connection to the same
target. The loser in the race will then have to Close the
connection and use the winner's instead.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 19:10:54 +01:00
Paulo Gomes b264a3513d
libgit2: refactor max length values into constants
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 19:10:50 +01:00
Paulo Gomes 3ab95a4bf0
libgit2: close discarded connections
Cached connections can be shared across concurrent
operations, and their disposal must take that into
account to avoid closing a connection that is stale for
one goroutine, but is still valid for another.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 18:37:40 +01:00
Paulo Gomes add07745f3
libgit2: restrict smart creds to Type SSH Memory
Avoid asking for SSH credential in files, as they won't be
used. The cacheKeyAndConfig func already enforces this
behaviour.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 18:37:39 +01:00
Paulo Gomes d86f0a280a
libgit2: validate URL max length
The major Git SaaS providers have repository URLs
for both HTTP and SSH that tops around 250
characters in length.

The limits chosen were a lot higher to align with use
cases in which users may have on-premise servers with
long domain names and paths.

For SSH the validation is around path length only,
which is now limited to 4096 characters, which is
at the higher end of the range in Linux.

For HTTP the validation is around the full URL
provided by the caller.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 18:37:38 +01:00
Paulo Gomes 54d0794d19
libgit2: handle the closing of stale connections
Internal and upstream calls to sshSmartSubtransport.Close()
when dealing with an stale connection, may lead to misleading
errors.

Focus should instead be redirected to ensuring that Close()
releases resources and ensures that a new SubTransport can be
created, so new operations can succeed.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 18:37:38 +01:00
Paulo Gomes 69c3f00172
libgit2: retry on stale connections
SSH servers that block the reuse of SSH connections for
multiple SSH sessions may lead to EOF when a new session
is being created.

This fixes the issue of long-running connections resulting
in EOF for GitLab servers.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-07 18:37:37 +01:00
Sanskar Jaiswal 5c84ea7e96 add flag to allow configuration of SSH kex algos
Adds a flag `ssh-kex-algos` which configures the gogit and libgit2
managed clients to use the specified list of kex algos for ssh. If not
used the default list in `golang/x/crypto/ssh` is used.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-04-07 16:43:15 +05:30
Paulo Gomes 36fcdeeb5e
libgit2: fix access to nil t.stdin and improve observability
All errors that were previously not handled are now logged through
traceLog, to further help during transport investigations.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-30 14:44:41 +01:00
Paulo Gomes c84179088c Fixes regression accessing GitLab public repositories
Some git servers are more accommodating than others. Gitlab will try to
validate credentials when they are provided, even if they are empty and
the target repository is public, leading to a failed authentication error.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-30 10:50:40 +02:00
Paulo Gomes 92ad1f813b
Cache SSH connections
The underlying SSH connections are kept open and are reused
across several SSH sessions. This is due to upstream issues in
which concurrent/parallel SSH connections may lead to instability.

https://github.com/golang/go/issues/51926
https://github.com/golang/go/issues/27140
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-28 11:58:10 +01:00
Paulo Gomes 017707a71c
Improve managed transport observability
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 19:08:54 +00:00
Paulo Gomes 5091b69ad5
Force ssh.Dial timeout
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 19:08:52 +00:00
Paulo Gomes a860ebee04
Reuse HTTP connections in managed transport
Ensure all requests are completely processed and closed,
to prove odds of the underlying connections to be reused.

The transport now is pooled and reused whenever possible.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 19:08:51 +00:00
Paulo Gomes 45641b17c7 Fix dpanic issue when logging odd number of args
Info was receiving odd number of arguments leading to:
odd number of arguments passed as key-value pairs for logging

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 14:27:04 +01:00
Paulo Gomes 115040e9ea
Support redirects for libgit2 managed transport
For backwards compatibility, support for HTTP redirection is enabled when targeting
the same host, and no TLS downgrade took place.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes 43661dd15e
Enforce effective URL on error messages
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes d1a7e5d609
Fix race condition on httpSmartSubTransport
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes 822788b79e
Fix word casing
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes 4ed54bc359
Optimise basic auth for libgit2 managed transport
The initial implementation was based off upstream, which cause
an initial request to fail, and only then the credentials would
be added into the request.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes 24bc95e1d8
Add license headers
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes a000d8b859
Add tests for experimental libgit2 transport
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes aa3288112e
Implement Managed Transport for libgit2
libgit2 network operations are blocking and do not provide timeout nor context capabilities,
leading for several reports by users of the controllers hanging indefinitely.

By using managed transport, golang primitives such as http.Transport and net.Dial can be used
to ensure timeouts are enforced.

Co-Authored-by: Sunny <darkowlzz@protonmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:20 +05:30
Hidde Beydals d17e724124 build: properly wire integration tests
- Ensure secrets are set as env variables
- Wire `GO_TAGS` together with `GO_STATIC_FLAGS`

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:43:35 +01:00
Hidde Beydals ccb65c74f0 Add ManagedIdentity with `AZURE_CLIENT_ID`
This ensures the Managed Identity authentication works with multiple
identities assigned to a single node.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-08 14:57:56 +01:00
Hidde Beydals d7ce67979f Reorder Azure token credential composition
- `authorityHost` and `clientCertificateSendChain` can now be set where
  applicable.
- AZ CLI fields have been removed.
- Fallback to `ChainedTokenCredential` with `EnvironmentCredential` and
  `ManagedIdentityCredential` with defaults if no Secret is given.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-08 14:57:38 +01:00
Hidde Beydals c5c9160ec5 Support fields from `az` generated Azure SP
This supports the fields as documented in the AKS documentation:
https://docs.microsoft.com/en-us/azure/aks/kubernetes-service-principal?tabs=azure-cli#manually-create-a-service-principal

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-08 14:48:51 +01:00
Hidde Beydals cc805b4c55 Change Azure authentication order
Based on recommendations from Microsoft, change the order valid
authentication options are taken into account. Mainly to ensure it works
as expected when multiple Managed Identities are bound on the same VM
node.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-08 14:48:51 +01:00
Hidde Beydals 37e602a2e3 Address review nitpicks
- Use octal syntax for permissions.
- Fix typo.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-08 14:48:51 +01:00
Hidde Beydals bd12cdba17 Allow Azure ClientCertificate authentication
This commit allows for a Secret to be configured with `tenantId`,
`clientId` and `clientCertificate` data fields (with optionally
`clientCertificatePassword`) to authenticate using TLS.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-03 13:20:48 +01:00
Hidde Beydals ec5bc1ac9a Implement Azure Blob BucketProvider
This commit introduces an Azure Blob BucketProvider implementation,
capable of fetching from objects from public and private "container"
buckets.

The supported credential types are:

- ManagedIdentity with a `resourceId` Secret data field.
- ManagedIdentity with a `clientId` Secret data field.
- ClientSecret with `tenantId`, `clientId` and `clientSecret` Secret
  data fields.
- SharedKey with `accountKey` Secret data field, the Account Name is
  extracted from the endpoint URL specified on the object.

If no Secret is provided, the Bucket is assumed to be public.

Co-authored-by: Zhongcheng Lao <Zhongcheng.Lao@microsoft.com>
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-03 13:20:48 +01:00
pa250194 ed6c6ebc3c Introduce BucketProvider interface
This commit introduces a BucketProvider interface for fetch operations
against object storage provider buckets. Allowing for easier
introduction of new provider implementations.

The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.

The client implementations do now include safe-guards to ensure the
fetched object is the same as metadata has been collected for. In
addition, minor changes have been made to the object fetch operation
to take into account that:

- Etags can change between composition of index and actual fetch, in
  which case the etag is now updated.
- Objects can disappear between composition of index and actual fetch,
  in which case the item is removed from the index.

Lastly, the requirement for authentication has been removed (and not
referring to a Secret at all is thus allowed), to provide support
for e.g. public buckets.

Co-authored-by: Hidde Beydals <hello@hidde.co>
Co-authored by: Michael Bridgen <michael@weave.works>
Signed-off-by: pa250194 <pa250194@ncr.com>
2022-03-01 10:15:10 +01:00
Sunny f472cadab4 Add bucket controller tests for reconcileGCPSource
- Introduce mock GCP Server to test the gcp bucket client against mocked
  gcp server results.
- Add tests for reconcileGCPSource().
- Patch GCPClient.BucketExists() to return no error when the bucket
  doesn't exists. This keeps the GCP client compatible with the minio
  client.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny b814070bc2 Fixes to gitrepo reconciler tests
NOTE: This should be amended with the previous commit which has
commented out tests.

Update reconcileSource() to work with the test case where no secret is
set. A minimal auth options is created and used for git checkout.

Update TestGitRepositoryReconciler_verifyCommitSignature() to use the
new git.Commit type.

Update TestGitRepositoryReconciler_reconcileSource_checkoutStrategy to
add skipForImplementation for branch commit test case.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals eff40e22e9 git/libgit2: assert proper test of default branch
If there is no configuration set for `init.defaultBranch`, it does not
return an error but an empty string. We now take this into account so
we do not overwrite the default, and make the default `master` to match
with libgit2 defaults.

In addition, some comments have been added to not get confused about
what commits we are checking against.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-22 16:31:56 +01:00
Hidde Beydals 15c064abdf git/libgit2: set CheckoutForce on branch strategy
In the recent update from libgit2 1.1.x to 1.3.x, something seems to
have changed upstream. Resulting in the clone of a branch ending up
with a semi-bare file system state (in other words: without any files
present in the directory).

This commit patches the clone behavior to set the `CheckoutForce`
strategy as `CheckoutOption`, which mitigates the issue.

In addition, test cases have been added to ensure we do not run into
this again by asserting the state of the branch after cloning.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-22 16:31:56 +01:00
Paulo Gomes 8429708997
Upgrade libgit2 to libgit2-1.3.0-2
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 11:39:11 +00:00
Paulo Gomes e5d032fe9c
Add libgit2 checkout test with ED25519 key
This adds a test to detect any regression in libgit2's ED25519 key
support. go-git supports ED25519 but not the current version of
libgit2 used in flux. The updates to libgit2 in v1.2.0 adds support
for ED25519. This test would help ensure the right version of libgit2
is used.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:17:32 +00:00
Paulo Gomes f44302eed0
Bump libgit2 to 1.3.0 and git2go to V33.
Downstream breaking changes introduced since git2go@V31:
- git2go.ErrorCode was deprecated in favour of the native error type.
- FetchOptions no longer expects a pointer, but rather the actual value of git2go.FetchOptions.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:17:31 +00:00
Sunny 649d33ca37
pkg/git: Include commit message and URL in error
go-git: Include the commit message in the returned commit object.
libgit2: Set the URL in the checkout error.

Add new method Commit.ShortMessage() for returning short commit
message.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-14 21:34:27 +05:30
Paulo Gomes 9479d04779
Fix host mismatch in libgit2
Depending on libgit2 version or from its dependencies, the hostname may or may not contain ports

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-01-26 17:25:18 +00:00
Robert Clarke 3d276b679b libgit2: Configured libgit2 clone ProxyOptions
This configures ProxyOptions for all libgit2 Checkout functions when
cloning and configures the options based on current environment
settings using the git2go.ProxyTypeAuto option.

Refs: #131
Signed-off-by: Robert Clarke <rob@robertandrewclarke.com>
Co-authored-by: Aurélien GARNIER <aurelien.garnier@atos.net>
2022-01-18 19:35:47 +00:00
Kevin McDermott 4029241b3b Don't assume master is the default branch.
This introduces a quick lookup of the default branch rather than
assuming "master".

Signed-off-by: Kevin McDermott <kevin@weave.works>
2022-01-17 14:25:05 +00:00
Hidde Beydals ab089ddb07 libgit2: use `bytes.Equal` instead of `.Compare`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals 9c1c8149a5 sourceignore: return err of nested load
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals cd7da9df8c Drop deprecated `io/ioutil`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 16:31:25 +01:00
Hidde Beydals cc71517ed3 tests: ensure proper garbage collection
- Ensure the proper path is garbage collected for libgit2 repositories,
  as the `Path` method on the repository object returns the `.git`
  directory, and not the root path.
- Ensure the Helm test server does not get swapped during tests,
  with as side-effect that no obsolete temporary directories remain.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-16 10:39:58 +01:00
Sunny 68a3ea2e4d Add tests for libgit2 remote callbacks
- Adds tests for the libgit2 remote callbacks
- Adds tests for CheckoutStrategyForImplementation with context timeout
  and verify timeout is respected by both the git implementations.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-04 16:17:23 +05:30
Sunny 65e5c6a686 libgit2: Add more RemoteCallbacks
Add SidebandProgressCallback to be able to cancel the network operation
before any transfer operation.
Add PushTransferProgressCallback to be able to cancel the push transfer
operation.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-04 16:17:23 +05:30
Sunny d407c824d6 libgit2: Add early return to transferProgressCallback
In transferProgressCallback(), if the received objects is equal to the
total objects, return early with OK.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-04 16:17:23 +05:30
Hidde Beydals afa82bbc7a libgit2: ensure context timeout cancels transfer
With the information from the refactor still fresh in mind, I continue
to find new paths now I mentally tamed the git2go beast.

`libgit2` seems to assume that a transport will eventually tell by
itself that it has timed out. This also means that at present any
timeout configuration does not seem have an effect. It will continue
to transfer until the remote (or _something_ else) tells it is no
longer transfering.

This commit introduces a simple check (without tests) which was used
to confirm the theory in combination with the tests in
`pkg/git/strategy` (by setting it to a very low timeout and observing
it fail).

A future iteration should probably take the data given to the callback
into account to ensure it doesn't error out if the given data[1]
reports it has successfully received all objects. Another candidate
for this check may be `CompletionCallback`, but one should study the
C code (and likely some Go code as well) before this.

In addition, to ensure the same timeout is taken into account for push
operations, `PushTransferProgressCallback` may require a likewise
helper.

[1]: https://github.com/libgit2/git2go/blob/main/remote.go#L50-L58

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-04 16:17:23 +05:30
Sunny 10d262032c libgit2: handle EOF in parseKnownHosts()
parseKnownHosts() uses golang.org/x/crypto/ssh's ParseKnownHosts() for
parsing known hosts. It returns EOF error when the input is not a host
public key, but a valid known_hosts content, like a comment line.

With this fix, lines causing EOF error are skipped and the parsing of
the known_hosts file continues. But invalid lines still cause parsing
failure.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-28 17:35:46 +05:30
Sunny a7f2e870bf transportAuth(): Add checks for invalid transports
Update GitRepositoryReconciler to use a nil authOpts unless it's
configured.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 16:59:26 +05:30
Sunny f9a34045e1 Update gittestserver
New gittestserver fixes the issue with custom branch in an
initialized repo.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 00:43:31 +05:30
Sunny 5bd08a6960 Add Host field check in AuthOptions.Validate()
For ssh, Host field is required in AuthOptions.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 00:43:31 +05:30
Sunny 562af6d658 Add git.CheckoutStrategy SemVer checkout tests
Adds tests for git.CheckoutStrategy to check if both the git
implementations follow the same SemVer tag selection rules.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 00:43:31 +05:30
Hidde Beydals 99428f593e libgit2: change credentialsCallback logic
This changes the logic of `credentialsCallback` so that it takes the
`allowedTypes` passed on by `git2go` into account. Reason for this
change is because this prepares it to work with `v33`, but also
because it can provide better guidance when `libgit2` has been
compiled with a different configuration, which e.g. doesn't
allow for "in-memory SSH keys".

Because `AuthOptions#Identity` now gets validated by the callback
and go-git does its own validaiton, the check has been removed
from `Validate` (and now does a simple check if the fields are set).

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals 7a5d8b116d git: styling nitpicks
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals c814e0fa26 git: remove ', error:' from returned error
As we properly nest errors.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Sunny 8c581ddfbc Add git.CheckoutStrategy auth tests
Adds tests for git.CheckoutStrategy to check if both the git
implementations work with all the authentication methods.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 00:43:31 +05:30
Sunny 942c310195 pkg/git: AuthOptions.Validate() test improvements
Adds more test cases for Validate() and an error for unknown transport.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-10-27 00:43:31 +05:30
Hidde Beydals 4a23126c6a libgit2: make RemoteCallbacks helper public
Main requirement for this is the image-automation-controller
depending on being able to get a working auth configuration.

Once the package is moved, we should add push logic to it,
so that the controller is able to use that instead.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals b7376ce94c gogit: allow checkout of commit without branch
This commit changes the `gogit` behavior for commit checkouts,
now allowing one to reference to just a commit while omitting any
branch reference. Doing this creates an Artifact with a
`HEAD/<commit>` revision.

If both a `branch` and `commit` are defined, the commit is expected
to exist within the branch. This results in a more efficient clone
of just the target branch, and also makes this change backwards
compatible.

Fixes #407
Fixes #315

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals 5a1fcc213b git: standardise commit and (PGP) verification
This commit refactors the previous `Commit` interface into a
standardised `Commit` struct. This object contains sufficient
information for referencing, observating and (PGP) verification.

- `libgit2` commit checkout does now return `HEAD/<SHA1>` as
  the branch is not taken into account.
- `git2go` objects are now properly `Free`d everywhere
- `Verify` logic is tested.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals 80b9807550 gogit: add CheckoutStrategy tests
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals 0cf0d4e756 git: refactor AuthStrategy into AuthOptions
This commit moves the previous `AuthStrategy` wiring to a more generic
`AuthOptions`, breaking free from implementation specific details in
the `git` package.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Hidde Beydals 9ff5334688 go-git: ensure original tag is used for TS lookup
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 13:25:24 +02:00
Hidde Beydals 4c94f88ebb libgit2: ensure original tag is used for TS lookup
This adds an additional selection candidate, to ensure test flakiness
happens more often.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 13:19:34 +02:00
Hidde Beydals 56201f30fe libgit2: Free most objects
This commit ensures most of the `git2go` objects `Free` themselves from
the underlying C object.

Ensuring all objects are freed is not possible yet, due to the way
commits are wired in to facilitate verification later on. In a later
follow up, we should change this and e.g. validate as part of the
checkout process, and move the implementation specific authentication
configuration from `git` into `libgit2`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 12:01:45 +02:00
Hidde Beydals d9473d008c libgit2: add remaining checkout strategy tests
This commit is a follow up on 4dc3185c5f
and adds tests for the remaining checkout strategies, while
consolidating some of the logic.

The consolidated logic ensures that (SemVer) tag and commit checkouts
happen using the same "checkout detached HEAD" logic.
The branch checkout is left unmodified, and simply checks out at the
current HEAD of the given branch.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-22 12:01:41 +02:00