Injects transport and auth options at the transport level directly to
bypass the inbuilt credentials callback because of it's several
shortcomings. Moves some of the pre-existing logic from the reconciler
to the checkout implementation.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.
Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.
Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.
The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce concrete and partial commits. Concrete commits have all the
information from remote including the hash and commit content. Partial
commits are based on locally available copy of a repo, they may only
contain the commit hash and reference.
IsConcreteCommit() can be used to find out if a given commit is based on
local information or full remote repo information.
Update go-git and libgit2 branch/tag clone optimization to return a
partial commit and no error.
Update and simplify the go-git and libgit2 tests for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Hashed known_hosts was previously only supported when using
go-git. Now both Git implementations benefit from this
features, and the code coverage across them can ensure no
future regression.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Both libgit2 and go-git now have tests that confirms
support for the supported authentication algorithms.
This test was created previously and is no longer
needed.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Connection caching was a feature created to resolve
upstream issues raised from concurrent ssh connections.
Some scenarios were based on multiple key exchange
operations happening at the same time.
This PR removes the connection caching, and instead:
- Services Session.StdoutPipe() as soon as possible,
as it is a known source of blocking SSH connections.
- Reuse SSH connection within the same subtransport,
eliminating the need for new handshakes when talking
with the same server.
- Simplifies the entire transport logic for better
maintainability.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
getBlankRepoAndRemote's callers are responsible for the disposal
of the returned objects. However, the caller does not expect to
need to dispose objects when err != nil, which may result to memory
leaks.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.
This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Enables the setting of HostKey algorithms to be used from
a client perspective. This implementation supports go-git
and libgit2 when in ManagedTransport.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Some specific scenarios may lead libgit2 or git2go to panic.
Adding a recovery logic, ensures a predictable execution path
for callers, and safeguards the controller's stability.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Previously the mutex.Lock was acquired before creating
a new connection. The lock would then hold until the
process was finished, and all network latency would be
absorbed by other goroutines trying to establish a new
connection.
Now the lock is acquired after the connection has been
created. The downside of this approach is that concurrent
goroutine may be trying to open a connection to the same
target. The loser in the race will then have to Close the
connection and use the winner's instead.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Cached connections can be shared across concurrent
operations, and their disposal must take that into
account to avoid closing a connection that is stale for
one goroutine, but is still valid for another.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Avoid asking for SSH credential in files, as they won't be
used. The cacheKeyAndConfig func already enforces this
behaviour.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The major Git SaaS providers have repository URLs
for both HTTP and SSH that tops around 250
characters in length.
The limits chosen were a lot higher to align with use
cases in which users may have on-premise servers with
long domain names and paths.
For SSH the validation is around path length only,
which is now limited to 4096 characters, which is
at the higher end of the range in Linux.
For HTTP the validation is around the full URL
provided by the caller.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Internal and upstream calls to sshSmartSubtransport.Close()
when dealing with an stale connection, may lead to misleading
errors.
Focus should instead be redirected to ensuring that Close()
releases resources and ensures that a new SubTransport can be
created, so new operations can succeed.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
SSH servers that block the reuse of SSH connections for
multiple SSH sessions may lead to EOF when a new session
is being created.
This fixes the issue of long-running connections resulting
in EOF for GitLab servers.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Adds a flag `ssh-kex-algos` which configures the gogit and libgit2
managed clients to use the specified list of kex algos for ssh. If not
used the default list in `golang/x/crypto/ssh` is used.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
All errors that were previously not handled are now logged through
traceLog, to further help during transport investigations.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Some git servers are more accommodating than others. Gitlab will try to
validate credentials when they are provided, even if they are empty and
the target repository is public, leading to a failed authentication error.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Ensure all requests are completely processed and closed,
to prove odds of the underlying connections to be reused.
The transport now is pooled and reused whenever possible.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Info was receiving odd number of arguments leading to:
odd number of arguments passed as key-value pairs for logging
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
For backwards compatibility, support for HTTP redirection is enabled when targeting
the same host, and no TLS downgrade took place.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The initial implementation was based off upstream, which cause
an initial request to fail, and only then the credentials would
be added into the request.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
libgit2 network operations are blocking and do not provide timeout nor context capabilities,
leading for several reports by users of the controllers hanging indefinitely.
By using managed transport, golang primitives such as http.Transport and net.Dial can be used
to ensure timeouts are enforced.
Co-Authored-by: Sunny <darkowlzz@protonmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
NOTE: This should be amended with the previous commit which has
commented out tests.
Update reconcileSource() to work with the test case where no secret is
set. A minimal auth options is created and used for git checkout.
Update TestGitRepositoryReconciler_verifyCommitSignature() to use the
new git.Commit type.
Update TestGitRepositoryReconciler_reconcileSource_checkoutStrategy to
add skipForImplementation for branch commit test case.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If there is no configuration set for `init.defaultBranch`, it does not
return an error but an empty string. We now take this into account so
we do not overwrite the default, and make the default `master` to match
with libgit2 defaults.
In addition, some comments have been added to not get confused about
what commits we are checking against.
Signed-off-by: Hidde Beydals <hello@hidde.co>
In the recent update from libgit2 1.1.x to 1.3.x, something seems to
have changed upstream. Resulting in the clone of a branch ending up
with a semi-bare file system state (in other words: without any files
present in the directory).
This commit patches the clone behavior to set the `CheckoutForce`
strategy as `CheckoutOption`, which mitigates the issue.
In addition, test cases have been added to ensure we do not run into
this again by asserting the state of the branch after cloning.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Sanskar Jaiswal
Paulo Gomes
Sunny
Somtochi Onyekwere
Eng Zer Jun
Hidde Beydals