Update initRepoWithRemote() so that it overwrites the remote url with
the provided url if the remote already exists, instead of erroring out.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Decommission libgit2 unmanaged transport and remove the related feature
gate, making managed transport the default.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
BitBucket servers don't accept a username with an empty password, so a
secret with no http auth creds will result in a 401, since we
fall back to "git" for the username and used to set basic auth with that
username without a password.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Known hosts can be a difficult problem to troubleshoot.
To make it easier for end users, the generic message has
now been changed with a much more user friendly one.
Now if a known_host is not set, an error message will be
returned, instead of it simply being ignored.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The connection type was created to group the connection related fields
and use mutex to prevent race conditions. Since that's no longer the
case, this puts back those fields in sshSmartSubtransport.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
The variable used to store the information about proxied request was
being written to in the proxy server request handler and read for
assertion at the end of the test.
Replace the boolean variable with an atomic counter to count the number
of requests proxied, preventing the race condition.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Debugging connection issues can be extremely difficult, even more so at scale or when
concurrent connections are required to trigger specific issues.
Changes:
- Add a correlation identifier for each reconciliation, which allows for greater traceability when
going through all the reconciliation operations - including at transport level.
- Add transportType to segregate HTTP and SSH transport logging.
- SSH operations are now enriched with addr containing server address, and HTTP url.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Refactors libgit2 checkout tests to test managed and unmanaged
transport by making sure the tests requiring unmanaged transport are run
before, any tests that require managed transport (since disabling
managed transport isn't possible). This is done via arranging the tests
carefully in alphabetically sorted names, i.e. the tests with unmanaged
transport go in `checkout_test.go`, which forces golang to run the tests
in that file before any other tests.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
The average SubTransport lifecycle encompass two Actions calls. Previously,
it was attempted to share the same connection across both calls. That did
not work as some Git Servers do not support multiple sessions from the same
connection. The implementation was not fully transitioned into the
"one connection per action" model, which led to connection being leaked.
The transition to RW mutex was to avoid the unnecessary blocking in the
goroutine at the start of the second action call.
It is worth mentioning that now when the context is done, the client level
resources (connection) will also be freed. This ensures that SSH connections
will not outlive the subtransport.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Git repositories may be redirected to different URLs
when they are accessed via HTTP. The two most obvious
scenarios are from HTTP to HTTPS and when the .git suffix
is missing.
By improving the logging on this process users can identify
changes required to their GitRepository objects.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Earlier, host key verification could potentially fail if there were
multiple entries in the known_hosts file and if the intended encryption
algorithm wasn't the first entry. This happened because we used the same
hasher object to compute the sum of all the public keys present in the
known_hosts file, which led to invalid hashes, resulting in a mismatch
when compared with the hash of the advertised public key. This is fixed,
by not creating the hasher ourselves and instead delegating that to the
function actually doing the matching, ensuring that a new hasher is used
for each comparison.
Regression introduced in v0.25.0 and reported in
https://github.com/fluxcd/image-automation-controller/issues/378
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Gitlab only supports HTTP redirection for GET operations,
and fails POST operations targeting a repository without
the .git suffix.
Fixes: https://github.com/fluxcd/image-automation-controller/issues/379
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Race conditions in ssh smart subtransport caused some goroutines to
panic, resulting in crashing the whole controller, mostly evident in
image-automation-controller CI runs. Panic recovery in the main thread
do not handle goroutine panics. So, the existing panic recovery code in
libgit2 Checkout() methods weren't able to handle it.
This change groups the fields in ssh smart subtransport that may be
accessed by multiple goroutines into a new struct with a mutex. Also
adds panic recovery in the created goroutine to handle any other
possible panics.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Use the target commit, to checkout tree and set the head to the desired
branch instead of doing a hard reset to the target commit.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
TestMain() without m.Run() resulted in none of the tests being run.
This libgit2 package contains tests for both managed and unmanaged
transports, enabling it in TestMain() for everything results in the
unmanaged transport tests to not run. Add managed transport
initialization call to all the managed transport tests to allow
unmanaged transport tests to run.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
The context passed to Checkout() has a timeout. Pass it forward to
the transport in the option for all the checkouts.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Some scenarios could lead a goroutine to be running indefinetely within managed ssh.
Previously between the two git operations, the reconciliation
could take twice the timeout set for the Flux object.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Injects transport and auth options at the transport level directly to
bypass the inbuilt credentials callback because of it's several
shortcomings. Moves some of the pre-existing logic from the reconciler
to the checkout implementation.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.
Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.
Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.
The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce concrete and partial commits. Concrete commits have all the
information from remote including the hash and commit content. Partial
commits are based on locally available copy of a repo, they may only
contain the commit hash and reference.
IsConcreteCommit() can be used to find out if a given commit is based on
local information or full remote repo information.
Update go-git and libgit2 branch/tag clone optimization to return a
partial commit and no error.
Update and simplify the go-git and libgit2 tests for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Hashed known_hosts was previously only supported when using
go-git. Now both Git implementations benefit from this
features, and the code coverage across them can ensure no
future regression.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Both libgit2 and go-git now have tests that confirms
support for the supported authentication algorithms.
This test was created previously and is no longer
needed.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Connection caching was a feature created to resolve
upstream issues raised from concurrent ssh connections.
Some scenarios were based on multiple key exchange
operations happening at the same time.
This PR removes the connection caching, and instead:
- Services Session.StdoutPipe() as soon as possible,
as it is a known source of blocking SSH connections.
- Reuse SSH connection within the same subtransport,
eliminating the need for new handshakes when talking
with the same server.
- Simplifies the entire transport logic for better
maintainability.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
getBlankRepoAndRemote's callers are responsible for the disposal
of the returned objects. However, the caller does not expect to
need to dispose objects when err != nil, which may result to memory
leaks.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.
This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Stefan Prodan
Somtochi Onyekwere
Philip Laine
Sanskar Jaiswal
Sunny
Paulo Gomes