Also, removes the panic recovery code added in cloneInto() as the
underlying Checkout() in SC already has a panic handler. The issues were
due to goroutine panicking, which are now handled in the SC libgit
managed transport.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Fixes regression in which we fail to push to a branch after switching to
a branch, if origin is ahead of local. Fixed by setting the upstream
commit as the local branch target.
Regression introduced in #330, and partially addressed in #369.
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
Fixes regression in which IAC will fail to update push branch
when the push branch already exists and checkout branch is
ahead.
The reconciliation errors with: 'cannot push because a reference
that you are trying to update on the remote contains commits
that are not present locally.'
Regression introduced on #330.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
- Panic recovery for Git operations.
- Improved SSH connection management without use of caching.
- Enforce context timeout for managed SSH.
- Remove dependency to callback functions.
- Add support for hashed known_hosts.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
libgit2 network operations are blocking and do not provide timeout nor context capabilities,
leading to several reports of the controllers hanging indefinitely.
By using managed transport, golang primitives such as http.Transport and net.Dial can be used
to ensure timeouts are enforced.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
ImageUpdateAutomation objects can now refer to GitRepository objects in other
namespaces. Implemented by switching sourceRef from a SourceReference to a
dependency.CrossNamespaceDependencyReference.
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
os.RemoveAll can return an error; even if we're exiting the procedure
at the time, it's worth knowing when it's failing, because typically
the working directory is a memory-backed volume and you can run out
quite easily.
Signed-off-by: Michael Bridgen <michael@weave.works>
Pass cancellable context to libgit2.RemoteCallbacks to be able to cancel
the remote operations when the context is cancelled.
For git clone, fetch and push, a context is created with the timeout of
the target GitRepository.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit makes use of the refactored `git` package, which has
been reworked to increase stability and test coverage, and ensures
implementation details do not leak out into the "main wrapper".
This indirectly seems to resolve a memory leak that happenedd with
the previous wiring, thereby fixing #247.
The code changes for this controller itself are minimal, mostly
ensuring the auth and checkout configurations are created in the
"new way".
Signed-off-by: Hidde Beydals <hello@hidde.co>
All functions are not supported, The following are discarded:
// Date functions
"date",
"date_in_zone",
"date_modify",
"now",
"htmlDate",
"htmlDateInZone",
"dateInZone",
"dateModify",
// Strings
"randAlphaNum",
"randAlpha",
"randAscii",
"randNumeric",
"uuidv4",
// OS
"env",
"expandenv",
// Network
"getHostByName",
Signed-off-by: Soule BA <bah.soule@gmail.com>
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.
This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.com/hiddeco/golang-with-libgit2#usage) for more
detailed information).
The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.
Signed-off-by: Hidde Beydals <hello@hidde.co>
github.com/fluxcd/pkg/runtime/logger now supports the trace log level,
and has consts for the V(...) argument to supply. This commit updates
that module so to use those consts.
Previously, trace messages were output at the debug level; after this
commit they will be in their own level, so you need
--log-level=trace
to see them.
Signed-off-by: Michael Bridgen <michael@weave.works>
libgit2's Push method will succeed even when ref updates are rejected,
meaning it can silently fail if you e.g., use branch protection in
GitHub.
To make these errors visible, a callback is supplied to Push, which
checks for a non-empty status (on the advice of
https://libgit2.org/libgit2/#HEAD/group/callback/git_push_update_reference_cb).
For whatever reason, gogit seems overly sensitive to hook errors (in a
way that `git` and libgit2 aren't), and reports "invalid pkg-len
found" when it sees a rejected ref message. This doesn't affect the
runtime code, since that uses libgit2 -- but it does affect the test
code, which initialises the git repo used in many tests, so more care
is needed to push only the main branch, so as not to trigger a
rejection.
Signed-off-by: Michael Bridgen <michael@weave.works>
This bumps the version of the image reflector types to v1beta1. This
doesn't technically make a difference for the minute, since the
Kubernetes API server would convert between versions, but is tidier.
Signed-off-by: Michael Bridgen <michael@weave.works>
This does the following:
- copies the type definitions from v1alpha2 to v1beta1
- changes the "stored" version to v1beta1
- gives the CRD a conversion strategy of None, meaning just rewrite the version
- switches the controller to use v1beta1
- moves the generated documentation to v1beta1
This effectively rebadges the v1alpha2 version of this part of the
image API to v1beta1. The v1alpha2 version is left in place; there are
no conversion issues, as with v1alpha1->v1alpha2. The CRD specifies
that converting between v1alpha2 and v1beta1 just means changing the
version (i.e., the schema and semantics are the same).
Signed-off-by: Michael Bridgen <michael@weave.works>
The controller is now working with 4 concurrent workers by default.
This value is configurable through the `--concurrent` flag.
Signed-off-by: Max Jonas Werner <mail@makk.es>
- trace different code paths, e.g., how the push branch is chosen
- move debug output so it records things not already covered by e.g.,
errors, events
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit finesses the use of the debug log a little, and introduces
a trace log. The trace log gets threaded through calls to utility
procedures -- it's a little awkward putting loggers into func
parameters and structs, but it always is.
Signed-off-by: Michael Bridgen <michael@weave.works>
source-controller/pkg/git does shallow clones when using the go-git
implementation, and apparently this causes problems when fetching a
branch that has been merged at the origin:
https://github.com/fluxcd/image-automation-controller/issues/164
So far as I can tell, getting a shallow clone breaks the automation,
no matter whether go-git or libgit2 is used for operations after
cloning. So: just use libgit2 for cloning, which means non-shallow
clones; and, for fetch and push, since there's no functional
difference between the implementations for those.
Signed-off-by: Michael Bridgen <michael@weave.works>
As `golang.org/x/crypto/openpgp` has been deprecated (see
https://github.com/golang/go/issues/44226 for details), and this is the
most active/used fork.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This changes the API so that the checkout field has a ref, the same as
GItRepository. This means you can check out a branch or a tag or a
particular commit. Most of these won't work unless you supply a branch
to push to as well.
An addtional change is that you can leave out the checkout altogether,
and the ref will default to that given in the GitRepository, or its
default. In the latter case, again you will need to provide a push
branch.
Signed-off-by: Michael Bridgen <michael@weave.works>
This is a bit neater to read and write, and since I'm making breaking
changes anyway.
The name is now optional; an email is enough.
Signed-off-by: Michael Bridgen <michael@weave.works>
This finishes the v1alpha2 API, and rewrites everything needed so that
the controller supports it and the tests pass. For the most part, that
is just changing the location of fields. However, there's a few
notable extras:
- check that the `sourceRef` is a git repo (that's the default), and
that a `.spec.git` is supplied;
- change a test that blindly patched an update object, so that it
first gets the object it's patching. Previously, it succeeded
because it was OK to patch everything to empty strings, but that's
no longer the case since SourceReference.Kind is an enum.
Signed-off-by: Michael Bridgen <michael@weave.works>
There is a bug in go-git which leads to it reporting broken, absolute
symlinks as modified whether they are or not:
https://github.com/go-git/go-git/issues/253
To date, the controller checks whether the repo it has run an update
on is Clean, and as a consequence will run into the bug above if a
broken symlink is in the repo. The result is that it makes and pushes
an empty commit every interval.
To work around the problem, this commit adds a more careful check of
the repo status. Each file reported as modified is validated by
checking specifically that it's not a broken symlink: if `os.Lstat`
says it's a symlink and `os.Stat` reports the (target) file is
missing, it can be ignored. (Why not just ignore any missing file?
Because a missing file might indicate some other problem, so better to
let it fail).
For convenience, I have moved a few procedures around so they can be
used more readily by go tests.
Signed-off-by: Michael Bridgen <michael@weave.works>
For the "push to branch" feature, the controller must either switch to
the branch given, or create it starting at the checked-out HEAD. The
func `switchBranch` encapsulates this decision -- but it assumes that
if the branch exists at the remote, it will have been fetched when
cloning, and this is not always true. In particular, cloning with
go-git avoids fetching all refs:
https://github.com/fluxcd/source-controller/blob/v0.11.0/pkg/git/gogit/checkout.go
This commit adds a step to fetch the remote branch to a local branch,
before attempting to switch to the local branch. This makes
`switchBranch` a little simpler, and doesn't rely on any refs having
been fetched ahead of time.
Signed-off-by: Michael Bridgen <michael@weave.works>
Prior to #27, controller indexed the automation objects against image
policies, since an automation could depend on a specific image
policy. That PR removed the references and the watch; however,
automation objects still depend on image policy objects, just
indirectly through the git repo.
This commit reinstates the watch, and makes sure the generation change
/ reconcile request predicate applies only to the watch on automation
object themselves.
Signed-off-by: Michael Bridgen <michael@weave.works>
Paulo Gomes
Stefan Prodan
Sunny
Sanskar Jaiswal
Somtochi Onyekwere
Aurélien GARNIER
Luke Mallon (Nalum)
Aurel Canciu
Michael Bridgen
Hidde Beydals
Soule BA
David Korczynski
Max Jonas Werner