This commit replaces the explicit use of event recorder and metrics
with the helpers from github.com/fluxcd/pkg/runtime.
Doing so necessitated a few consts to represent the event "reasons",
so it is not a mechanical translation. Without thinking overly long
about it, I came up with a handful of reasons that I think will be
useful for filtering.
Signed-off-by: Michael Bridgen <michael@weave.works>
A pretty straight-forward change. NB the location of the `defer ...`
_after_ the exit if the object is suspended; this maintains the
invariant that the object will not be touched at all when suspended.
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit rewrites the controller code so it uses the conditions
helpers from fluxcd/pkg/runtime/conditions.
It needs the fluxcd/source-controller/api module from the
reconcilers-dev branch, otherwise they will disagree on the methods in
fluxcd/pkg/apis/meta. Hence, variations on
go get github.com/fluxcd/source-controller{,api}@reconcilers-dev
until I got both ./go.mod and ./api/go.mod to use modules from the
branch.
Signed-off-by: Michael Bridgen <michael@weave.works>
libgit2's Push method will succeed even when ref updates are rejected,
meaning it can silently fail if you e.g., use branch protection in
GitHub.
To make these errors visible, a callback is supplied to Push, which
checks for a non-empty status (on the advice of
https://libgit2.org/libgit2/#HEAD/group/callback/git_push_update_reference_cb).
For whatever reason, gogit seems overly sensitive to hook errors (in a
way that `git` and libgit2 aren't), and reports "invalid pkg-len
found" when it sees a rejected ref message. This doesn't affect the
runtime code, since that uses libgit2 -- but it does affect the test
code, which initialises the git repo used in many tests, so more care
is needed to push only the main branch, so as not to trigger a
rejection.
Signed-off-by: Michael Bridgen <michael@weave.works>
This bumps the version of the image reflector types to v1beta1. This
doesn't technically make a difference for the minute, since the
Kubernetes API server would convert between versions, but is tidier.
Signed-off-by: Michael Bridgen <michael@weave.works>
This does the following:
- copies the type definitions from v1alpha2 to v1beta1
- changes the "stored" version to v1beta1
- gives the CRD a conversion strategy of None, meaning just rewrite the version
- switches the controller to use v1beta1
- moves the generated documentation to v1beta1
This effectively rebadges the v1alpha2 version of this part of the
image API to v1beta1. The v1alpha2 version is left in place; there are
no conversion issues, as with v1alpha1->v1alpha2. The CRD specifies
that converting between v1alpha2 and v1beta1 just means changing the
version (i.e., the schema and semantics are the same).
Signed-off-by: Michael Bridgen <michael@weave.works>
The controller is now working with 4 concurrent workers by default.
This value is configurable through the `--concurrent` flag.
Signed-off-by: Max Jonas Werner <mail@makk.es>
- trace different code paths, e.g., how the push branch is chosen
- move debug output so it records things not already covered by e.g.,
errors, events
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit finesses the use of the debug log a little, and introduces
a trace log. The trace log gets threaded through calls to utility
procedures -- it's a little awkward putting loggers into func
parameters and structs, but it always is.
Signed-off-by: Michael Bridgen <michael@weave.works>
source-controller/pkg/git does shallow clones when using the go-git
implementation, and apparently this causes problems when fetching a
branch that has been merged at the origin:
https://github.com/fluxcd/image-automation-controller/issues/164
So far as I can tell, getting a shallow clone breaks the automation,
no matter whether go-git or libgit2 is used for operations after
cloning. So: just use libgit2 for cloning, which means non-shallow
clones; and, for fetch and push, since there's no functional
difference between the implementations for those.
Signed-off-by: Michael Bridgen <michael@weave.works>
As `golang.org/x/crypto/openpgp` has been deprecated (see
https://github.com/golang/go/issues/44226 for details), and this is the
most active/used fork.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This changes the API so that the checkout field has a ref, the same as
GItRepository. This means you can check out a branch or a tag or a
particular commit. Most of these won't work unless you supply a branch
to push to as well.
An addtional change is that you can leave out the checkout altogether,
and the ref will default to that given in the GitRepository, or its
default. In the latter case, again you will need to provide a push
branch.
Signed-off-by: Michael Bridgen <michael@weave.works>
This is a bit neater to read and write, and since I'm making breaking
changes anyway.
The name is now optional; an email is enough.
Signed-off-by: Michael Bridgen <michael@weave.works>
This finishes the v1alpha2 API, and rewrites everything needed so that
the controller supports it and the tests pass. For the most part, that
is just changing the location of fields. However, there's a few
notable extras:
- check that the `sourceRef` is a git repo (that's the default), and
that a `.spec.git` is supplied;
- change a test that blindly patched an update object, so that it
first gets the object it's patching. Previously, it succeeded
because it was OK to patch everything to empty strings, but that's
no longer the case since SourceReference.Kind is an enum.
Signed-off-by: Michael Bridgen <michael@weave.works>
There is a bug in go-git which leads to it reporting broken, absolute
symlinks as modified whether they are or not:
https://github.com/go-git/go-git/issues/253
To date, the controller checks whether the repo it has run an update
on is Clean, and as a consequence will run into the bug above if a
broken symlink is in the repo. The result is that it makes and pushes
an empty commit every interval.
To work around the problem, this commit adds a more careful check of
the repo status. Each file reported as modified is validated by
checking specifically that it's not a broken symlink: if `os.Lstat`
says it's a symlink and `os.Stat` reports the (target) file is
missing, it can be ignored. (Why not just ignore any missing file?
Because a missing file might indicate some other problem, so better to
let it fail).
For convenience, I have moved a few procedures around so they can be
used more readily by go tests.
Signed-off-by: Michael Bridgen <michael@weave.works>
For the "push to branch" feature, the controller must either switch to
the branch given, or create it starting at the checked-out HEAD. The
func `switchBranch` encapsulates this decision -- but it assumes that
if the branch exists at the remote, it will have been fetched when
cloning, and this is not always true. In particular, cloning with
go-git avoids fetching all refs:
https://github.com/fluxcd/source-controller/blob/v0.11.0/pkg/git/gogit/checkout.go
This commit adds a step to fetch the remote branch to a local branch,
before attempting to switch to the local branch. This makes
`switchBranch` a little simpler, and doesn't rely on any refs having
been fetched ahead of time.
Signed-off-by: Michael Bridgen <michael@weave.works>
Prior to #27, controller indexed the automation objects against image
policies, since an automation could depend on a specific image
policy. That PR removed the references and the watch; however,
automation objects still depend on image policy objects, just
indirectly through the git repo.
This commit reinstates the watch, and makes sure the generation change
/ reconcile request predicate applies only to the watch on automation
object themselves.
Signed-off-by: Michael Bridgen <michael@weave.works>
This changes the functionality when updating the status to use Patch
rather than Updating, which is more resilient to changes.
Signed-off-by: Kevin McDermott <bigkevmcd@gmail.com>
- Add optional `path` field to `spec.update`, defaults to the git repo root
- Restrict updates to the specified `spec.update.path`
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
This adapts the controller so that it will honour the
`.spec.push.branch` field.
The behaviour _without_ that field is to check out the branch given in
`.spec.checkout.branch`, commit, and push to the origin.
With `.spec.push.branch` present, it will try to check out that
branch; if it doesn't exist, it'll create it, starting from
`.spec.checkout.branch`. Either way it'll commit to that branch and
push to the origin.
The effect is that all automation will happen on the "push" branch,
and (most likely) not be applied into the cluster until merged into
whichever branch is synced. When the push branch is deleted, it'll be
created anew; otherwise, commits will pile up there as more changes
are made.
Signed-off-by: Michael Bridgen <michael@weave.works>
This explains the data available to the commit message template in the
API guide. While writing it, I realised it could be made more
convenient, so:
- mask external types by embedding them
- make the most useful parts of an image ref available using a
wrapper struct and interface
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit:
- passes a value including the update result to the commit message
template
- gives the template result a method for enumerating the
objects regardless of file
This means you can access the images updated either by file
(`.Files`), by object (`.Objects()`), or just as a list
(`.Images()`). The additional test case shows how to use these.
Signed-off-by: Michael Bridgen <michael@weave.works>
It's desirable (see #6) to be able to enumerate the updates that were
made by automation, in the commit message and perhaps in an event
announcing success.
Doing this is counter-intuitively difficult. A `kyaml.setters2.Set`
filter will keep a count of the times its used. Previously, one `Set`
was used with the `SetAll` flag set, which would replace any marker
that corresponded to an image, in one traversal. But to keep track of
images individually, you need to have a setter for _each_ image (and
its tag, and its name, since those can be used separately). This means
`3 x policies` traversals of each node! The saving grace, possibly, is
that only files with a marker in them are considered.
Since you might want to dice the results in different ways, the result
returned is a nested map of file->object->image.
Signed-off-by: Michael Bridgen <michael@weave.works>
libgit2 and go-git both have flaws in the way they treat errors from
the remote. go-git takes only the first line, meaning that it gets a
blank error message from GitLab which like to respond with a
banner. libgit2 returns the whole response, including blank lines and
fences ("=========...").
This commit corrects for both these flaws, by supplying a message if
go-git has taken a blank line, and stripping out blank lines and
fences from libgit2's error. This is unavoidably a brittle approach,
so I have limited it to just the situation that was reported as a
problem: pushing to the upstream git repo.
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit rearranges update tests so that those that check that
updates are made can be run against a git server using SSH as well as
HTTP.
The local clone, used to provoke automated updates and to check
results, still uses HTTP. Those operations are not under test.
libgit2 wants to be asked for authentication when using SSH, and will
balk if it's not requested by the server. To avoid that, auth must be
switched on for the git test server.
This also switches auth on for HTTP, so it's necessary to use a git
URL that includes credentials for setting things up with a local
clone. I have also used that URL for the git-over-HTTP tests -- it's
arguable whether it's necessary to test that works, here.
Signed-off-by: Michael Bridgen <michael@weave.works>
The "auth strategy", which depends on the GitImplementation, was
hard-wired to the "gogit" constant, but it should come from the
GitRepository spec. When the implementation is "libgit2" and the git
URL entails SSH, the result would normally include a callback for
checking the host key against known_hosts; but since it was
hard-wired, it was missing that callback.
This explains at least some instances of the error `user cancelled
hostkey check` from #106. The error, or a close relative, might also
arise if the callback rejects the host key because the host as it
appears in the known_hosts doesn't match that host as passed to the
callback -- see
https://github.com/fluxcd/source-controller/issues/287.
Signed-off-by: Michael Bridgen <michael@weave.works>
It's convenient to be able to leave out the update strategy, since
there is only one possible value at present; and if there were
alternatives, the present choice would still be a reasonable
default. However, with the format as it is, this doesn't work with
OpenAPIv3 schema, so you have to supply a value, even though there are
no parameters:
```yaml
spec:
update:
setters: {}
```
A more self-explanatory format which _does_ work with defaulting is to
name the strategy rather than relying on the presence of a field:
```yaml
spec:
update:
strategy: Setters
```
The whole `update` field can be elided and left to default. This
doesn't preclude having other strategies later, even those with
parameters, e.g.,
```yaml
spec:
update:
strategy: Foo
fooParam: 5
```
This commit changes the API types and code that uses them, and the CRD
manifest, and adds a test that checks the defaulting actually works.
Signed-off-by: Michael Bridgen <michael@weave.works>
This follows up the stage-setting in prior commits, by respecting the
GitImplementation field given in the GitRepository object. NB it only
matters for cloning and pushing, so gogit is used in the "middle" to
record the commit in the local checkout.
Signed-off-by: Michael Bridgen <michael@weave.works>
This separates the commit and push steps, since the commit step just
uses gogit, while the push step will dispatch based on the git
implementation.
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit changes the clone and push code to use libgit2. In the
case of clone, this means simply passing the const representing the
libgit2 implementation to the source-controller/pkg/git function. In
the case of push, this means adding a small helper to do the required
invocation.
NB:
- there's no need to use libgit2 for operations other than clone and
push; in particular, commits can have a single, go-git
implementation.
- libgit2's push is quite sensitive to the refspec it's given;
`<branch>:refs/heads/<branch>` didn't work, and supplying no
refspecs makes it time out.
- libgit2 push will only work with a repository on disk that was
cloned by libgit2 -- it's initialising the repo differently to
go-git. This is surprising (a git repo is a git repo, isn't it?),
but fine -- any given automation will use _either_ go-git or
libgit2 for both of clone and push.
Signed-off-by: Michael Bridgen <michael@weave.works>
This updates the source controller module to v0.5.2, and adjusts the
use of the git package therein. The main change there is that it now
accounts for two different git implementations -- go-git and libgit2
-- but I have not exposed that difference, just made it work as it did
before.
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.
- Upgrade `runtime` to v0.6.2 to include `controller-runtime` changes.
- Logger has been removed from the reconciler, and is now retrieved
from the `context.Context` passed to the `Reconcile` method and
downwards functions.
- Logger configuration flags are now bound to the flag set using
`BindFlags` from `runtime/logger`, ensuring the same contract across
GitOps Toolkit controllers, and the `--log-json` flag has been
deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
`ReconcileRequestedPredicate`, and is now chained with the
`GenerationChangedPredicate` from `controller-runtime` using
`predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
`client.Object`, removing the requirement to e.g. call
`runtime.Object#Object`.
- The `leader-election-role` was changed, as leader election now works
via the `coordination/v1` API.
Other notable changes:
- Upgrade of `image-reflector-controller` API package to include
controller-runtime changes.
- Upgrade of `source-controller` API package to v0.6.1.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This adds fields to the ImageUpdateAutomation status for recording the
commit last pushed; handy to see when you are expecting a change.
It also adapts the "steady state" message of the ready condition to
mention the last commit, in case that's where people are looking.
Signed-off-by: Michael Bridgen <michael@weave.works>
This adds the external event recorder (a.k.a., notifications client)
to the reconciler, and expands the definition of
`<reconciler>.event(...)` so that it will send a notification whenever
an event is emitted. This is the conventional way of handling events
amongst the GitOps Toolkit controllers.
Signed-off-by: Michael Bridgen <michael@weave.works>
The convention is now to simply exit, when a GOTK object is
suspended. Previously this would update the status to indicate that it
was unready; now it just leaves it in whichever state it was before.
This also applies to the reconcile request annotation; it will _not_
be marked as seen if the object is suspended. The effect of this is
any change to the object will be passed by the predicate and therefore
reach Reconcile, until the object is unsuspended. Since it will _also_
exit early until unsuspended, this is harmless except for some extra
log lines. (But changing that ordering might be worth considering in
the future.)
This change required a few changes to tests:
- to check that suspend makes the reconciliation exit without doing
anything, explicitly run `r.Reconcile(...)`.
- to avoid waiting for the reconciler's caching client to see
changes, use an uncached client.
- (fix a problem caused by comparing a time pointer with its alias)
Signed-off-by: Michael Bridgen <michael@weave.works>
The convention among GOTK controllers is to use a "reconcile request"
annotation to force a reconcilation, outside of spec or dependency
changes. This is used by e.g., the incoming webhooks handler. The
predicate `ChangePredicate`, already used by this controller, takes
this into account by allowing events that either caused the generation
to increment, _or_ changed the reconcile request annotation.
This commit adds a test that the automation will indeed run when the
annotation is set. This is a little delicate, because I have to rule
out _other_ reasons it might run. To do so, the test makes a change to
the git repo that will be overwritten by an automation run -- a commit
will not trigger a Reconcile call since it's entirely outside
Kubernetes.
Signed-off-by: Michael Bridgen <michael@weave.works>
The `LastAutomationRun` field records the last time the automation
completed, whether it made changes or not. The purpose is to inform
users, so they can check against their expectations.
It should only be set once the automation run has completed without
error, otherwise it will provide false reassurance that things are
happening when they are not.
Signed-off-by: Michael Bridgen <michael@weave.works>
This gives ImageUpdateAutomation objects .status.conditions and
.status.observedGeneration fields, which are maintained by the
controller in the GOTK-standard way.
The only condition used is a Ready condition, compatible with kstatus
(and in common with other GOTK controllers). An object is marked Ready
if the reconciliation exits without an error, whether or not changes
were actually made. If the automation run cannot proceed, e.g.,
because the git repository referred to does not exist, or is not
cloneable, it will be marked as not ready.
This means the condition is a reliable guide to whether the particular
automation is operating or not; new objects will be marked ready as
soon as they have been run through successfully, and will stay ready
until there's a problem. Generally, if there _is_ a problem, the
object will be requeued with a backoff, or left to wait until
circumstances change (e.g., the object itself is edited); one way or
another, there will be a retry, and thereby an opportunity to
transition to ready.
Signed-off-by: Michael Bridgen <michael@weave.works>
The useful events to know about for the update automation are when it
either errors out while trying to update the git repo, or succeeds.
Signed-off-by: Michael Bridgen <michael@weave.works>
This gives the controller an event recorder, without using it yet, and
makes sure it is registered when setting everything up in main.go.
Signed-off-by: Michael Bridgen <michael@weave.works>
Michael Bridgen
Max Jonas Werner
Hidde Beydals
Stefan Prodan
LWJ
Kevin McDermott
Somtochi Onyekwere