38317ab7c0
Extract HelmRepository secret retrieval
...
Method getHelmRepositorySecret on the HelmChartReconciler
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 15:27:11 +02:00
f3370d8004
Extract chart repo url normalization logic
...
Move the logic to helm/utils exported as func
NormalizeChartRepositoryURL
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 14:57:12 +02:00
ad995d80cc
Reorganize helm chart testdata for controllers
...
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 14:57:12 +02:00
54aaef5380
Index HelmRepository resources by their URL
...
To facilitate an inexpensive lookup when collecting credentials and
index artifacts while working with chart dependencies.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 14:57:12 +02:00
08d98ff214
Add finalizers to RBAC
...
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-10-28 14:39:53 +02:00
c9eb8f03c0
Switch to Masterminds/semver and pkg/version libs
...
Co-authored-by: Illia Ovchynnikov <illia.ovchynnikov@gmail.com>
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-28 12:21:08 +01:00
582c2092b6
Change copyright to Flux authors
...
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-27 17:54:22 +01:00
684624b1a0
Add support for loading packaged helm charts
...
The feature allows the source-controller to load packaged helm charts
for HelmChart resource artifacts from GitRepository and Bucket sources
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-27 11:15:17 +02:00
504ae25954
Switch to controller-runtime utils for finalizers
2020-10-21 11:50:13 +02:00
03e32491bf
Implement Prometheus instrumentation
...
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-10-13 13:59:30 +03:00
c8c2eec3a6
Update fluxcd/pkg/runtime to v0.1.0
...
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-10-13 13:34:47 +03:00
f16e5f041a
Promote API to v1beta1
2020-09-30 16:11:00 +03:00
431ea05427
Implement `fluxcd/pkg/meta/api` in APIs
2020-09-30 10:18:43 +02:00
8bf7d8f440
Factor out Helm repo index and chart download
2020-09-24 12:40:04 +02:00
b9576d56f1
Prevent resources getting stuck on transient err
...
This commit ensures that resources will only return early if they are
already in a `Ready==True` state. If not, but the status object somehow
still reports that it has an artifact, the reconciliation will continue
to ensure and/or guarantee state, and to prevent a deadlock from
happening.
2020-09-22 17:00:54 +02:00
969a46f4d6
internal/helm: return callback on empty TLS config
...
...and no-op on empty valuesFile string.
2020-09-21 23:27:38 +02:00
4da80b65e5
storage: add CopyFromPath method
2020-09-21 23:16:13 +02:00
270b6a5c0c
api: add HasRevision method to Artifact
2020-09-21 22:41:51 +02:00
03ce9d96da
Support Helm charts from Bucket sources
2020-09-21 12:40:15 +02:00
6f8c3816f4
Add secrets read-only access to RBAC
2020-09-18 16:11:28 +03:00
2ca47fb4ba
Guard against missing S3 credentials
2020-09-18 16:11:24 +03:00
c2c61b10a1
Implement S3 Bucket controller
2020-09-18 16:11:24 +03:00
7268c8b61d
Refactor and factor out chart values replacement
2020-09-11 12:16:41 +02:00
fd36d2d4f9
Allow overwriting chart values from HelmRepository
2020-09-11 11:32:54 +02:00
2e0b6cb6c9
Allow overwriting chart values from GitRepository
2020-09-11 11:32:45 +02:00
f40a42115b
Conditionally remove artifact from source status
...
We only want to remove it if it is no longer in storage, otherwise we
still want to keep it around till we have a new artifact.
2020-09-11 10:10:47 +02:00
b5d869db03
Include PATCH rule for events in manager-role
...
During high custom resource count / low interval tests, I was greated
with a `cannot patch resource "events"` message. This happened due to
event compaction, where it will perform a patch instead of a create.
By giving the role the permission to do so this should no longer pose
a problem.
2020-09-10 21:17:10 +02:00
d03f4fa4c4
Change advertised artifact URLs on hostname change
2020-09-10 14:13:46 +02:00
7a3a5938d3
Mark resources as progressing on spec changes
2020-09-10 14:13:41 +02:00
42706a342b
Calculate checksums during file writes
2020-09-10 13:00:08 +02:00
1ab12869ac
Make storage file writes atomic
2020-09-10 12:02:32 +02:00
901463eaaa
Use pkg/runtime for predicate and logger
2020-09-09 17:57:16 +01:00
68ccf2598f
storage: actually record relative path in artifact
2020-09-09 16:32:25 +02:00
68947cfca6
controllers: resolve issue with gc on delete
...
When a delete of a resource is requested a `deletionTimestamp` is set
on the resource by the requester, this also results in a generation
change of the resource.
If the resource is under reconciliation while this timestamp is set, and
had not produced an artifact earlier on, this becomes a problem as the
artifact metadata is used to determine what should be garbage collected
on a deletion, resulting in stray files for resources that are no longer
present.
To resolve this for now, we always create a new artifact object for the
resource when `all==true` on the GC method call, and no longer rely on
the presence of the artifact object on the resource itself.
2020-09-09 13:41:56 +02:00
99b74da044
api/artifact: add checksum field to artifact
...
This includes a change to how the revision for HelmRepository sources is
recorded, as this will now equal to the generated timestamp from the index
in RFC3339Nano format.
2020-09-09 13:41:56 +02:00
0b752178b1
storage: only store relative path in artifact
...
As the storage base directory is determined during runtime, and
artifacts may live longer than that if they are e.g. stored in a
persistent volume but the mount path configuration changes.
2020-09-08 17:21:56 +02:00
f8c4bd31ca
api/artifact: add ArtifactDir helper func
...
To make it easier to construct just the directory path for the
artifact (relative to the storage base path).
2020-09-08 15:16:58 +02:00
2c4dcfe72d
helmchart: use dir of artifact path on package run
2020-09-01 16:08:20 +02:00
b7603f9fd3
storage: change logic of `ArtifactExist` method
...
Given that:
* The produced artifact as advertisted in the path should always
be a regular file (including the exclusion of symlinks).
* The produced artifact should be readable, so any type of error
should count as "does not exist".
We should use `os.Lstat` to not follow symlinks; return `false`
on any error we run in to, or return if the file mode information
reports a regular file.
2020-09-01 16:01:19 +02:00
24f47ac183
Support Helm charts from GitRepository sources
2020-08-31 16:21:53 +02:00
f4d047a4a2
Support Helm repository indexes with relative URLs
2020-08-31 10:18:30 +02:00
d38b8fe193
Support proper semver ranges for Helm charts
...
This commit changes the semver range parser to `blang/semver`, which
is also used to parse semver tags for GitRepository sources.
2020-08-31 10:03:47 +02:00
3475d741a3
Switch to dedicated git and helm testserver pkgs
2020-08-18 12:11:29 +02:00
9d947b8efa
Use testserver package from fluxcd/pkg
2020-08-18 12:11:29 +02:00
2044de40fd
Add HelmRepository timeout test
2020-08-12 15:32:55 +03:00
4486ab7a5e
Implement HelmRepository timeout
2020-08-12 14:10:03 +03:00
8e1b213da5
Change CRD domain to 'source.toolkit.fluxcd.io'
...
Due to required domain changes for the helm-controller so that it
can co-exist in a cluster with the Helm Operator, other Toolkit
components are moving to a *.toolklit.fluxcd.io domain too.
2020-07-30 21:50:46 +02:00
a0357172bc
Helm: allow configuration of TLS certs without CA
2020-07-20 21:33:55 +02:00
e65e2f0623
Make git package public
2020-07-20 13:10:18 +03:00
e72855f88d
Refactor garbage collectors to finalizers
2020-07-16 14:18:33 +02:00
ba3ed63a9d
Remove HelmRepository owner ref on HelmChart
...
As the HelmChart is not the result of the HelmRepository, but more
like a Deployment -> Secret relationship.
2020-07-16 12:39:53 +02:00
70b095dccd
Merge pull request #90 from erikh/fix-bug-in-filepath-walk
...
Storage: fix a small bug in RemoveAllButCurrent when the directory is invalid
2020-07-16 09:21:11 +03:00
687b79a7dd
Storage: fix a small bug in RemoveAllButCurrent when the directory is invalid
...
filepath.Walk can return a `nil` for the stat value, when it does, the
directory is invalid and the error will be set. This causes a
panic+crash if the directory does not currently exist when
RemoveAllButCurrent is called.
The following patch makes the behavior an error instead.
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
2020-07-15 18:50:09 +00:00
eb20a8f465
Improve error handling and reporting
...
- return reconciliation error so that controller runtime metrics record failures
- change structure logging labels to match the controller runtime format
- log the reconciliation duration for all kinds
- normalise log messages and labels across all controllers
2020-07-15 12:24:11 +03:00
b3e58b48dd
Rename syncAt annotation to reconcileAt
...
Use `fluxcd.io/reconcileAt` annotation across all controllers.
2020-07-13 16:06:16 +03:00
c90be06345
Add constants for kind names
2020-07-10 12:07:54 +02:00
b9dc2ecf64
Add ignore field to GitRepository spec
2020-07-09 14:48:02 +03:00
a723b9e3e7
Archive and storage tests
...
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
2020-07-08 23:23:33 +00:00
b1b1dbcec5
Support programming excluded patterns in gitrepository spec
...
-- More coming in this commit message soon
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
2020-07-08 14:38:44 +00:00
6f89f2d4c5
Implement event recording
...
- emit Kubernetes events for artifact acquisition actions
- forward events to notification controller
2020-07-03 08:46:26 +03:00
e92cbbdd45
Migrate to fluxcd/pkg
2020-06-30 17:30:21 +03:00
0bc28f7a42
testserver: move from internal to pkg
2020-06-30 11:48:14 +02:00
384bf9fe7c
testserver: suffix structs with Server
2020-06-30 11:42:52 +02:00
39ee34897a
Replace tar/gzip shell-out with Go implementation
2020-06-06 19:15:52 +02:00
24b77d37a8
controllers: GitRepository ref
2020-05-29 14:24:31 +02:00
b27c82dd1b
controllers: GitRepository test refs
2020-05-29 14:24:31 +02:00
77822bd4d1
controllers: simple GitRepository test
2020-05-29 14:24:31 +02:00
96a76c2ad8
Merge pull request #55 from fluxcd/tar-ignore
2020-05-13 20:06:01 +02:00
e880a45474
controllers: control tar ignores w/ exclude files
...
This commit changes the file excludes for tarballs generated for
Git repository artifacts from a fixed set of strings to include
exclusion files files. It currently takes `.sourceignore` and
in the root of the given directory into account.
In addition to this the Git VCS related files that are ignored have
been extended to not only include the .git/ directory, but also the
.gitignore, .gitmodules and .gitattributes files. Mimicking part of
the --exclude-vcs flag not available on all tar versions.
2020-05-13 19:45:31 +02:00
9c67baa158
controllers: implement checkout strategies
2020-05-03 23:04:12 +02:00
e42561f13b
git: refactor transport into strategies
2020-05-03 10:03:47 +02:00
d0487302f6
git/ssh: use in-memory known hosts database
2020-05-03 10:03:42 +02:00
48fcc52a3d
controllers: factor out GitRepository verification
2020-05-02 21:44:27 +02:00
ca884a9b37
test: fix GC tests for macOS
2020-04-30 09:56:46 +03:00
2c0b175807
controllers: check GitRepository reference != nil
...
To prevent the reconciler from panicing when no reference is
configured for the GitRepository.
2020-04-29 16:45:25 +02:00
5ee875afbd
status: add revision to the ready condition message
2020-04-29 08:49:45 +03:00
f0497d3164
Fix revision for git tags
2020-04-28 20:18:33 +03:00
9540efe9de
git: add archive integrity check
2020-04-27 13:39:36 +03:00
8071dadbf0
status: record progressing
...
Set ready condition to unknown while the reconciliation is progressing.
This allows other operators to wait for a sync to complete.
2020-04-27 11:16:35 +03:00
502a80b194
git: Update status after sync failure
2020-04-26 23:21:16 +03:00
2103d38a70
Merge pull request #33 from fluxcd/git-timeout
...
api: add timeout field to GitRepositorySpec
2020-04-22 12:16:31 +02:00
920d37fcda
api: add timeout field to GitRepositorySpec
...
This commit adds a timeout field to the GitRepositorySpec to be used
during the git clone operation when reconciling the resource.
When no interval is defined the default timeout returned by the getter
is 20 seconds.
The timeout can not be added yet to the Helm related sources as it
is currently not possible to inject anything custom into the HTTP
client from the Helm HTTP getter except for the authentication
options built in. A submit has been submitted to make this possible
and is waiting for review.
This commit includes some context changes to the other reconcilers
to tidy them up and make them depend on a single background context.
It also includes some added docblocks that crossed my path.
2020-04-22 11:42:27 +02:00
61d49cab5e
controller: Move SyncAt annotation to API
2020-04-20 10:51:33 +03:00
5b77100589
controllers: make concurrent reconciles config opt
...
Introduces new helpers and config structs to all reconcilers to
set the max concurrent reconciles number.
Introduces a new flag `--concurrent` to configure the number of
concurrent reconciles per reconciler, defaults to `2`.
2020-04-19 11:52:10 +02:00
236a6950cd
controllers: test HelmChart missing HelmRepository
2020-04-17 20:39:33 +02:00
7e7f1ccd89
controllers: test HelmChart secretKey usage
2020-04-17 20:29:45 +02:00
d36efa70dd
controllers: slightly restructure HelmRepository tests
2020-04-17 19:21:55 +02:00
6083d886ce
controllers: add HelmChart reconciler tests
2020-04-17 19:21:48 +02:00
31656c2d62
controllers: test HelmRepository invalid URL
2020-04-17 12:26:42 +02:00
8f09c453ab
controllers: test HelmRepository missing secretRef
2020-04-17 12:17:04 +02:00
3c70c8d333
controllers: test HelmRepository TLS auth
2020-04-17 10:41:24 +02:00
1cc6464b73
controllers: test HelmRepository HTTP basic auth
2020-04-17 10:36:24 +02:00
a50ea436fa
controllers: basic HelmRepository reconciler tests
2020-04-17 10:35:52 +02:00
ab3879d170
controllers: check if != nil before defering
2020-04-16 11:11:51 +02:00
40c1851ffc
Add verification failed reason
2020-04-14 18:35:23 +03:00
440c70d010
Fix public key naming
...
Co-Authored-By: Hidde Beydals <hiddeco@users.noreply.github.com>
2020-04-14 18:26:59 +03:00
96e289fda0
Implement PGP signature verification
...
- check if HEAD commit is PGP signed
- load GPG public keys from Kubernetes secret
- verify PGP signature
2020-04-14 18:25:25 +03:00
45da46216c
Merge pull request #21 from fluxcd/log-context
...
Use same logger during reconcile operations
2020-04-14 12:08:53 +02:00
1832b2d11d
Introduce source interface
...
- add source interface with `GetArtifact` and `GetInterval` funcs
- implement source interface for all types
- fix HelmChart requeue
2020-04-14 12:49:31 +03:00
b0f4908af0
Use same logger during reconcile operations
2020-04-14 11:36:46 +02:00
e88d7219ca
Change artifact path format
...
- move artifact path composition to API
- change path format to `<source-kind>/<source-namespace>/<source-name>/<artifact-filename>`
2020-04-13 19:52:30 +03:00
f8e0685af1
Merge pull request #14 from fluxcd/git-auth-helper
...
Internal helpers for Git auth methods from secrets
2020-04-13 13:44:00 +02:00
90348ebf08
Internal helpers for Git auth methods from secrets
2020-04-13 13:19:23 +02:00
23f0b38ac2
Add helmchart/finalizers RBAC rule for OpenShift
2020-04-13 13:12:41 +02:00
194371c4c0
Set owner ref to HelmRepository on HelmChart
2020-04-13 12:29:13 +02:00
ea610829c3
Helm repository and chart HTTP and TLS auth
2020-04-13 11:47:51 +02:00
1b391f4896
Move artifact gc to delete event predicate
2020-04-13 01:35:44 +02:00
629da726c6
Rename RepositoryChange to SourceChange predicate
2020-04-13 01:11:07 +02:00
4bde6bf880
Prevent delete from wiping all artifacts for kind
2020-04-12 22:07:57 +02:00
3e0810aefe
Give namespace precedence in calc artifact path
2020-04-12 22:07:57 +02:00
7e99998c5f
Factor out injection of kind string in controllers
2020-04-12 22:07:53 +02:00
d378bd1852
Introduce HelmChart API and controller
...
- Add the HelmChart types and controller
- Semver expressions are found by utilizing Helm repository index
helpers. As Helm makes use of `masterminds/semver`, the support
for i.e. ranges less mature than the `GitRepository` implementation.
- Recorded semver is as defined in the metadata of the chart. The
used name for the artifact does however include the checksum of the
chart archive, as chart maintainers may not always properly apply
semver.
- Switches to `sigs.k8s.io/yaml` for YAML operations as this among
other things is able to properly unmarshal embedded structures.
- Directly requeues on transient errors instead of using the defined
interval as a back-off strategy is applied on repeated failures.
2020-04-12 18:44:37 +02:00
f9a35a6613
Move status condition helpers to API
2020-04-12 18:12:28 +03:00
2bbcd91544
Add artifact to Git and Helm repos status
...
- create index.yaml symlink for Helm repos
- set symlink URL in status
2020-04-12 17:39:07 +03:00
41d36f54ae
Wrap errors in git controller
2020-04-12 17:13:50 +03:00
12d25ca47a
Add symlink for latest Git artifact
2020-04-12 09:56:52 +03:00
786071fe3c
Add symlink helper to storage
2020-04-12 09:56:03 +03:00
06af12739d
Implement Git SSH authentication
2020-04-11 21:30:39 +03:00
bbc25c3de0
Implement Git HTTP authentication
2020-04-11 12:14:16 +03:00
e2d28296e1
Add GitRepositoryRef type
...
- add commit ref field
- implement commit checkout
2020-04-11 11:21:12 +03:00
d1f76995ab
Add ready condition helpers
2020-04-10 22:44:09 +03:00
131b9b8e33
Lock artifact on helm index download
...
- cleanup locks with GC
2020-04-10 16:45:23 +03:00
98deb3fa54
Add file lock helper to Storage
...
- use https://github.com/golang/go/tree/master/src/cmd/go/internal/lockedfile
2020-04-10 16:08:05 +03:00
9dae1c43d7
Use storage helper in HelmRepositoryReconciler
...
- move checksum and file write operations to Storage
- implement GC
2020-04-10 15:16:31 +03:00
ce01399c15
Refactor git controller fs operations
...
- use storage helper in GitRepositoryReconciler
- implement artifacts GC
- rename status artifacts to artifact
2020-04-10 12:01:06 +03:00
d1fb8e1ade
Add storage helpers
...
- add storage and artifact types
- extract fs operations to storage functions
- construct artifact paths from Kubernetes objects metadata
2020-04-10 11:51:15 +03:00
8cd8d8f6f2
Consolidate status conditions
...
- Use the same condition type as https://github.com/kubernetes/enhancements/pull/1624 so it can be dropped in favour of the Kubernetes type when that PR is merged
2020-04-10 08:38:40 +03:00
81873e7553
Download Helm repository indexes w/ Helm's getter
2020-04-09 15:53:38 +02:00
4d30a82ef4
Rename project to source-controller
2020-04-08 16:12:17 +03:00
62350a944b
Implement git tag semver filter
2020-04-07 13:22:55 +03:00
98901f2909
Expose git artifacts inside the cluster
2020-04-07 10:49:57 +03:00
037db0bc02
Implement repository artifacts local storage
2020-04-06 19:02:46 +03:00
3deb7caf9a
Implement force sync via predicates
2020-04-06 15:41:57 +03:00
de55b987e8
Implement Git sync with go-git
2020-04-06 10:43:06 +03:00
f740d434dc
Add repository sync interval to API
2020-04-05 18:10:30 +03:00
2fadfbd6fc
Prevent reconciliation loop on status updates
2020-04-05 17:49:22 +03:00
d9ee2ab288
Implement Helm repository readiness
2020-04-05 16:41:26 +03:00
92b7b1fe43
Scaffold repository controllers
2020-04-05 12:34:29 +03:00
Aurel Canciu
Hidde Beydals
Stefan Prodan
Michael Bridgen
Erik Hollensbe
Hidde Beydals