Commit Graph

209 Commits

Author SHA1 Message Date
Paulo Gomes ce92881013
libgit2: remove connection caching
Connection caching was a feature created to resolve
upstream issues raised from concurrent ssh connections.
Some scenarios were based on multiple key exchange
operations happening at the same time.

This PR removes the connection caching, and instead:
- Services Session.StdoutPipe() as soon as possible,
  as it is a known source of blocking SSH connections.
- Reuse SSH connection within the same subtransport,
  eliminating the need for new handshakes when talking
  with the same server.
- Simplifies the entire transport logic for better
  maintainability.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-13 11:09:02 +01:00
Paulo Gomes 262efc08bc
Add git-optimized-clones feature gate
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.

This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 14:39:21 +01:00
Paulo Gomes 2b59150fbe
tests: algorithms test coverage for go-git and libgit2
Assures support for:
- Authentication Key Types
  - rsa
  - ecdsa P256
  - ecdsa P384
  - ecdsa P521
  - ed25519
- Key Exchange Algoritms:
  - diffie-hellman-group14-sha1
  - diffie-hellman-group14-sha256
  - curve25519-sha256
  - ecdh-sha2-nistp256
  - ecdh-sha2-nistp384
  - ecdh-sha2-nistp521
  - curve25519-sha256@libssh.org
- HostKey Algoritms:
  - ssh-rsa
  - rsa-sha2-256
  - rsa-sha2-512
  - ecdsa-sha2-nistp256
  - ecdsa-sha2-nistp384
  - ecdsa-sha2-nistp521
  - ssh-ed25519

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-09 13:31:54 +01:00
Hidde Beydals fb383c9fcd Release v0.24.4
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-05-03 13:15:04 +02:00
Hidde Beydals 5b32ef8269 Update dependencies
- github.com/fluxcd/pkg/runtime to v0.14.2
- golang.org/x/crypto to v0.0.0-20220427172511-eb4f295cb31f
- google.golang.org/api to v0.77.0
- k8s.io/api to v0.23.6
- k8s.io/client-go to v0.23.6

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-05-03 12:44:15 +02:00
Hidde Beydals 57e7c79098 Release v0.24.3
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-28 09:11:42 +02:00
Hidde Beydals 4c0f22171c Release v0.24.2
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-26 08:53:17 +02:00
Hidde Beydals d2132086d0 Release v0.24.1
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-22 10:50:29 +02:00
Hidde Beydals c83f0ad96c Release v0.24.0
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-19 12:53:03 +02:00
Hidde Beydals 67ef697630 Update github.com/elazarl/goproxy
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-19 12:19:54 +02:00
Soule BA 0df8dcccec
Enable dependency manager to use in memory cache
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
  cache to retrieve reconciled HelmRepositories indexes.
- record cache events.

Signed-off-by: Soule BA <soule@weave.works>
2022-04-19 11:46:27 +02:00
Hidde Beydals 18b59351c1 Update dependencies
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.

- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 16:41:42 +02:00
Hidde Beydals 59df52d332 Update sigs.k8s.io/cli-utils to v0.29.4
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 15:47:41 +02:00
Stefan Prodan 0f64fef2a1
Update dependencies
- helm.sh/helm/v3 v3.8.2
- k8s.io/api v0.23.5
- fluxcd/pkg/runtime v0.14.1

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-04-15 10:49:31 +03:00
Paulo Gomes b750f19d19
Release v0.23.0
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-12 11:48:24 +01:00
Paulo Gomes 4198191759
Add flags to configure exponential back-off retry
Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-12 10:44:46 +01:00
Hidde Beydals eb7d3c9c92 Release v0.22.5
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-30 18:15:48 +02:00
Sunny d939e98ec2
Introduce separate positive polarity conditions
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.

Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:02:59 +05:30
Paulo Gomes 57336b8fe0
Release v0.22.4
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-28 14:19:29 +01:00
Paulo Gomes b73f18a091
Update golang.org/x/crypto
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 19:08:53 +00:00
Hidde Beydals f60239bb3b Release v0.22.3
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-25 16:14:20 +01:00
Hidde Beydals 3257d222f1 Release v0.22.2
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-23 19:46:31 +01:00
Hidde Beydals ec6d2f42c2 Release v0.22.1
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-23 10:47:46 +01:00
Hidde Beydals 1711fcaed1 Release v0.22.0
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-17 18:32:00 +01:00
Hidde Beydals 55b07f97f0 Update various dependencies
- cloud.google.com/go/storage to v1.21.0
- github.com/Azure/azure-sdk-for-go/sdk/azcore to v0.22.0
- github.com/Azure/azure-sdk-for-go/sdk/azidentity to v0.13.2
- github.com/elazarl/goproxy to v0.0.0-20220115173737-adb46da277ac
- github.com/fluxcd/pkg/apis/meta to v0.12.1
- github.com/fluxcd/pkg/gittestserver to v0.5.1
- github.com/fluxcd/pkg/helmtestserver to v0.7.1
- github.com/fluxcd/pkg/runtime to v0.13.2
- github.com/fluxcd/pkg/ssh to v0.3.2
- github.com/libgit2/git2go/v33 to v33.0.9
- github.com/minio/minio-go/v7 to v7.0.23
- github.com/onsi/gomega to v1.18.1
- google.golang.org/api to v0.73.0
- sigs.k8s.io/cli-utils to v0.29.3

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-16 11:15:59 +01:00
Hidde Beydals ec687b5c61 Add note about github.com/ProtonMail/go-crypto
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-15 22:32:12 +01:00
Hidde Beydals 44198da663 Update golang.org/x/crypto to latest
This pins it to the latest version which introduces support for
rsa-ssh2-256/512.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-15 22:17:01 +01:00
Hidde Beydals 458d67d0a2 Remove github.com/docker/cli replace
This was required by an older version of Helm, which has now updated
the dependency itself.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-15 22:11:52 +01:00
Hidde Beydals c5bfe4f227 Update github.com/docker/distribution to v2.8.0
This mitigates GMS-2022-20.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-15 22:10:53 +01:00
Hidde Beydals 91aba67781 Update Helm to patched v3.8.1
(Forked) version still includes patch to manage HTTP transport.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 12:53:30 +01:00
Hidde Beydals ec5bc1ac9a Implement Azure Blob BucketProvider
This commit introduces an Azure Blob BucketProvider implementation,
capable of fetching from objects from public and private "container"
buckets.

The supported credential types are:

- ManagedIdentity with a `resourceId` Secret data field.
- ManagedIdentity with a `clientId` Secret data field.
- ClientSecret with `tenantId`, `clientId` and `clientSecret` Secret
  data fields.
- SharedKey with `accountKey` Secret data field, the Account Name is
  extracted from the endpoint URL specified on the object.

If no Secret is provided, the Bucket is assumed to be public.

Co-authored-by: Zhongcheng Lao <Zhongcheng.Lao@microsoft.com>
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-03 13:20:48 +01:00
Stefan Prodan fe773c510d
Update dependencies
- runc v1.1.0 (CVE fix https://github.com/advisories/GHSA-fgv8-vj5c-2ppq)
- containerd v1.6.1 (CVE fix https://github.com/containerd/containerd/security/advisories/GHSA-crp2-qrr5-8pq7)

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-03-03 12:49:01 +02:00
Paulo Gomes 7d61553463
Remove dependency to helm v2
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 17:58:15 +00:00
Paulo Gomes d9d789fdb1
Reuse transport for helm chart download
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.

Crypto material is now only processed in-memory and does not
touch the disk.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00
Hidde Beydals b28669e446
Update to patched version of Helm 3.8.0
This commit updates to a version of Helm 3.8.0, with patches applied to
deal with memory leak and HTTP transport issues. The latter being
described in https://github.com/fluxcd/source-controller/issues/578.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-02 13:02:07 +00:00
pa250194 ed6c6ebc3c Introduce BucketProvider interface
This commit introduces a BucketProvider interface for fetch operations
against object storage provider buckets. Allowing for easier
introduction of new provider implementations.

The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.

The client implementations do now include safe-guards to ensure the
fetched object is the same as metadata has been collected for. In
addition, minor changes have been made to the object fetch operation
to take into account that:

- Etags can change between composition of index and actual fetch, in
  which case the etag is now updated.
- Objects can disappear between composition of index and actual fetch,
  in which case the item is removed from the index.

Lastly, the requirement for authentication has been removed (and not
referring to a Secret at all is thus allowed), to provide support
for e.g. public buckets.

Co-authored-by: Hidde Beydals <hello@hidde.co>
Co-authored by: Michael Bridgen <michael@weave.works>
Signed-off-by: pa250194 <pa250194@ncr.com>
2022-03-01 10:15:10 +01:00
Paulo Gomes be02e15410
Update github.com/sosedoff/gitkit (indirect) to v0.3.0 (CVE fix)
Fixes https://snyk.io/vuln/SNYK-GOLANG-GITHUBCOMSATORIGOUUID-72488 in the indirect dependency github.com/satori/go.uuid.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-25 13:25:03 +00:00
Sunny 9c7661dcbd helmrepo: Make NewArtifact event human friendly
Inform index size and repo instead of a revision.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:37:48 +01:00
Sunny e9ae0c232e Add kstatus client conformance tests
Use kstatus to compute the status of the objects.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:37:45 +01:00
Hidde Beydals 07a539e3d6 build: update meta API and pkg/runtime to non RC
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals ceb61a318c controllers: remove legacy test suite
As it no longer contains any test cases.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 5767291b58 gitrepo: Add more reconciler design improvements
- Remove ArtifactUnavailable condition and use Reconciling condition to
  convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
  and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
  execute them.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 31d2e6d65c Rewrite `GitRepositoryReconciler` to new standards
This commit rewrites the `GitRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying
to adhere better to Kubernetes API conventions.

More specifically it introduces:

- Implementation of more explicit Condition types to highlight
  abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
  objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
  conditions.
- First (integration) tests that solely rely on `testenv` and do not
  use Ginkgo.

There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 349739b7e4 Implement new runtime interfaces, prepare testenv
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.

Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Paulo Gomes 6e46d7fe55
Upgrade docker/distribution to v2.8.0
Fixes https://github.com/advisories/GHSA-qq97-vm5h-rrhg

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-17 20:51:48 +00:00
Paulo Gomes 00ff9fb2fa
Upgrade controller-runtime to v0.11.1
Fix for CVE-2022-21698 by upgrading the trasient dependency github.com/prometheus/client_golang.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-17 20:51:47 +00:00
Paulo Gomes f44302eed0
Bump libgit2 to 1.3.0 and git2go to V33.
Downstream breaking changes introduced since git2go@V31:
- git2go.ErrorCode was deprecated in favour of the native error type.
- FetchOptions no longer expects a pointer, but rather the actual value of git2go.FetchOptions.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:17:31 +00:00
Stefan Prodan e3e9deb3fd
Release v0.21.2
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-02-07 11:50:32 +02:00
Hidde Beydals 01946f22a3 Release v0.21.1
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-01-27 11:16:21 +01:00
Stefan Prodan 2049246adf
Release v0.21.0
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-01-26 15:38:10 +02:00