Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This includes some rewiring of tests, and slight changes in how we work
with the local chart reference. `Path` is expected to be relative to
`WorkDir`, and both fields are now mandatory.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This introduces our own `secureloader` package, with a directory
loader that's capable of following symlinks while validating they stay
within a certain root boundary.
Signed-off-by: Hidde Beydals <hello@hidde.co>
We require these to be able to mimic Helm's own directory loader, and
surprisingly (for `ignore` at least), these are not public.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Previously the mutex.Lock was acquired before creating
a new connection. The lock would then hold until the
process was finished, and all network latency would be
absorbed by other goroutines trying to establish a new
connection.
Now the lock is acquired after the connection has been
created. The downside of this approach is that concurrent
goroutine may be trying to open a connection to the same
target. The loser in the race will then have to Close the
connection and use the winner's instead.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Cached connections can be shared across concurrent
operations, and their disposal must take that into
account to avoid closing a connection that is stale for
one goroutine, but is still valid for another.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Avoid asking for SSH credential in files, as they won't be
used. The cacheKeyAndConfig func already enforces this
behaviour.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The major Git SaaS providers have repository URLs
for both HTTP and SSH that tops around 250
characters in length.
The limits chosen were a lot higher to align with use
cases in which users may have on-premise servers with
long domain names and paths.
For SSH the validation is around path length only,
which is now limited to 4096 characters, which is
at the higher end of the range in Linux.
For HTTP the validation is around the full URL
provided by the caller.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Internal and upstream calls to sshSmartSubtransport.Close()
when dealing with an stale connection, may lead to misleading
errors.
Focus should instead be redirected to ensuring that Close()
releases resources and ensures that a new SubTransport can be
created, so new operations can succeed.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
SSH servers that block the reuse of SSH connections for
multiple SSH sessions may lead to EOF when a new session
is being created.
This fixes the issue of long-running connections resulting
in EOF for GitLab servers.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
We try to avoid affecting the source reconciliation when there's a
garbage collection related failure.
The event logging was resulting in events and notifications related to
GC failure when the artifact directory isn't created in the first
reconciliation of an object.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce two new flags to configure the ttl of an artifact and the max
no. of files to retain for an artifact. Modify the gc process to
consider the options and use timeouts to prevent the controller from
hanging.
This helps in situations when the SC has already garbage collected the
current artifact but the advertised artifact url is still the same,
which leads to the server returning a 404.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Adds a flag `ssh-kex-algos` which configures the gogit and libgit2
managed clients to use the specified list of kex algos for ssh. If not
used the default list in `golang/x/crypto/ssh` is used.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
If implemented, will provide users with a way to cache index files.
This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.
The caching strategy used is cache aside, and the cache is a k/v store
with expiration.
The cache number of entries and ttl for entries are configurable.
The cache is optional and is disabled by default
Signed-off-by: Soule BA <soule@weave.works>
All errors that were previously not handled are now logged through
traceLog, to further help during transport investigations.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Update alll the other reconcilers similar to the GitRepository
reconcilers to introduce positive condition ArtifactInStorage and
reorder the status conditions.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.
Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Expands on the current documentation to help contributors debug the controller
regardless of all its existing dependencies.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Some git servers are more accommodating than others. Gitlab will try to
validate credentials when they are provided, even if they are empty and
the target repository is public, leading to a failed authentication error.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Paulo Gomes
Hidde Beydals
Hidde Beydals
Sunny
Sanskar Jaiswal
Stefan Prodan
Soule BA
Daniel Holbach
Ilan