Commit Graph

58 Commits

Author SHA1 Message Date
Stefan Prodan f812f045c6
Set module name to `github.com/fluxcd/cli-utils`
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2023-11-20 08:45:35 +02:00
Ramon Quitales c494d01128 refactor: Fix linting issues found in golangci-lint v1.50.0 2022-11-01 22:21:31 +00:00
Karl Isenberg c46949360e feat: replace StatusPoller w/ StatusWatcher
- Add DefaultStatusWatcher that wraps DynamicClient and manages
  informers for a set of resource objects.
  - Supports two modes: root-scoped & namespace-scoped.
  - Root-scoped mode uses root-scoped informers to efficiency and
    performance.
  - Namespace-scoped mode uses namespace-scoped informers to
    minimize the permissions needed to run and the size of the
    in-memory object cache.
  - Automatic mode selects which mode to use based on whether the
    objects being watched are in one or multiple namespaces.
    This is the default mode, optimizing for performance.
  - If CRDs are being watched, the creation/deletion of CRDs can
    cause informers for those custom resources to be created/deleted.
  - In namespace-scope mode, if namespaces are being watched, the
    creation/deletion of namespaces can also trigger informers to
    be created/deleted.
  - All creates/updates/deletes to CRDs also cause RESTMapper reset.
  - Allow pods to be unschedulable for 15s before reporting the
    status as Failed. Any update resets the timer.
- Add BlindStatusWatcher for testing and disabling for dry-run.
- Add DynamicClusterReader that wraps DynamicClient.
  This is now used to look up generated resources
  (ex: Deployment > ReplicaSets > Pods).
- Add DefaultStatusReader which uses a DelegatingStatusReader to
  wrap a list of conventional and specific StatusReaders.
  This should make it easier to extend the list of StatusReaders.
- Move some pending WaitEvents to be optional in tests, now that
  StatusWatcher can resolve their status before the WaitTask starts.
- Add a new Thousand Deployments stress test (10x kind nodes)
- Add some new logs for easier debugging
- Add internal SyncEvent so that apply/delete tasks don't start
  until the StatusWatcher has finished initial synchronization.
  This helps avoid missing events from actions that happen while
  synchronization is incomplete.
- Filter optional pending WaitEvents when testing.

BREAKING CHANGE: Replace StatusPoller w/ StatusWatcher
BREAKING CHANGE: Remove PollInterval (obsolete with watcher)
2022-05-10 10:40:05 -07:00
Karl Isenberg 4599e4fb00 feat: Add dependency filter
- Pass TaskContext into TaskBuilder.Build
- Combine dependency graph for apply and prune objects.
  This is required to catch dependencies that would have been deleted.
- Replace graph.SortObjs into DependencyGraph + Sort + HydrateSetList
- Replace graph.ReverseSortObjs with ReverseSetList to perform on the
  combined (apply + prune) set list.
- Add planned pending applies and prune to the InventoryManager
  before executing the task queue.
  This allows the DependencyFilter to validate against the planned
  actuation strategy of objects that haven't been applied/pruned yet.
- Add the dependency graph to the TaskContext, for the
  DependencyFilter to use.
  This can be removed in the future if the filters are managed by the
  solver.
- Make Graph.Sort non-destructive, so the graph can be re-used by the
  DependencyFilter.
- Add Graph.EdgesFrom and EdgesTo for the DependencyFilter to use.
  This requires storing the reverse edge list.
- Add an e2e test for the DependencyFilter
- Add an e2e test for the LocalNamespaceFilter

Fixes https://github.com/kubernetes-sigs/cli-utils/issues/526
Fixes https://github.com/kubernetes-sigs/cli-utils/issues/528
2022-02-28 17:31:45 -08:00
Karl Isenberg 64fd95ebe7 chore: Log YAML to V7, instead of V4
- Rename "resource" to "object" in mutation code.
  Resource is a REST interface. Object is an instance of a Resource.
2022-02-14 23:03:21 -08:00
Jingfang Liu 6e3bb7a9ad Revert "capture the apply/reconcile status in the inventory object"
This reverts commit edee25bd85.
2022-02-02 13:03:19 -08:00
Jingfang Liu edee25bd85 capture the apply/reconcile status in the inventory object
Capture the apply status for individual resources after each apply.
If the inventory object supports status field, it is updated
after setting the inventory list at then of one apply process.

BREAKING CHANGE: Update the inventory client and inventory interfaces
to pass the apply/reconcile status.
2022-02-02 09:54:59 -08:00
Kubernetes Prow Robot 5fb19a18af
Merge pull request #488 from karlkfi/karl-validation-event
feat: Add ValidationPolicy & ValidationEvent
2022-01-27 16:12:45 -08:00
Karl Isenberg 9acdbce963 feat: Add ValidationPolicy & ValidationEvent
- Add ValidationPolicy:
  ExitEarly (default) - exit before apply/prune if any objects are invalid
  SkipInvalid - apply/prune valid objects & skip invalid ones
- Add ValidationEvent to be sent once for every invalid object or set of
  objects, when the SkipInvalid policy is selected. For ExitEarly,
  return an error for reverse compatibility.
- Add validation.Collector to simplify aggregating validation errors
  from multiple sources and extracting invalid object IDs.
- Add invalid objects to the TestContext so they can be retained in
  the inventory (only if already present). This primarily applies to
  invalid annotations and dependencies. Objects without name or kind
  should never be added to the inventory.
- Update Solver to use validation.Collector and filter invalid objects.
- Add e2e test for invalid objects.
- Update Printers to handle ValidationEvent
- Add ExternalDependencyError & InvalidAnnotationError to make it easier
  to handle and introspect validation errors.
2022-01-25 12:36:02 -08:00
Morten Torkildsen 8325b7f6c4 fix: Use the dynamic client in the InventoryClient 2022-01-24 19:35:45 -08:00
Mikhail Mazurskiy df4661a8f5
chore: fix linter failures 2022-01-21 12:57:11 +11:00
Karl Isenberg 6abffa8abe fix: Make CyclicDependencyError edges stable
- Add sorting for Edges & Verticies
- Improve tests that contain CyclicDependencyError
2022-01-19 15:24:39 -08:00
Karl Isenberg a42a1517d4 fix: Update ObjMetadataSet.Hash to not error 2022-01-19 11:15:08 -08:00
Karl Isenberg f67aaa87ac feat: MultiError for invalid annotations
- Move Validator to pkg/object/validation
- Replace ValidationError with validation.Error
- Replace MultiValidationError with generic MultiError
- Update Validator & SortObjs to use MultiError
- Add ResourceReferenceFromObjMetadata
- Rename NewResourceReference -> ResourceReferenceFromUnstructured
- Delete duplicate ResourceReference.ObjMetadata()
- Modify some error messages for consistency and clarity
- Use templating to generate some test artifacts

BREAKING CHANGE: apply-time-mutation namespace required for namespace-scoped resources
2022-01-12 15:24:53 -08:00
Kubernetes Prow Robot d7d63f4b62
Merge pull request #505 from karlkfi/karl-no-die
fix: Don't panic when ObjMetadata is invalid (missing kind/name)
2022-01-07 19:27:03 -08:00
Karl Isenberg 242b6aa83e fix: Don't panic when ObjMetadata is invalid
- UnstructuredToObjMetaOrDie -> UnstructuredToObjMetadata
- UnstructuredToObjMeta -> UnstructuredToObjectMetadata
- UnstructuredsToObjMetasOrDie -> UnstructuredSetToObjMetadataSet
- UnstructuredsToObjMetas -> UnstructuredSetToObjMetadataSet
- Remove CreateObjMetadata (Validator performs validation)
2022-01-07 18:50:30 -08:00
Karl Isenberg a0743e6912 fix: Clean up object.FieldPath
- Use string.Builder to be slightly more efficient
- Format invalid types with Go syntax, wrapped with brackets
- Test that invalid types don't panic
2022-01-07 18:27:57 -08:00
Karl Isenberg fb98bc7877 fix: Handle more validation errors as field errors
- Using field.Error allows more errors to be wrapped as a
  ValidationError, instead of interrupting validation and exiting
  early.
- Add explicit validation for kind (clearer error).
- Move NestedField from testutil to object, so it can be used at
  runtime.
2022-01-07 12:06:43 -08:00
Karl Isenberg c5636c0243 feat: Always wait for reconciliation
- Converted WaitTask to use a context for timeout/cancellation, to
  improve readability and reduce error cases. Now it only sends
  TaskResult events from one place, removing the need for a token to
  silence subsequent complete() calls.
- Add pending object tracking to WaitTask to ensure all objects are
  accounted for at least one WaitEvent.
- Upgraded applier tests to use full event comparison, for better signal
  on breaking changes.
- Enhanced task tests to consume all events before validating and test
  actual event output.
- Add set sorting to graph.SortObjs, to make wait event ordering more
  consistent, to make testing easier.

BREAKING CHANGE: wait tasks always execute after apply/prune/delete (except dry runs)
BREAKING CHANGE: wait tasks default to waiting until cancelled (previously 1m default)
2021-11-10 12:34:54 -08:00
Karl Isenberg b977b5960f fix: race condition in BuildInfo
Historically, the objects in ApplyTask were being modified in the
Start func by BuildInfo to remove the path annotation, but this can
cause a race condition in the task runner if it tried to read the
task objects.

Now a DeepCopy is made in BuildInfo instead. This avoids the race
condition but broke mutation, which executes after BuildInfo on the
original object. So we now extract the object from the into after
BuildInfo and use that instead for mutations, filters, and events.
2021-11-03 13:11:20 -07:00
Kubernetes Prow Robot f4b64aada8
Merge pull request #430 from ash2k/ash2k/crd-check-version
Check versions a CRD defines
2021-11-01 13:35:47 -07:00
Karl Isenberg eda3554fb1 fix: skipped deletes no longer cause waiting
- Added SkippedApplies and SkippedDeletes to the TaskContext
- Modified tasks to use the new skipped tracking, replacing usage
  of failure tracking, where skipped is more accurate.
- Renamed some TaskContext methods for consistency
- Added ObjMetadataSetFromMap for use by TaskContext
- Added ObjectMetadataSet.Intersection for use by InvSetTask
- Cleaned up InvSetTask to be more readable with comments explaining
  intended behavior, including handling of skips.
- Added apply and prune tests for skipped, failure, and abandoned
2021-10-29 10:28:06 -07:00
Morten Torkildsen 73dca2b0dc Update to latest version of kyaml and k8s libs 2021-10-19 17:41:57 -07:00
Mikhail Mazurskiy 7e47e6bd0e
Check exact version when determining scope 2021-10-15 12:36:25 +11:00
Mikhail Mazurskiy 9f6552d595
Check versions a CRD defines 2021-10-15 12:36:23 +11:00
Mikhail Mazurskiy 7333873c91
Drop k8s.io/apiextensions-apiserver dependency 2021-10-11 21:27:48 +11:00
Karl Isenberg 604c9deb8b Add UnstructuredSet to encapsulate set functions
- Refactor usages to []Unstructured to use UnstructuredSet
- Add UnstructuredSet.Equal to handle set equality comparison
- Avoid updating tests to prove reverse compatibility
  (except: Graph.SortObj, which returns a list of sets)
2021-10-08 13:45:46 -07:00
Karl Isenberg d83ce93efd Add ObjMetadataSet to encapsulate set functions
- Refactor usages of []ObjMetadata to use ObjMetadataSet
- Move Union, Diff, Contains, Hash, Remove, and Equal into
  ObjMetadataSet
- Add ToStringMap and FromStringMap for inventory serialization
2021-10-06 14:25:12 -07:00
Karl Isenberg 3d81e21acf e2e test cleanup
- Unstructured() added to ResourceReference to aid test operations
- Test artifacts moved into artifacts_test.go
- deploymentManifest and apiserviceManifest replaced with
  yaml->unstructured, for consistency with other test artifacts
- Added withReplicas, withNamespace, and withDependsOn for easier inline
  mutation of test artifacts
- Added deleteUnstructuredAndWait, assertUnstructuredExists, and
  assertUnstructuredDoesNotExist to validate actual cluster state,
  not just events. Also reduced flakiness by waiting for delete.
- Removed error check before event validation. The event diff
  provides much more context for debugging.
- Replaced typed resources with unstructured and used
  testutil.NestedField to extract values.
- Updated prune error test to use full event list comparison and
  validate server state.
- Add 2nd resource to continue on error test
- Don't fail early on error from event list
- Fix flakey prune error test
  Add wait for pod2 after creation
  Add wait for pod2 after deletion
  Re-use inventory to ensure prune of pod2 not skipped
- Add optional klog support for e2e tests
2021-09-24 17:54:17 -07:00
Karl Isenberg 4f4bd30c57 Change YamlStringer to only accept Unstructured
- This avoids a runtime schema dependency
2021-09-24 10:35:28 -07:00
Karl Isenberg a468a88337 Unmarshal with k8s.io/apimachinery/pkg/util/yaml
- apimachinery yaml.Unmarshal fixes number types to int64 and float64
- Fix inaccurate error message in mutator
- Move YamlStringer to object pkg for reuse by other pkgs
- Fix jsonpath tests parsing y as a bool
- Fix kstatus example_test.go to actually be a test
2021-09-22 16:33:06 -07:00
Karl Isenberg 02c0fb7229 Add apply-time-mutation feature
- Detect `config.kubernetes.io/apply-time-mutation` annotation
- Parse annotation string value as YAML
- Treat source resource as a dependency
- Before applying, apply specified substitutions
- Each mutation may include one or more substitution
- Substitutions may optionally replace a token in the existing
  string value, or replace the whole value (e.g. non-strings)
- Source and target fields are specified with JSONPath expressions
- Using github.com/spyzhov/ajson because it supports mutation, and
  not just retrieval
- ApplyTimeMutator uses an in-memory ResourceCache to reduce GETs
2021-09-21 11:00:37 -07:00
Karl Isenberg 15ef662b14 Rename Marshal/Unmarshal to Format/Parse 2021-09-13 13:43:24 -07:00
Karl Isenberg 325537204d Refactor depends-on code
- Move from pkg/object to pkg/object/dependson
- Add DependencySet type to abstract some complexity
- Add Marshal/Unmarshal funcs for ObjMetadata and DependencySet
- Refactor primary API to ReadAnnotation and WriteAnnotation
- Move Write/Marshal behavior from testutil into dependson pkg
- Make constants private to encourage using functions as primary API
2021-09-13 13:20:29 -07:00
Sean Sullivan 769f5bd735 Expose cyclic dependency error 2021-08-23 11:24:05 -07:00
Mikhail Mazurskiy 4b8dd62c53
Remove strings.TrimSpace()
Names and labels cannot have any whitespace. If they do, it's an error that should be reported and fixed.
2021-08-08 11:29:43 +10:00
Mikhail Mazurskiy 275f2c27e0
Drop github.com/go-errors/errors dependency 2021-07-02 10:53:03 +10:00
Morten Torkildsen 136d1efeeb Better validation of resources prior to actuation 2021-07-01 17:02:57 -07:00
Sean Sullivan bf6bc5f346 Adds object dependency sorting and tests 2021-06-25 20:01:59 -07:00
Sean Sullivan 5b31557b7e Re-organizes pkg/object, adds annotation helpers 2021-06-23 15:49:25 -07:00
Sean Sullivan fcd4cffa80 Object graph functionality 2021-06-07 20:53:50 -07:00
Mikhail Mazurskiy 3514065221
Simplify: remove unneeded meta.Accessor() usage 2021-05-28 15:03:09 +10:00
Mikhail Mazurskiy a0d1024504
Reduce allocations in set operations 2021-05-20 15:46:24 +10:00
Morten Torkildsen ae80e561e2 Improve the event hierarchy 2021-05-18 14:58:41 -07:00
Sean Sullivan 5af6137d68 Remove special-case object name validation 2021-02-02 10:31:53 -08:00
Jingfang Liu 5d8fb759a0 update events to include indentifier and error 2020-11-17 10:07:50 -08:00
Morten Torkildsen 3ac57786f6 Update the ManifestReader interface and implementations to use Unstructured 2020-10-26 12:13:44 -07:00
Morten Torkildsen 507d11f31e Use Unstructured instead of Info 2020-10-20 15:38:13 -07:00
Sean Sullivan 2de4d86359 Applies inventory namespace first if necessary 2020-09-11 16:11:50 -07:00
Sean Sullivan 9d0ec2cd71 Allow colon in the name of RBAC resources 2020-08-19 12:14:28 -07:00