* Add support for using cert manager to generate webhook certificates
Signed-off-by: Yi Chen <github@chenyicn.net>
* update certificate provider unit tests
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add a newline at the end of file
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add examples for configuring duration and renewBefore
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add timeZone to ScheduledSparkApplication
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
* Update api/v1beta2/scheduledsparkapplication_types.go
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
---------
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
* Add option for disabling leader election
Signed-off-by: Yi Chen <github@chenyicn.net>
* Remove related RBAC rules when disabling leader election
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* chore: add helm unittest step to integration test workflow
Signed-off-by: Yi Chen <github@chenyicn.net>
* fix: contrainer security context unit test
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Retry after driver pod now found if recent submission
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add a test
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Make grace period configurable
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Update test
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add an extra test with the driver pod
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Separate context to create and delete the driver pod
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Tidy
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Autoformat
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Update error message
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add helm paramater
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Update internal/controller/sparkapplication/controller.go
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Newlines between helm tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
---------
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
* create a tmp dir for the controller to write Spark artifacts to and set the controller to readOnlyRootFilesystem
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* mount a dir for the webhook container to generate its certificates in and set readOnlyRootFilesystem: true for the webhook pod
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* update the securityContext in the controller deployment test
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* update securityContext of the webhook container in the deployment_test
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* update README
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* remove -- so comments are not rendered in the README.md
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* recreate README.md after removal of comments for volumes and volumeMounts
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* make indentation for volumes and volumeMounts consistent with rest of values.yaml
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* Revert "make indentation for volumes and volumeMounts consistent with rest of values.yaml"
This reverts commit dba97fc3d9458e5addfff79d021d23b30938cbb9.
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* fix indentation in webhook and controller deployment templates for volumes and volumeMounts
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
* Update charts/spark-operator-chart/values.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/values.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/values.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/values.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/templates/controller/deployment.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/templates/controller/deployment.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/templates/webhook/deployment.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* Update charts/spark-operator-chart/templates/webhook/deployment.yaml
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
* add additional securityContext to the controller deployment_test.yaml
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
---------
Signed-off-by: Nick Gretzon <npgretz@gmail.com>
Signed-off-by: Nicholas Gretzon <50811947+npgretz@users.noreply.github.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
* Update API definition to support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Mark pod template field as schemaless
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add kubebuilder marker to preserve unknown fields
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add example for using pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update default container security context
Signed-off-by: Yi Chen <github@chenyicn.net>
* Push user and group directives into Dockerfile
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add allowPrivilegeEscalation to container security context
Signed-off-by: Yi Chen <github@chenyicn.net>
* fix: fsGroup should be moved to pod security context
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Still working on tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Maybe progress
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* First working validation
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Lots of cleanup needed but it actually reproduced
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Working but ugly get schema from CRD
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Satisfactory test
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add missing omitempty for optional values
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove omitempty on required fields
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Run update-crd
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove temp schema config
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Tidy
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* go import
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Cover more test cases
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests that spec and metadata are required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests against error content
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* `go mod tidy`
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Fix lint
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove test - hopefully we can add a better test as a follow up
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Make `mainApplicationFile` required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Regenerated api-docs
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add `MainApplicationFile` in tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
---------
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* implement an upper bound limit to the number of tracked executor
Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>
* add upper bound limit to the number of tracked executor to helm chart
Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>
---------
Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: add adjust pointer if is nil
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: Optimize code to avoid null pointer exceptions
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove backup crd files
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove BatchSchedulerOptions.PriorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* fix: add driver and executor pod priorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
---------
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
Co-authored-by: Kevin Wu <kevin.wu@momenta.ai>
* Support coscheduling with kube-scheduler plugins
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add example for using kube-schulder coscheduling
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add default batch scheduler argument
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
* Add helm unit test
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
---------
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
* Update .helmignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add release docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update release workflow
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update integration test workflow
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add workflow for pushing tag when VERSION file changes
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update
Signed-off-by: Yi Chen <github@chenyicn.net>
* Remove the leading 'v' from chart version
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update docker image tags
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update .gitignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update .dockerignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update Makefile
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the process to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the workflow to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Use controller-gen to generate CRD and deep copy related methods
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update helm chart CRDs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update workflow for building spark operator
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update README.md
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* feat: give an option to set the priority class for spark-operator pod
Signed-off-by: Praveen Gajulapalli <13733716+pkgajulapalli@users.noreply.github.com>
* feat: bumped up helm chart version
Signed-off-by: Praveen Gajulapalli <13733716+pkgajulapalli@users.noreply.github.com>
* fix: fixed issue with position of priorityClassName
Signed-off-by: Praveen Gajulapalli <13733716+pkgajulapalli@users.noreply.github.com>
* Add restartPolicy field to SparkApplication CRDs
Signed-off-by: Mark Schroering <mark.schroering@gmail.com>
* Add restartPolicy field to SparkApplication CRDs
Signed-off-by: Mark Schroering <mark.schroering@gmail.com>
* Add restartPolicy field to SparkApplication CRDs
Signed-off-by: Mark Schroering <mark.schroering@gmail.com>
---------
Signed-off-by: Mark Schroering <mark.schroering@gmail.com>
Yi Chen
jbhalodia-slack
Daniel Freitas
Jacob Salway
Vikas Saxena
Manabu McCloskey
Tarek Abouzeid
Thomas Newton
Nicholas Gretzon
Aran Shavit
Roberto Devesa
bo
tcassaert
Sébastien Maintrot
Kevinz
Carlos Sánchez Páez
Cian Gallagher
Praveen Gajulapalli
Mark Schroering
Aakcht