* Add timeZone to ScheduledSparkApplication
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
* Update api/v1beta2/scheduledsparkapplication_types.go
Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
---------
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
* Update API definition to support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Mark pod template field as schemaless
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add kubebuilder marker to preserve unknown fields
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add example for using pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Still working on tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Maybe progress
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* First working validation
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Lots of cleanup needed but it actually reproduced
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Working but ugly get schema from CRD
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Satisfactory test
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add missing omitempty for optional values
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove omitempty on required fields
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Run update-crd
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove temp schema config
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Tidy
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* go import
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Cover more test cases
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests that spec and metadata are required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests against error content
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* `go mod tidy`
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Fix lint
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove test - hopefully we can add a better test as a follow up
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Make `mainApplicationFile` required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Regenerated api-docs
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add `MainApplicationFile` in tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
---------
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: add adjust pointer if is nil
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: Optimize code to avoid null pointer exceptions
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove backup crd files
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove BatchSchedulerOptions.PriorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* fix: add driver and executor pod priorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
---------
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
Co-authored-by: Kevin Wu <kevin.wu@momenta.ai>
* Update .gitignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update .dockerignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update Makefile
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the process to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the workflow to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Use controller-gen to generate CRD and deep copy related methods
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update helm chart CRDs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update workflow for building spark operator
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update README.md
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add driverIngressOptions in SparkApplication CRD
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update chart version to 1.3.0
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update helm chart README
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Fix make detect-crds-drift
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update api-docs.md
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
---------
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Updated default registry to ghcr.io
* Bumped chart version and built api-docs
* Updated build-helm-chart job, step Run chart-testing (install) to look for the correct registry
This change renames the github workflows to be clearer about their purpose, and adds a set of tests which
aim to force developers to increment the appVersion if they have changed anything with the spark-operator docker container
or the chart version if they have updated the chart.
Signed-off-by: Tom Hellier <me@tomhellier.com>
* Allow user to specify service annotation on Spark UI service
* Add service annotation in chart crd
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
* Bump chart version
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Co-authored-by: Khor Shu Heng <khor.heng@gojek.com>
Spark 3.0 will support specifying a k8s service account for the executor pods. This CL prepares the operator to support that in the upcoming Spark 3.0.0 release.
* support custom seriveport and targetport
* support customizing ingress object with specific port and annotations
* change expositionOptions into sparkUIOptions + code enhancement
* update return of functions getUIServicePort and getUITargetPort
* add fix for metricsProperties when HasPrometheusConfigFile is true.
* add new config MetricsPropertiesFile.
* add missing auto-generated code from privous PRs.
* fix monitoring_config_test.go test condition, redo the configmap logic in monitoring_config.go.
* redo the configmap & javaOption logic in monitoring_config.go.
* set back the configmap & javaOption logic in monitoring_config.go
* update log.
jbhalodia-slack
Daniel Freitas
Jacob Salway
Yi Chen
Thomas Newton
bo
Kevinz
Bo Yang
Cian Gallagher
Zev Isert
Cian Gallagher
ordukhanian
Andrei Neagoe
TomHellier
Khor Shu Heng
Sébastien Maintrot
Marco Zoveralli
Josh Soref
Luiz Muller
Pascal GILLET
Yinan Li
Abhilash Pallerlamudi
Ali Gouta
jinxingwang