* Update API definition to support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Mark pod template field as schemaless
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add kubebuilder marker to preserve unknown fields
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add example for using pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
* Support pod template
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Still working on tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Maybe progress
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* First working validation
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Lots of cleanup needed but it actually reproduced
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Working but ugly get schema from CRD
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Satisfactory test
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add missing omitempty for optional values
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove omitempty on required fields
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Run update-crd
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove temp schema config
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Tidy
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* go import
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Cover more test cases
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests that spec and metadata are required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add tests against error content
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* `go mod tidy`
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Fix lint
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Remove test - hopefully we can add a better test as a follow up
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Make `mainApplicationFile` required
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Regenerated api-docs
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* Add `MainApplicationFile` in tests
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
---------
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: support driver and executor pod use different priority
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: merge the logic of setPodPriorityClassName into addPriorityClassName
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: add adjust pointer if is nil
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* feat: Optimize code to avoid null pointer exceptions
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove backup crd files
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
* fix: remove BatchSchedulerOptions.PriorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
* fix: add driver and executor pod priorityClassName test code
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
---------
Signed-off-by: Kevin Wu <kevin.wu@momenta.ai>
Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai>
Co-authored-by: Kevin Wu <kevin.wu@momenta.ai>
* Update .helmignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add release docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update release workflow
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update integration test workflow
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add workflow for pushing tag when VERSION file changes
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update
Signed-off-by: Yi Chen <github@chenyicn.net>
* Remove the leading 'v' from chart version
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update docker image tags
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update .gitignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update .dockerignore
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update Makefile
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the process to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update the workflow to generate api docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Use controller-gen to generate CRD and deep copy related methods
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update helm chart CRDs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update workflow for building spark operator
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update README.md
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update docs
Signed-off-by: Yi Chen <github@chenyicn.net>
* Remove docs and update README
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add link to monthly community meeting
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Add driverIngressOptions in SparkApplication CRD
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update chart version to 1.3.0
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update helm chart README
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Fix make detect-crds-drift
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Update api-docs.md
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
---------
Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
* Add emptyDir sizeLimit support
Signed-off-by: Jacob Salway <jacob.salway@rokt.com>
* Bump appVersion and add sizeLimit example
Signed-off-by: Jacob Salway <jacob.salway@rokt.com>
---------
Signed-off-by: Jacob Salway <jacob.salway@rokt.com>
* Add helm unit tests
Signed-off-by: Yi Chen <github@chenyicn.net>
* Fix: failed to render spark service account when extra annotations are specified
Signed-off-by: Yi Chen <github@chenyicn.net>
* Update developer guide
Signed-off-by: Yi Chen <github@chenyicn.net>
* Bump helm chart version
Signed-off-by: Yi Chen <github@chenyicn.net>
---------
Signed-off-by: Yi Chen <github@chenyicn.net>
* Updated default registry to ghcr.io
* Bumped chart version and built api-docs
* Updated build-helm-chart job, step Run chart-testing (install) to look for the correct registry
This change renames the github workflows to be clearer about their purpose, and adds a set of tests which
aim to force developers to increment the appVersion if they have changed anything with the spark-operator docker container
or the chart version if they have updated the chart.
Signed-off-by: Tom Hellier <me@tomhellier.com>
* Allow user to specify service annotation on Spark UI service
* Add service annotation in chart crd
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
* Bump chart version
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Co-authored-by: Khor Shu Heng <khor.heng@gojek.com>
* Updating Spark version from 3.0 to 3.1
* Adding github actions
* Add configmaps resources into spark-role
Spark 3.1 needs permission to create ConfigMaps, Pods and services
* Typo fix - Change unit test to unit tests
* Format fix on workflow
* Update main.yaml
* Update main.yaml
* Update Makefile and Bump Chart version
* Bump Chart version to 1.1.0
* Add Chart lint docs
Yi Chen
Thomas Newton
bo
Kevinz
Yuri Niitsuma
Bo Yang
Matthew Rossi
Cian Gallagher
Jacob Salway
Duc Nguyen
Zev Isert
Cian Gallagher
shahsiddharth08
Taeyeop Kim
Jacob Salway
Volodymyr Kot
Jose Soto
ordukhanian
JunaidChaudry
Alexander Ryabov
Mat Schaffer
Ashish Pushp
André Claudino
Harry Su
David Dillon
bhargav joshi
Andrei Neagoe
TomHellier
Hoa Le
Mark Schroering
Juan David Correa
Beni Ben zikry
william-wang
Khor Shu Heng
Sébastien Maintrot
Run Lin
Luiz Muller
lomo
Marco Zoveralli
ChethanUK
Duyet Le