spark-operator

Commit Graph

Author	SHA1	Message	Date
jbhalodia-slack	ca37f6b7b3	Add ShuffleTrackingEnabled to DynamicAllocation struct to allow disabling shuffle tracking (#2511 ) * Add ShuffleTrackingEnabled bool to DynamicAllocation struct to allow disabling shuffle tracking Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> Run make generate Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * make manifests Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * make update-crd && make build-api-docs Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * Update internal/controller/sparkapplication/submission.go Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * Go fmt Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * Refactor defaultExecutorSpec func Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * Refactor dynamicAllocationOption func Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> * Add IsDynamicAllocationEnabled func Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> --------- Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com> Co-authored-by: Yi Chen <github@chenyicn.net>	2025-05-14 05:49:22 +00:00
Daniel Freitas	5a97ca4daa	Enable the override of MemoryLimit through webhook (#2478 ) * Documentation and interface definition Signed-off-by: danielrs <danielrs@ibm.com> * addMemoryLimit and convertion methods Signed-off-by: danielrs <danielrs@ibm.com> * Unit tests Signed-off-by: danielrs <danielrs@ibm.com> * Deepcopy Signed-off-by: danielrs <danielrs@ibm.com> * Adjustments after make command Signed-off-by: danielrs <danielrs@ibm.com> * Address comments Signed-off-by: danielrs <danielrs@ibm.com> --------- Signed-off-by: danielrs <danielrs@ibm.com> Co-authored-by: danielrs <danielrs@ibm.com>	2025-04-21 07:12:26 +00:00
Jacob Salway	50ae7a0062	Add timeZone to ScheduledSparkApplication (#2471 ) * Add timeZone to ScheduledSparkApplication Signed-off-by: Jacob Salway <jacob.salway@gmail.com> * Update api/v1beta2/scheduledsparkapplication_types.go Co-authored-by: Yi Chen <github@chenyicn.net> Signed-off-by: Jacob Salway <jacob.salway@gmail.com> --------- Signed-off-by: Jacob Salway <jacob.salway@gmail.com> Co-authored-by: Yi Chen <github@chenyicn.net>	2025-03-31 02:12:30 +00:00
Yi Chen	d0daf2fd17	Support pod template for Spark 3.x applications (#2141 ) * Update API definition to support pod template Signed-off-by: Yi Chen <github@chenyicn.net> * Mark pod template field as schemaless Signed-off-by: Yi Chen <github@chenyicn.net> * Add kubebuilder marker to preserve unknown fields Signed-off-by: Yi Chen <github@chenyicn.net> * Add example for using pod template Signed-off-by: Yi Chen <github@chenyicn.net> * Support pod template Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net>	2024-10-24 02:23:30 +00:00
Thomas Newton	5ff8dcf350	`omitempty` corrections (#2255 ) * Still working on tests Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Maybe progress Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * First working validation Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Lots of cleanup needed but it actually reproduced Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Working but ugly get schema from CRD Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Satisfactory test Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add missing omitempty for optional values Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Remove omitempty on required fields Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Run update-crd Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Remove temp schema config Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Tidy Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * go import Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Cover more test cases Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add tests that spec and metadata are required Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add tests against error content Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * `go mod tidy` Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Fix lint Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Remove test - hopefully we can add a better test as a follow up Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Make `mainApplicationFile` required Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Regenerated api-docs Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> * Add `MainApplicationFile` in tests Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com> --------- Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>	2024-10-18 11:06:20 +00:00
bo	a1de26dd31	feat: support archives param for spark-submit (#2256 ) Signed-off-by: kaka-zb <sin19990111@gmail.com>	2024-10-16 06:20:18 +00:00
Yi Chen	143b16ee75	Update integration test workflow and add golangci lint check (#2197 ) * Update integration test workflow Signed-off-by: Yi Chen <github@chenyicn.net> * Update golangci lint config Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net>	2024-10-08 02:06:07 +00:00
Kevinz	6ae1b2f69c	feat: support driver and executor pod use different priority (#2146 ) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: if app.Spec.Driver.PriorityClassName and app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: if app.Spec.Driver.PriorityClassName and app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove backup crd files Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> --------- Signed-off-by: Kevin Wu <kevin.wu@momenta.ai> Signed-off-by: Kevin.Wu <kevin.wu@momenta.ai> Co-authored-by: Kevin Wu <kevin.wu@momenta.ai>	2024-09-10 06:27:19 +00:00
Yi Chen	0dc641bd1d	Use controller-runtime to reconsturct spark operator (#2072 ) * Use controller-runtime to reconstruct spark operator Signed-off-by: Yi Chen <github@chenyicn.net> * Update helm charts Signed-off-by: Yi Chen <github@chenyicn.net> * Update examples Signed-off-by: Yi Chen <github@chenyicn.net> --------- Signed-off-by: Yi Chen <github@chenyicn.net>	2024-08-01 12:29:06 +00:00

9 Commits