Commit Graph

10 Commits

Author SHA1 Message Date
jbhalodia-slack ca37f6b7b3
Add ShuffleTrackingEnabled to DynamicAllocation struct to allow disabling shuffle tracking (#2511)
* Add ShuffleTrackingEnabled *bool to DynamicAllocation struct to allow disabling shuffle tracking

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Run make generate

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* make manifests

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* make update-crd && make build-api-docs

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Update internal/controller/sparkapplication/submission.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Go fmt

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Refactor defaultExecutorSpec func

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Refactor dynamicAllocationOption func

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Add IsDynamicAllocationEnabled func

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

---------

Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
2025-05-14 05:49:22 +00:00
Tan Qi 68e6c1d586
change env in executorSecretOption (#2467)
* change env in executorSecretOption

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

* Use spark.executorEnv instead

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

* Remove V2 and update SparkExecutorEnvTemplate

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

---------

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>
2025-03-20 02:01:20 +00:00
jbhalodia-slack d10b8f5f3a
Make image optional (#2439)
* Make app.Spec.Driver.Image and app.Spec.Image optional
Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Make app.Spec.Executor.Image optional
Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>
2025-02-20 04:03:42 +00:00
Cian (Keen) Gallagher 2999546dc6
Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316)
Signed-off-by: Cian Gallagher <cian@ciangallagher.net>
2024-11-11 02:13:15 +00:00
Yi Chen 763682dfe6
Fix: should not add emptyDir sizeLimit conf if it is nil (#2305)
Signed-off-by: Yi Chen <github@chenyicn.net>
2024-11-04 11:17:15 +00:00
Yi Chen d0daf2fd17
Support pod template for Spark 3.x applications (#2141)
* Update API definition to support pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

* Mark pod template field as schemaless

Signed-off-by: Yi Chen <github@chenyicn.net>

* Add kubebuilder marker to preserve unknown fields

Signed-off-by: Yi Chen <github@chenyicn.net>

* Add example for using pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

* Support pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
2024-10-24 02:23:30 +00:00
Thomas Newton 735c7fc9e5
Fix retries (#2241)
* Attempt to requeue after correct period

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Syntactically correct

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* I think correct requeueing

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Same treatment for the other retries

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Requeue after deleting resources

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Try to fix submission status updates

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Correct usage of submitSparkApplication

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix error logging

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Bring back ExecutionAttempts increment that I forgot about

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Log after reconcile complete

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix setting submission ID

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy logging

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update comment

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Start a new test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Working Fails submission and retries until retries are exhausted test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add Application fails and retries until retries are exhausted

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Comments

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Move fail configs out of the examples directory

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix lint

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Move TimeUntilNextRetryDue to `pkg/util/sparkapplication.go`

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update internal/controller/sparkapplication/controller.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update test/e2e/sparkapplication_test.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* camelCase

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* make fo-fmt

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* PR comments

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

---------

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
2024-10-23 13:13:30 +00:00
bo a1de26dd31
feat: support archives param for spark-submit (#2256)
Signed-off-by: kaka-zb <sin19990111@gmail.com>
2024-10-16 06:20:18 +00:00
Nick Tan 7fb14e629e
fix: imagePullPolicy was ignored (#2222)
Signed-off-by: xuqingtan <missedone@gmail.com>
2024-10-08 02:41:07 +00:00
Yi Chen 0dc641bd1d
Use controller-runtime to reconsturct spark operator (#2072)
* Use controller-runtime to reconstruct spark operator

Signed-off-by: Yi Chen <github@chenyicn.net>

* Update helm charts

Signed-off-by: Yi Chen <github@chenyicn.net>

* Update examples

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
2024-08-01 12:29:06 +00:00