kubeflow
/
spark-operator
mirror of https://github.com/kubeflow/spark-operator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,097 Commits 14 Branches 104 Tags 64 MiB
Commit Graph

24 Commits

Author SHA1 Message Date
Yi Chen 12b97522ab Update imports 
Signed-off-by: Yi Chen <github@chenyicn.net>
 2025-04-27 11:25:21 +08:00
Tan Qi 68e6c1d586
change env in executorSecretOption (#2467) 
* change env in executorSecretOption

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

* Use spark.executorEnv instead

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

* Remove V2 and update SparkExecutorEnvTemplate

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>

---------

Signed-off-by: Qi Tan <16416018+TQJADE@users.noreply.github.com>
 2025-03-20 02:01:20 +00:00
jbhalodia-slack d10b8f5f3a
Make image optional (#2439) 
* Make app.Spec.Driver.Image and app.Spec.Image optional
Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>

* Make app.Spec.Executor.Image optional
Signed-off-by: jbhalodia-slack <jbhalodia@salesforce.com>
 2025-02-20 04:03:42 +00:00
Jacob Salway 25ca90cb07
Support Kubernetes 1.32 (#2416) 
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
Signed-off-by: Jacob Salway <jacob.salway@rokt.com>
 2025-02-12 12:02:29 +00:00
Manabu McCloskey 2995a0a963
ensure passed context is used (#2432) 
Signed-off-by: Manabu McCloskey <manabu.mccloskey@gmail.com>
 2025-02-12 08:44:29 +00:00
Thomas Newton d815e78c21
Robustness to driver pod taking time to create (#2315) 
* Retry after driver pod now found if recent submission

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add a test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Make grace period configurable

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add an extra test with the driver pod

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Separate context to create and delete the driver pod

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Autoformat

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update error message

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add helm paramater

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update internal/controller/sparkapplication/controller.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Newlines between helm tests

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

---------

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
 2024-12-04 12:58:59 +00:00
Cian (Keen) Gallagher 2999546dc6
Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316) 
Signed-off-by: Cian Gallagher <cian@ciangallagher.net>
 2024-11-11 02:13:15 +00:00
Yi Chen 763682dfe6
Fix: should not add emptyDir sizeLimit conf if it is nil (#2305) 
Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-11-04 11:17:15 +00:00
Yi Chen d0daf2fd17
Support pod template for Spark 3.x applications (#2141) 
* Update API definition to support pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

* Mark pod template field as schemaless

Signed-off-by: Yi Chen <github@chenyicn.net>

* Add kubebuilder marker to preserve unknown fields

Signed-off-by: Yi Chen <github@chenyicn.net>

* Add example for using pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

* Support pod template

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-10-24 02:23:30 +00:00
Thomas Newton 735c7fc9e5
Fix retries (#2241) 
* Attempt to requeue after correct period

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Syntactically correct

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* I think correct requeueing

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Same treatment for the other retries

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Requeue after deleting resources

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Try to fix submission status updates

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Correct usage of submitSparkApplication

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix error logging

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Bring back ExecutionAttempts increment that I forgot about

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Log after reconcile complete

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix setting submission ID

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy logging

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update comment

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Start a new test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Working Fails submission and retries until retries are exhausted test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add Application fails and retries until retries are exhausted

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Comments

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Move fail configs out of the examples directory

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix lint

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Move TimeUntilNextRetryDue to `pkg/util/sparkapplication.go`

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update internal/controller/sparkapplication/controller.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Update test/e2e/sparkapplication_test.go

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* camelCase

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* make fo-fmt

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* PR comments

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

---------

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
 2024-10-23 13:13:30 +00:00
Thomas Newton 5ff8dcf350
`omitempty` corrections (#2255) 
* Still working on tests

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Maybe progress

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* First working validation

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Lots of cleanup needed but it actually reproduced

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Working but ugly get schema from CRD

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Satisfactory test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add missing omitempty for optional values

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Remove omitempty on required fields

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Run update-crd

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Remove temp schema config

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Tidy

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* go import

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Cover more test cases

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add tests that spec and metadata are required

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add tests against error content

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* `go mod tidy`

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix lint

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Remove test - hopefully we can add a better test as a follow up

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Make `mainApplicationFile` required

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Regenerated api-docs

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Add `MainApplicationFile` in tests

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

---------

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
 2024-10-18 11:06:20 +00:00
bo a1de26dd31
feat: support archives param for spark-submit (#2256) 
Signed-off-by: kaka-zb <sin19990111@gmail.com>
 2024-10-16 06:20:18 +00:00
Yi Chen 3acd0f1a90
remove redundant test.sh file (#2243) 
* Fix go lint error

Signed-off-by: Yi Chen <github@chenyicn.net>

* Remove test.sh

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-10-14 01:23:16 +00:00
Jacob Salway 718e2444a4
Upgrade to Spark 3.5.3 (#2202) 
* Upgrade to Spark 3.5.3

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Check result of err

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

---------

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
 2024-10-13 15:43:15 +00:00
Sébastien Maintrot a8b5d644b5
implement an upper bound limit to the number of tracked executor (#2181) 
* implement an upper bound limit to the number of tracked executor

Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>

* add upper bound limit to the number of tracked executor to helm chart

Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>

---------

Signed-off-by: ImpSy <3097030+ImpSy@users.noreply.github.com>
 2024-10-11 05:54:10 +00:00
Nick Tan 7fb14e629e
fix: imagePullPolicy was ignored (#2222) 
Signed-off-by: xuqingtan <missedone@gmail.com>
 2024-10-08 02:41:07 +00:00
Yi Chen c855ee4c8b
Fix: spark application does not respect time to live seconds (#2165) 
* Add time to live seconds example spark application

Signed-off-by: Yi Chen <github@chenyicn.net>

* fix: spark application does not respect time to live seconds

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-09-19 12:40:29 +00:00
tcassaert ed3226ebe7
Add specific error in log line when failed to create web UI service (#2170) 
* Add specific error in log line when failed to create web UI service

Signed-off-by: tcassaert <tcassaert@inuits.eu>

* Update log to reflect correct resource that could not be created

Co-authored-by: Yi Chen <github@chenyicn.net>
Signed-off-by: tcassaert <tcassaert@protonmail.com>

---------

Signed-off-by: tcassaert <tcassaert@inuits.eu>
Signed-off-by: tcassaert <tcassaert@protonmail.com>
Co-authored-by: Yi Chen <github@chenyicn.net>
 2024-09-19 08:11:28 +00:00
tcassaert eb48b349a1
fix: The logger had an odd number of arguments, making it panic (#2166) 
Signed-off-by: tcassaert <tcassaert@inuits.eu>
 2024-09-13 09:24:23 +00:00
Yi Chen e8d3de9e1a
Support extended kube-scheduler as batch scheduler (#2136) 
* Support coscheduling with kube-scheduler plugins

Signed-off-by: Yi Chen <github@chenyicn.net>

* Add example for using kube-schulder coscheduling

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-09-03 03:23:13 +00:00
Yi Chen c93b0ec0e7
Adding support for setting spark job namespaces to all namespaces (#2123) 
Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-09-03 02:55:14 +00:00
Jacob Salway 9cc1c02c64
Add default batch scheduler argument (#2143) 
* Add default batch scheduler argument

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Add helm unit test

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

---------

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
 2024-08-28 02:53:03 +00:00
Jacob Salway 8fcda12657
Support gang scheduling with Yunikorn (#2107) 
* Add Yunikorn scheduler and example

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Add test cases

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Add code comments

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Add license comment

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Inline mergeNodeSelector

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

* Fix initial number implementation

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>

---------

Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
 2024-08-22 04:15:57 +00:00
Yi Chen 0dc641bd1d
Use controller-runtime to reconsturct spark operator (#2072) 
* Use controller-runtime to reconstruct spark operator

Signed-off-by: Yi Chen <github@chenyicn.net>

* Update helm charts

Signed-off-by: Yi Chen <github@chenyicn.net>

* Update examples

Signed-off-by: Yi Chen <github@chenyicn.net>

---------

Signed-off-by: Yi Chen <github@chenyicn.net>
 2024-08-01 12:29:06 +00:00