0c2f973883
Added basic jsonschema validation for SparkApplication
2018-02-06 21:51:06 -08:00
3bb2168a5d
Use the cache store instead of using the runningApp
2018-02-06 11:55:51 -08:00
5887d27deb
Simplified the handling of pod state updates
...
Combined communicating driver and executor pod updates into a single channel.
2018-02-01 14:10:57 -08:00
fd23d8443b
Do not wait for application to complete
2018-02-01 09:38:38 -08:00
542a13854f
Delete the CRD definition when the operator is stopped
...
When the operator is stopped and started, the CRD definition stays the same. There is a problem with this: changes to the CRD definition won't take effect if the definition is not updated to the API server. Deleting the definition when the operator stops and recreating it when the operator starts effectively updates the definition.
2018-01-29 15:05:44 -08:00
2520f0547e
Ran go fmt on code
2018-01-29 12:54:47 -08:00
165428d795
Added a field for specifying the init-container image
2018-01-29 10:15:37 -08:00
313a45e910
Added support for more Spark configuration properties
2018-01-29 10:00:37 -08:00
c3b21ff95f
Added support for core limits, labels, and annotations
2018-01-26 17:03:07 -08:00
3373b30e6d
Added support for uploading local dependencies to GCS
2018-01-25 10:23:08 -08:00
b7d3401df5
Set application state based on driver pod phase
...
Unless submission fails, in which case the application state is set to "SUBMISSION_FAILED".
2018-01-23 21:51:42 -08:00
7d3aa7a39e
Updated to support the single image
2018-01-23 10:41:43 -08:00
fa9483358c
Fixed import order
2018-01-21 14:22:56 -08:00
f2c15ee34f
Better handle crashes in the components
2018-01-19 14:40:25 -08:00
d1f3b3a48e
Fixed a bug in adding configmap mounting annotations
2018-01-19 14:18:01 -08:00
0eae0ee94b
Some refactoring to controller.go
2018-01-16 10:58:10 -08:00
a1d8356b02
Switched to use a work queue in the crd controller
2018-01-16 10:36:20 -08:00
321d37dc0d
Refactored the controller code
2018-01-15 22:06:12 -08:00
8c16ca8c82
Use versioned pod client to list and watch pods
2018-01-14 21:26:54 -08:00
e021f07993
Switched to use auto-generated clientset
2018-01-14 12:36:00 -08:00
d36b30ceac
Simplified the way the UI service is named
2018-01-12 15:24:14 -08:00
3dd2de685b
Support application restart with a configurable restart policy
2018-01-12 15:07:12 -08:00
cd7df2b5f8
Added auto generated clientset, informers, and listers
2018-01-10 21:49:50 -08:00
6d88b9bd75
Made the unit tests to use a fake event recorder
2018-01-10 13:22:04 -08:00
ef75f05b5d
Added Kubernetes event recording for SparkApplications
2018-01-10 11:24:24 -08:00
a2882f8a3b
Simplified names of some fields of SparkApplicationSpec
2018-01-08 15:49:53 -08:00
76e5bff5fa
Do not use shared informer in the initializer and pod monitor
...
This is because we may move to using an admission webhook to replace the initializer in the future and also because the two are using different ListWatch anyway.
2018-01-08 14:19:15 -08:00
28e4c710ef
Wait for caches to sync
2018-01-08 14:00:42 -08:00
11edfadd57
Used shared pod informer in the initializer and spark pod monitor
2018-01-08 12:43:34 -08:00
bd24ec6b48
Migrated to base off Spark 2.3
2018-01-03 16:20:25 -08:00
03bffd6ca2
Fixed build
2018-01-03 15:01:30 -08:00
552ad22601
Set submission time and completion time
2018-01-03 09:41:42 -08:00
b8a58ab9c4
Refactored types.go
2018-01-03 09:41:29 -08:00
7691e7a61f
Implement handling of update of SparkApplication
2018-01-03 09:41:16 -08:00
14da4914d7
Added CONTRIBUTING.md and license headers
2017-12-18 11:31:49 -08:00
4bf598a253
More unit tests and fixed the nodeport used for the driver UI service
2017-12-18 10:32:37 -08:00
7900fb5793
Added more unit tests for the controller
2017-12-15 16:29:25 -08:00
7339d7530f
Added a new field for supporting in-cluster client mode
2017-12-15 10:44:10 -08:00
e27dd1d642
Initial unit tests for the crd controller
2017-12-06 21:29:08 -08:00
febc11392b
Refactored some names to not with prefixes the same as package names
2017-12-06 10:46:54 -08:00
9fd90f8d30
Separate Hadoop config properties out into a field of their own
2017-11-30 11:10:37 -08:00
5b20544aaf
Moved structs not supposed to be exposed to be package private
2017-11-21 13:13:42 -08:00
b89165a75b
Retry failed updates to the API server
2017-11-21 10:39:41 -08:00
6ff5ccc9c2
Avoid updating SparkApplication when nothing gets changed
2017-11-20 21:42:27 -08:00
0b6f97f619
Bug fixes around SparkApplication updating
2017-11-17 16:51:52 -08:00
ce89ef98cd
Fixed a bug on use of files nad pyFiles
2017-11-17 10:46:34 -08:00
4c28131091
Added missing code for mounting general ConfigMaps
2017-11-16 12:01:42 -08:00
e29daee845
Code refactoring to make the naming consistent and clear
2017-11-16 11:12:03 -08:00
3ba82a8682
Update application state based on the driver state
2017-11-15 13:32:02 -08:00
5a5f90d386
Added fields for specifying driver/executor cores and memory
2017-11-13 11:50:19 -08:00
73f55f6fe1
Set URL of the Spark web UI
2017-11-02 15:51:14 -07:00
8e7ce966d3
Added a flag for number of submission runner threads
2017-11-02 11:45:18 -07:00
b8f173b3ce
Delete the driver and its service when the crd is deleted
2017-11-02 11:33:06 -07:00
3fc215c089
Close the newly added channel for driver state updates
2017-11-02 10:59:53 -07:00
fe84083d03
Added reporting of driver state
2017-11-02 10:53:09 -07:00
a6ef7a1662
Added executor state updating
2017-11-01 22:51:03 -07:00
81bdaf7f24
Record application error message in SparkApplicationStatus
2017-11-01 12:14:46 -07:00
a41b8273e3
Added application state reporting back to controller
2017-10-31 16:00:25 -07:00
47477807f4
Restored creation of the UI service
2017-10-31 11:51:20 -07:00
af1dcf16b4
Fixed and maked application submission work
2017-10-31 10:09:58 -07:00
a9f10258d9
Removed unused code
...
And added app id label to driver/executor pods and ui service.
2017-10-05 10:28:48 -07:00
20a09dab1a
Added more unit tests and dependency testify/assert
2017-10-05 09:31:26 -07:00
30cdb078b9
Added a runner for running spark-submit commands
2017-10-04 14:21:05 -07:00
d19d40727c
Use spark-base as the base image
2017-09-28 22:41:29 -07:00
bf3a47925f
Upated the type definition to include driver and executor images
2017-09-25 10:11:19 -07:00
286d5966ec
Make sure the UI service gets deleted when a SparkApplication is deleted
2017-09-22 12:05:23 -07:00
d335b137dc
Completed the submission command construction
2017-09-22 11:54:25 -07:00
2aed58a995
Fixed crd controller logging and added an example crd yaml
2017-09-22 10:29:36 -07:00
619131f2a0
Added deepcopy code-gen and updated dependency versions
2017-09-20 08:54:51 -07:00
55ffb8080f
Fixed the build of the crd controller
2017-09-15 15:04:05 -07:00
93ac1c98f6
Fixed main.go and the use of stop channel
2017-08-31 12:08:37 -07:00
b941255f3c
Added some more unit tests for the initializer
2017-08-29 22:16:29 -07:00
1b01ed0fdd
Added more annotation definitions
2017-08-29 16:23:13 -07:00
9fa0f2baad
More refactoring of types.go and pkg/controller
2017-08-28 15:32:15 -07:00
4cf6191a8d
Added some more status fields for application and executor state
2017-08-25 16:15:44 -07:00
ea16bd2636
Added code for creating a service for the Spark UI
2017-08-24 08:55:04 -07:00
f315571870
Refactored the type definition
2017-08-22 08:54:53 -07:00
d7fcf86a86
Added code for mounting configmaps in the initializer
2017-08-18 16:32:18 -07:00
3692f32d48
Added doc.go for all packages
2017-08-17 09:07:59 -07:00
c49e4559f2
Added code for creating ConfigMap for Spark and Hadoop conf dirs
2017-08-17 09:07:51 -07:00
fc4bd05db5
Added initial implementation of API types, client, and controllers
2017-08-15 23:33:30 -07:00
Yinan Li
Yinan Li