kubeflow
/
spark-operator
mirror of https://github.com/kubeflow/spark-operator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
710 Commits 11 Branches 103 Tags 62 MiB
Commit Graph

37 Commits

Author SHA1 Message Date
Yinan Li 7a34c91f6c Use a Kubernetes Job to run spark-submit for multi-version support 2020-04-18 11:03:26 -07:00
Yinan Li 1c4486cdda
Removed legacy init-container related fields (#750) 2019-12-20 10:08:05 -08:00
Jiaxin Shan 0bd7592035 Add volumes support for Spark scratch space spark.local.dir (#707) 2019-12-15 15:05:40 -08:00
Yinan Li baafaafeea
Fixed the client version used in sparkctl (#681) 2019-11-03 18:50:21 -08:00
kevin hogeland 55a1eebc0c Generate CRD specs, bump to v1beta2 (#578) 
* Generate CRD specs, bump to v1beta2

* Add short/singular CRD names

* Merge upstream/master

* Tweak Cores validation

* Fix typo, merge upstream

* Update remaining docs for v1beta2
 2019-09-13 10:37:21 -07:00
Chaoran Yu 6609df0a18 Refactored integration tests (#445) 
* Refactored integration tests

* Addressed review comments
 2019-03-18 18:47:15 -07:00
Sarjeet Singh a6fc6fb207 Add InClusterconfig for sparkctl if no kubeconfig provided (#420) 
* Add InClusterconfig for sparkctl if no kubeconfig provided

* Importing os

* Updated based on review comment

* Minor fix for reordering import package
 2019-02-21 14:47:51 -08:00
Yinan Li c41576b5ff Updated the use the v1beta1 version of the APIs 2019-01-17 10:45:54 -08:00
Grzegorz Lyczba 73ecfa80ac rename CompletionTime to TerminationTime 2018-12-17 23:36:48 +01:00
akhurana001 8c7fdbb306 Operator State Management + Ingress Creation (#291) 
* SparkOperator: Prometheus Metrics Integration

* Prometheus Metric Update

* Spark Operator:Prometheus Metric Integration

* PositiveGauge rework

* remove unwanted dependencies

* Propogating ScheduledSpark App Labels

* Doc update

* Metric Description update

* fix app wait

* SparkOperator: Prometheus Metrics Integration

* Spark Operator metrics:PR Comments

* SparkOperator: Set completion time for Failed App

* Operator Metrics: PR comments

* Spark Operator: PR Comments

* Controller Update

* PR Comments

* Docs Update

* Driver State Transition Check Update

* Operator State Management

* Clean-up

* Exposing Spark Application Id in Operator

* SparkAppId updates

* Add Lyft as a user and contributor to operator

* Spark Operator Rework

* Reworking Restart-Policy

* Documentation update

* PR comments

* PR comments

* Ingress impl

* Ingress Tests + Updates

* go fmt

* PR Comments

* missing files

* AppId removal: Doc Update

* Doc update

* Delete UI/Ingress + Other minor changes

* Add PENDING_RETRY State

* PR comments

* PR comments

* Clean-up

* Update controller.go

* Add Terminal State

* Terminal State

* Spark improvements

* event type

* Events update

* Update controller.go

* Update controller.go

* PR Comments

* PR comments

* Support Best-effort Spec updates

* New State

* PR comments

* PR comments

* go fmt

* Docs update

* PR feedback

* PR Feedback

* PR comments
 2018-11-19 15:25:55 -08:00
Yinan Li c1ec30965a Added support for creating a SparkApplication from a ScheduledSparkApplication 2018-10-05 10:15:34 -07:00
Riccardo Castellotti 17883cf7dc add the KUBECONFIG check 2018-09-24 10:19:24 -07:00
Yinan Li 90e246faab Fixed go import path 2018-09-13 12:32:58 -07:00
Piotr Mrowczynski 8592f34eed Unify listing with status/event 2018-09-11 09:32:44 -07:00
Piotr Mrowczynski 5b9732cb51 Fix reader closing issue when err is nil 2018-09-04 21:03:45 -07:00
Yinan Li f5b3b9aab4 Changed subcommand events to event for sparkctl 2018-08-09 10:36:28 -07:00
Piotr Mrówczyński 55b7b95b7a Extend sparkctl to stream SparkApplication events (#243) 
* Trigger events on driver/executor pending, running and unknown

* Extend sparkctl to stream SparkApplication events
 2018-08-09 10:09:49 -07:00
Piotr Mrówczyński 8d95f535f7 Add s3 support for staging local dependencies in sparkctl #144 (#179) 
* Add s3a support in sparkctl for staging local dependencies

Add aws sdk as dependency

* Adjust upload-to to go-cloud
Add documentation for staging local dependencies in s3a
 2018-08-09 08:32:24 -07:00
Piotr Mrowczynski c21d0c6ae4 Add PySpark support to sparkctl 2018-07-27 11:36:47 -07:00
Yinan Li 00c6a7fd58 Do not check restart eligibility if the application state has not changed 2018-07-10 23:30:03 -07:00
Piotr Mrowczynski 6179aacf7e Adjust to kubernetes api v1.10 and client-go v7.0 2018-07-06 08:17:47 -07:00
Yinan Li 7e4c3a3270 Added support for streaming logs to sparkctl 2018-04-27 12:24:47 -07:00
Yinan Li 29c99bf1a0 Fixed a few typos in sparkctl 2018-02-07 15:50:36 -08:00
Yinan Li 2520f0547e Ran go fmt on code 2018-01-29 12:54:47 -08:00
Yinan Li 56ec190de2 Added basic validation of SparkApplication spec in create 2018-01-29 12:09:40 -08:00
Yinan Li 91c1852714 Make sure local files are closed 2018-01-25 16:16:19 -08:00
Yinan Li df9bbdc054 Added a flag to control whether to override files already exist remotely 2018-01-25 14:20:50 -08:00
Yinan Li bf5b536e6d Pretty printing executor status 2018-01-25 11:43:24 -08:00
Yinan Li 3373b30e6d Added support for uploading local dependencies to GCS 2018-01-25 10:23:08 -08:00
Yinan Li 7d3aa7a39e Updated to support the single image 2018-01-23 10:41:43 -08:00
Yinan Li b6656a7dc5 Added support for fetching executor logs and some refactoring 2018-01-22 10:14:22 -08:00
Yinan Li fa9483358c Fixed import order 2018-01-21 14:22:56 -08:00
Yinan Li d497960abb Added a forward command for accessing the Spark UI 2018-01-21 12:23:45 -08:00
Yinan Li 736272ca36 Added a list command 2018-01-19 14:32:00 -08:00
Yinan Li edf3417b52 Added basic implementation of log command of sparkctl 2018-01-19 12:53:41 -08:00
Yinan Li 8cbf3e7efe More work on sparkctl 2018-01-19 12:28:17 -08:00
Yinan Li abe1e3f255 Initial setup for sparkctl 2018-01-18 11:30:47 -08:00