kubeflow
/
spark-operator
mirror of https://github.com/kubeflow/spark-operator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
710 Commits 11 Branches 100 Tags 62 MiB
Commit Graph

45 Commits

Author SHA1 Message Date
Yinan Li 7a34c91f6c Use a Kubernetes Job to run spark-submit for multi-version support 2020-04-18 11:03:26 -07:00
Shiqi Sun 8a4a991a81
Fix typos and links in several docs (#853) 
Trivial fixes on typos, links and formats in several docs.
 2020-04-01 13:48:47 -07:00
Yinan Li 1c4486cdda
Removed legacy init-container related fields (#750) 2019-12-20 10:08:05 -08:00
Jiaxin Shan 0bd7592035 Add volumes support for Spark scratch space spark.local.dir (#707) 2019-12-15 15:05:40 -08:00
Clemens Valiente 29888bb088 fixing copy&paste error in sparkctl Readme (#696) 
fix copy&paste error & grammar
 2019-11-17 23:11:15 -08:00
Yinan Li baafaafeea
Fixed the client version used in sparkctl (#681) 2019-11-03 18:50:21 -08:00
kevin hogeland 55a1eebc0c Generate CRD specs, bump to v1beta2 (#578) 
* Generate CRD specs, bump to v1beta2

* Add short/singular CRD names

* Merge upstream/master

* Tweak Cores validation

* Fix typo, merge upstream

* Update remaining docs for v1beta2
 2019-09-13 10:37:21 -07:00
Chaoran Yu 6609df0a18 Refactored integration tests (#445) 
* Refactored integration tests

* Addressed review comments
 2019-03-18 18:47:15 -07:00
Sarjeet Singh a6fc6fb207 Add InClusterconfig for sparkctl if no kubeconfig provided (#420) 
* Add InClusterconfig for sparkctl if no kubeconfig provided

* Importing os

* Updated based on review comment

* Minor fix for reordering import package
 2019-02-21 14:47:51 -08:00
Stavros Kontopoulos 2feda257dd update test suite 2019-02-15 14:54:56 +02:00
Jessie A. Morris 4532f5263b
Fix install instructions in sparkctl readme 2019-02-05 14:26:42 -07:00
Yinan Li c41576b5ff Updated the use the v1beta1 version of the APIs 2019-01-17 10:45:54 -08:00
Grzegorz Lyczba 73ecfa80ac rename CompletionTime to TerminationTime 2018-12-17 23:36:48 +01:00
akhurana001 8c7fdbb306 Operator State Management + Ingress Creation (#291) 
* SparkOperator: Prometheus Metrics Integration

* Prometheus Metric Update

* Spark Operator:Prometheus Metric Integration

* PositiveGauge rework

* remove unwanted dependencies

* Propogating ScheduledSpark App Labels

* Doc update

* Metric Description update

* fix app wait

* SparkOperator: Prometheus Metrics Integration

* Spark Operator metrics:PR Comments

* SparkOperator: Set completion time for Failed App

* Operator Metrics: PR comments

* Spark Operator: PR Comments

* Controller Update

* PR Comments

* Docs Update

* Driver State Transition Check Update

* Operator State Management

* Clean-up

* Exposing Spark Application Id in Operator

* SparkAppId updates

* Add Lyft as a user and contributor to operator

* Spark Operator Rework

* Reworking Restart-Policy

* Documentation update

* PR comments

* PR comments

* Ingress impl

* Ingress Tests + Updates

* go fmt

* PR Comments

* missing files

* AppId removal: Doc Update

* Doc update

* Delete UI/Ingress + Other minor changes

* Add PENDING_RETRY State

* PR comments

* PR comments

* Clean-up

* Update controller.go

* Add Terminal State

* Terminal State

* Spark improvements

* event type

* Events update

* Update controller.go

* Update controller.go

* PR Comments

* PR comments

* Support Best-effort Spec updates

* New State

* PR comments

* PR comments

* go fmt

* Docs update

* PR feedback

* PR Feedback

* PR comments
 2018-11-19 15:25:55 -08:00
Chaoran Yu e0d6326ca3 Minor doc updates 2018-11-06 11:49:22 -05:00
Yinan Li c1ec30965a Added support for creating a SparkApplication from a ScheduledSparkApplication 2018-10-05 10:15:34 -07:00
Riccardo Castellotti 17883cf7dc add the KUBECONFIG check 2018-09-24 10:19:24 -07:00
Yinan Li 90e246faab Fixed go import path 2018-09-13 12:32:58 -07:00
Piotr Mrowczynski 8592f34eed Unify listing with status/event 2018-09-11 09:32:44 -07:00
Piotr Mrowczynski 5b9732cb51 Fix reader closing issue when err is nil 2018-09-04 21:03:45 -07:00
Piotr Mrowczynski 686c094c67 Adjust sparkctl documentation for event cmd 2018-08-13 09:31:10 -07:00
Yinan Li f5b3b9aab4 Changed subcommand events to event for sparkctl 2018-08-09 10:36:28 -07:00
Piotr Mrówczyński 55b7b95b7a Extend sparkctl to stream SparkApplication events (#243) 
* Trigger events on driver/executor pending, running and unknown

* Extend sparkctl to stream SparkApplication events
 2018-08-09 10:09:49 -07:00
Piotr Mrówczyński 8d95f535f7 Add s3 support for staging local dependencies in sparkctl #144 (#179) 
* Add s3a support in sparkctl for staging local dependencies

Add aws sdk as dependency

* Adjust upload-to to go-cloud
Add documentation for staging local dependencies in s3a
 2018-08-09 08:32:24 -07:00
Piotr Mrowczynski c21d0c6ae4 Add PySpark support to sparkctl 2018-07-27 11:36:47 -07:00
Yinan Li 00c6a7fd58 Do not check restart eligibility if the application state has not changed 2018-07-10 23:30:03 -07:00
Piotr Mrowczynski 6179aacf7e Adjust to kubernetes api v1.10 and client-go v7.0 2018-07-06 08:17:47 -07:00
Lena 22bccd5250 Ensure all dependencies are present comment 
For Go beginners, it might be valuable to state that there're instructions on how to build the operator and get dependencies prior to building sparkctl.
 2018-05-25 10:48:06 -07:00
Yinan Li 7e4c3a3270 Added support for streaming logs to sparkctl 2018-04-27 12:24:47 -07:00
Yinan Li 29c99bf1a0 Fixed a few typos in sparkctl 2018-02-07 15:50:36 -08:00
Yinan Li 2520f0547e Ran go fmt on code 2018-01-29 12:54:47 -08:00
Yinan Li 56ec190de2 Added basic validation of SparkApplication spec in create 2018-01-29 12:09:40 -08:00
Yinan Li 91c1852714 Make sure local files are closed 2018-01-25 16:16:19 -08:00
Yinan Li df9bbdc054 Added a flag to control whether to override files already exist remotely 2018-01-25 14:20:50 -08:00
Yinan Li bf5b536e6d Pretty printing executor status 2018-01-25 11:43:24 -08:00
Yinan Li 3373b30e6d Added support for uploading local dependencies to GCS 2018-01-25 10:23:08 -08:00
Yinan Li 7d3aa7a39e Updated to support the single image 2018-01-23 10:41:43 -08:00
Yinan Li b6656a7dc5 Added support for fetching executor logs and some refactoring 2018-01-22 10:14:22 -08:00
Yinan Li fa9483358c Fixed import order 2018-01-21 14:22:56 -08:00
Yinan Li 34aa172c0a Added a README for sparkctl 2018-01-21 13:32:49 -08:00
Yinan Li d497960abb Added a forward command for accessing the Spark UI 2018-01-21 12:23:45 -08:00
Yinan Li 736272ca36 Added a list command 2018-01-19 14:32:00 -08:00
Yinan Li edf3417b52 Added basic implementation of log command of sparkctl 2018-01-19 12:53:41 -08:00
Yinan Li 8cbf3e7efe More work on sparkctl 2018-01-19 12:28:17 -08:00
Yinan Li abe1e3f255 Initial setup for sparkctl 2018-01-18 11:30:47 -08:00