kubeflow
/
spark-operator
mirror of https://github.com/kubeflow/spark-operator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
710 Commits 11 Branches 103 Tags 62 MiB
Commit Graph

72 Commits

Author SHA1 Message Date
Yinan Li 0600e96b1c Fixed issues after merging 2020-04-18 11:21:12 -07:00
Yinan Li dd58015019 Fixed a broken test in multi-version-support 2020-04-18 11:04:47 -07:00
Yinan Li 7a34c91f6c Use a Kubernetes Job to run spark-submit for multi-version support 2020-04-18 11:03:26 -07:00
jinxingwang 5afcce2919
add fix for metricsProperties when HasPrometheusConfigFile is true. (#847) 
* add fix for metricsProperties when HasPrometheusConfigFile is true.

* add new config MetricsPropertiesFile.

* add missing auto-generated code from privous PRs.

* fix monitoring_config_test.go test condition, redo the configmap logic in monitoring_config.go.

* redo the configmap & javaOption logic in monitoring_config.go.

* set back the configmap & javaOption logic in monitoring_config.go

* update log.
 2020-03-31 09:14:10 -07:00
Yinan Li c8ae9e3520
Upgraded to Spark 2.4.5 (#798) 2020-02-10 09:01:06 -08:00
Yinan Li 5a1e3eaddf
Marked time field in status nullable (#769) 2020-01-13 16:17:19 -08:00
Yinan Li bbc3e71442
Updated CRD yamls and API docs (#749) 2019-12-19 16:21:09 -08:00
Yinan Li 35e24b1126
Removed initializers from all operator deployment yamls (#678) 2019-11-01 14:26:04 -07:00
Yinan Li f783611199
Fixed a typo in the validation tag for Prometheus port (#649) 2019-10-02 15:40:21 -07:00
Yinan Li 22e51f265e
Fixed RBAC permissions after enabling of subresources (#647) 2019-10-02 14:49:56 -07:00
akhurana001 61ce189fc8 Fix v1beta2: Use UpdateStatus to update subresource (#645) 
* Use UpdateStatus for subresource update

* Remove terminationTime check

* Remove status validation

* Use UpdateStatus for ScheduledSparkApplications
 2019-10-02 13:59:37 -07:00
Yinan Li 86ee076aab
Upgraded default Spark version from 2.4.0 to 2.4.4 (#625) 2019-09-20 11:03:55 -07:00
Coetzee van Staden c16a2f8249 Enumerate string spec on Python version (#629) 2019-09-20 07:43:47 -07:00
Yinan Li 409ac4c443
Run controller-gen to update CRD definitions (#623) 2019-09-18 09:12:22 -07:00
Yinan Li 522ae38c5b
Added missing omitempty tag for port (#622) 2019-09-17 14:52:07 -07:00
kevin hogeland 55a1eebc0c Generate CRD specs, bump to v1beta2 (#578) 
* Generate CRD specs, bump to v1beta2

* Add short/singular CRD names

* Merge upstream/master

* Tweak Cores validation

* Fix typo, merge upstream

* Update remaining docs for v1beta2
 2019-09-13 10:37:21 -07:00
kevin hogeland edcf4cdc32 Resource quota enforcement webhook (#544) 
* Cert configuration and reloading

* Add support for strict webhook error handling

* Improve webhook error handling

* Don't deregister the webhook when failure policy is strict

* standard error message capitalization

* have the webhook parse its own configuration from flags

* clean up cert provider code

* Add explanation for skipping deregistration

* Resource Quota enforcement webhook

* Fix bad merge

* Cleanup, fixes

* Cleanup

* Document the quota enforcer
 2019-08-28 14:11:38 -07:00
kevin hogeland bbab654a60 Implement leader election for HA (#518) 
* Implement leader election

* Remove unused import

* Remove hardcoded configuration

* Remove accidental checkin

* Start metrics and informer callbacks after leader election
 2019-08-19 15:04:39 -07:00
Abdullah Almariah 30ed5e9a97 Add sparkapplications, scheduledsparkapplications permissions to admin 2019-04-10 20:31:26 +02:00
Yinan Li 0c03db461a
Merge pull request #467 from amukhopad/master 
Limit SparkApplication name to 63 characters in JSON schema validation
 2019-04-04 09:51:08 -07:00
Oleksandr Muhkopad 9ccc0d6030 Limit SparkApplication name to 63 characters in JSON schema validation 
Fixes #465
 2019-04-03 19:53:41 +03:00
Chaoran Yu 6609df0a18 Refactored integration tests (#445) 
* Refactored integration tests

* Addressed review comments
 2019-03-18 18:47:15 -07:00
Oleksandr Mukhopad 4d6f9a059b Provide field validation in SparkApplication CRD yaml 2019-01-24 18:48:05 +02:00
Yinan Li c41576b5ff Updated the use the v1beta1 version of the APIs 2019-01-17 10:45:54 -08:00
Yinan Li b867cc662b Fixed the controller logic for ScheduledSparkApplications 2019-01-16 08:56:49 -08:00
Chaoran Yu 73d09c8c4d Added support for running the containers using a non-root user; fixed an issue with running multiple instances of operator; fixed doc to reflect webhook new default in the chart 2018-12-11 23:53:07 +08:00
Yinan Li 5ff9bd43df Various fixes for the operator 
1. Fixed SparkApplication controller to ignore reconcilation caused by SparkApplication not found errors.
2. Fixed the PySpark example.
3. Fixed code for updating the status of a SparkApplication in updateAndExportMetrics.
4. Logging refactoring.
 2018-12-05 14:50:26 -08:00
Yinan Li d909c246cd Added yamls for CRDs 2018-11-27 14:59:23 -08:00
akhurana001 8c7fdbb306 Operator State Management + Ingress Creation (#291) 
* SparkOperator: Prometheus Metrics Integration

* Prometheus Metric Update

* Spark Operator:Prometheus Metric Integration

* PositiveGauge rework

* remove unwanted dependencies

* Propogating ScheduledSpark App Labels

* Doc update

* Metric Description update

* fix app wait

* SparkOperator: Prometheus Metrics Integration

* Spark Operator metrics:PR Comments

* SparkOperator: Set completion time for Failed App

* Operator Metrics: PR comments

* Spark Operator: PR Comments

* Controller Update

* PR Comments

* Docs Update

* Driver State Transition Check Update

* Operator State Management

* Clean-up

* Exposing Spark Application Id in Operator

* SparkAppId updates

* Add Lyft as a user and contributor to operator

* Spark Operator Rework

* Reworking Restart-Policy

* Documentation update

* PR comments

* PR comments

* Ingress impl

* Ingress Tests + Updates

* go fmt

* PR Comments

* missing files

* AppId removal: Doc Update

* Doc update

* Delete UI/Ingress + Other minor changes

* Add PENDING_RETRY State

* PR comments

* PR comments

* Clean-up

* Update controller.go

* Add Terminal State

* Terminal State

* Spark improvements

* event type

* Events update

* Update controller.go

* Update controller.go

* PR Comments

* PR comments

* Support Best-effort Spec updates

* New State

* PR comments

* PR comments

* go fmt

* Docs update

* PR feedback

* PR Feedback

* PR comments
 2018-11-19 15:25:55 -08:00
Chaoran Yu 679f21dfc5
Merge branch 'master' into tests 2018-11-05 23:03:53 -05:00
Yinan Li b575f307ad Upgraded to use the Spark 2.4.0 image 2018-11-05 07:51:32 -08:00
Chaoran Yu 2ce8ae028d Added basic integration tests 2018-11-01 21:10:42 -04:00
Chaoran Yu 95a4d99dff Added e2e testing framework (#309) 
* Added e2e testing framework

* Reformatting with Go 1.10

* Addressed review comments

* Fixed variable

* Fixed remaining issues
 2018-10-19 21:17:43 -07:00
Piotr Mrowczynski bc9bf22a73 Add batch job to initialize the webhook secret to manifest, remove requirement of manual step 2018-10-05 10:16:41 -07:00
Sarjeet Singh 499adeac70 Updated namespace in manifest to be consistent with helm charts 2018-09-25 19:28:21 -07:00
Yinan Li 7e413a3bf7 Updated the label selector for service spark-webhook 2018-08-14 11:50:05 -07:00
Yinan Li be6b21fbb9 Fixed annotations in the YAML files 2018-08-08 23:06:27 -07:00
Yinan Li fdbc8511f7 Upgraded to Spark 2.3.1 2018-08-08 20:16:48 -07:00
akhurana001 ceb230481e SparkOperator: Prometheus Metrics Integration (#227) 
* SparkOperator: Prometheus Metrics Integration

* Spark Operator metrics:PR Comments

* SparkOperator: Set completion time for Failed App

* Operator Metrics: PR comments

* Spark Operator: PR Comments

* Controller Update

* PR Comments

* Docs Update

* Driver State Transition Check Update
 2018-08-03 13:56:58 -07:00
Yinan Li 9a08ae361c Get rid of the initializer (replaced by the mutating webhook) 2018-07-16 12:05:12 -07:00
Yinan Li 6d37a8efd9 Updated the manifest files 2018-07-16 11:49:19 -07:00
Yinan Li 368b4a5ac3 Add mutating admission webhook (#211) 
* Preparation to support affinity/anti-affinity

* Initial commit for the mutating admission webhook
 2018-07-16 11:43:46 -07:00
Yinan Li 9f92e254fc Removed check in the presence of kube-dns 2018-07-11 23:10:02 -07:00
Yinan Li fca2c5552f Fixed RBAC for the spark operator 2018-06-02 16:06:34 -07:00
Yinan Li a2d9f2db45 Fixed RBAC for the operator to grant just the necessary permissions 2018-05-24 09:22:41 -07:00
Yinan Li 74af880633 Updated spark-operator.yaml 2018-04-25 09:50:27 -07:00
Yinan Li 8f624469c6 Added documentation for cron support 2018-04-19 15:00:46 -07:00
Yinan Li 3395801314 Updated docker image 2018-03-20 16:48:28 -07:00
Yinan Li c14f034127 Updated spark-operator.yaml to use a Recreate strategy 2018-02-23 15:56:50 -08:00
Yinan Li 2cf8975b2a Added support for retrying failed submissions 2018-02-12 15:08:46 -08:00