7a34c91f6c
Use a Kubernetes Job to run spark-submit for multi-version support
2020-04-18 11:03:26 -07:00
8a4a991a81
Fix typos and links in several docs ( #853 )
...
Trivial fixes on typos, links and formats in several docs.
2020-04-01 13:48:47 -07:00
5afcce2919
add fix for metricsProperties when HasPrometheusConfigFile is true. ( #847 )
...
* add fix for metricsProperties when HasPrometheusConfigFile is true.
* add new config MetricsPropertiesFile.
* add missing auto-generated code from privous PRs.
* fix monitoring_config_test.go test condition, redo the configmap logic in monitoring_config.go.
* redo the configmap & javaOption logic in monitoring_config.go.
* set back the configmap & javaOption logic in monitoring_config.go
* update log.
2020-03-31 09:14:10 -07:00
aae36546e5
Fix for #826 and some refactoring ( #832 )
2020-03-11 10:32:42 -07:00
18572fa33b
Added terminationGracePeriodSeconds and pod/container lifecycle hook to driver pods ( #811 )
...
* feat: delete driver pods with a grace period
* feat: adding lifecycle pod spec for driver pods
* adding tests for grace period and lifecycle
* fix: adding user guide for termination grace period and container hooks
2020-03-05 14:29:25 -08:00
60676326be
Update user-guide.md ( #830 )
...
Expand the description of when to use volumes instead of `/tmp` for scratch space.
2020-03-04 21:52:39 -08:00
d968a5a287
Make spark-local-dir-1 be spark-local-dir-2 ( #829 )
...
This would make it match the location.
2020-03-03 17:56:18 -08:00
c8ae9e3520
Upgraded to Spark 2.4.5 ( #798 )
2020-02-10 09:01:06 -08:00
f0f1b1768e
Added support for using envFrom ( #794 )
2020-02-07 11:08:24 -08:00
c11e54a3e1
Fix broken Kubernetes documentation link ( #791 )
...
Fixes the broken link to the Kubernetes API documentation.
2020-02-05 10:48:46 -08:00
d212b29961
Minor typo in user guide soptionally ( #788 )
2020-02-03 21:00:59 -08:00
e1570cf6c3
Update user-guide.md
2020-01-15 19:40:08 -08:00
ef12de0d41
Fixed broken link to the API spec doc
2020-01-02 12:00:43 -08:00
86fb3bd51d
Added support for specifying init-containers for driver/executors ( #740 )
2019-12-16 16:15:52 -08:00
0bd7592035
Add volumes support for Spark scratch space spark.local.dir ( #707 )
2019-12-15 15:05:40 -08:00
74bd887581
Replaced 2.4.0 with 2.4.4 ( #634 )
2019-09-24 16:52:12 -07:00
86ee076aab
Upgraded default Spark version from 2.4.0 to 2.4.4 ( #625 )
2019-09-20 11:03:55 -07:00
e704c7b15d
Added TTL for SparkApplications ( #615 )
2019-09-13 14:21:58 -07:00
55a1eebc0c
Generate CRD specs, bump to v1beta2 ( #578 )
...
* Generate CRD specs, bump to v1beta2
* Add short/singular CRD names
* Merge upstream/master
* Tweak Cores validation
* Fix typo, merge upstream
* Update remaining docs for v1beta2
2019-09-13 10:37:21 -07:00
edcf4cdc32
Resource quota enforcement webhook ( #544 )
...
* Cert configuration and reloading
* Add support for strict webhook error handling
* Improve webhook error handling
* Don't deregister the webhook when failure policy is strict
* standard error message capitalization
* have the webhook parse its own configuration from flags
* clean up cert provider code
* Add explanation for skipping deregistration
* Resource Quota enforcement webhook
* Fix bad merge
* Cleanup, fixes
* Cleanup
* Document the quota enforcer
2019-08-28 14:11:38 -07:00
c5d04ab4a4
Added documentation for leader election and HA mode ( #575 )
2019-08-19 15:54:43 -07:00
7abb0e247e
Update user-guide.md ( #552 )
...
修改sparkConf，key:value都需要双引号
2019-07-22 23:56:14 -07:00
b0a4d1c923
Revert "Added support for overriding command and args ( #529 )" ( #532 )
...
This reverts commit 26ff3f93e4
2019-06-25 07:35:00 -07:00
26ff3f93e4
Added support for overriding command and args ( #529 )
...
* Added support for overriding command and args
* Style fixes
* Fixed incorrect path in docs
* Fixed typo/casing
* Fixed line spacing, trigger build
2019-06-24 09:32:50 -07:00
7f1de86a9f
Added DNS config option for the driver and executor pods ( #519 )
...
* Added DNS config option for the driver and executor pods
* Added user-guide documentation
2019-06-14 09:10:53 -07:00
504fbb877f
Updated user guide on specification of extra Java options ( #513 )
2019-06-12 13:57:56 -07:00
b3db87b3de
Add hostNetwork option for driver and executor. ( #510 )
...
* Add hostNetwork option for driver and executor.
* refine user guide
2019-06-07 09:32:19 -07:00
fbdd41bdd2
Add support for requesting GPUs ( #509 )
...
* Add support for requesting GPUs
* refine unit test
* Add GPU spec and refine unit test
* Add user guide about GPU and refine code format
* fix typo
2019-06-06 19:17:57 -07:00
0fa38e4fc9
Fixed job restart and upgraded Prometheus exporter jar and Go ( #486 )
...
* Fixed job restart; upgraded Prometheus agent jar
* Code cleanup
* Addressed review comments
* Refactored app state update
* Handled errors
* Handled errors
* Break once driver pod found
* Break once driver pod found
2019-05-01 18:39:28 -07:00
9fd38dd9da
Updated user guide
2019-04-21 19:30:44 -07:00
58499f55a0
Added support for sidecar containers
2019-04-21 18:15:39 -07:00
e9e9b64452
Added support for specifying PodSecurityContext
2019-03-11 14:56:53 -07:00
b7e3df9706
Update the user guide with info on the default secretType
...
I could not find what `secretType` I had to use if I didn't need any special configuration handling and I ended up having to look it up in the source code. I added a single line to help people find this information.
2019-02-08 11:57:10 +01:00
c41576b5ff
Updated the use the v1beta1 version of the APIs
2019-01-17 10:45:54 -08:00
d909c246cd
Added yamls for CRDs
2018-11-27 14:59:23 -08:00
8c7fdbb306
Operator State Management + Ingress Creation ( #291 )
...
* SparkOperator: Prometheus Metrics Integration
* Prometheus Metric Update
* Spark Operator:Prometheus Metric Integration
* PositiveGauge rework
* remove unwanted dependencies
* Propogating ScheduledSpark App Labels
* Doc update
* Metric Description update
* fix app wait
* SparkOperator: Prometheus Metrics Integration
* Spark Operator metrics:PR Comments
* SparkOperator: Set completion time for Failed App
* Operator Metrics: PR comments
* Spark Operator: PR Comments
* Controller Update
* PR Comments
* Docs Update
* Driver State Transition Check Update
* Operator State Management
* Clean-up
* Exposing Spark Application Id in Operator
* SparkAppId updates
* Add Lyft as a user and contributor to operator
* Spark Operator Rework
* Reworking Restart-Policy
* Documentation update
* PR comments
* PR comments
* Ingress impl
* Ingress Tests + Updates
* go fmt
* PR Comments
* missing files
* AppId removal: Doc Update
* Doc update
* Delete UI/Ingress + Other minor changes
* Add PENDING_RETRY State
* PR comments
* PR comments
* Clean-up
* Update controller.go
* Add Terminal State
* Terminal State
* Spark improvements
* event type
* Events update
* Update controller.go
* Update controller.go
* PR Comments
* PR comments
* Support Best-effort Spec updates
* New State
* PR comments
* PR comments
* go fmt
* Docs update
* PR feedback
* PR Feedback
* PR comments
2018-11-19 15:25:55 -08:00
de035eea8a
Fixed user-guide.md
2018-11-09 07:24:23 -08:00
f88bf06043
Cosmetic changes to the doc
2018-11-06 11:49:22 -05:00
b575f307ad
Upgraded to use the Spark 2.4.0 image
2018-11-05 07:51:32 -08:00
0d2df329df
Fix pyspark typo
2018-10-24 15:06:47 -07:00
f350a23efe
Add support for tolerations
2018-10-20 17:21:18 -07:00
2648ac2b2e
Miscellaneous Prometheus fixes
2018-09-19 21:49:43 -07:00
c43465e832
Used "Kubernetes Operator for Apache Spark" in documentation
2018-09-19 09:11:59 -07:00
c50747f374
Fixed a typo
2018-08-20 15:32:35 -07:00
c07b040de0
Added documentation for #255
2018-08-20 15:16:43 -07:00
96b2cfef07
Add PySpark support to Operator
2018-07-27 11:36:47 -07:00
368b4a5ac3
Add mutating admission webhook ( #211 )
...
* Preparation to support affinity/anti-affinity
* Initial commit for the mutating admission webhook
2018-07-16 11:43:46 -07:00
81bbafb4a3
addressing review comments
2018-06-28 19:11:16 -07:00
ea54071d54
Add steps for customizing spark opearator
2018-06-28 19:11:16 -07:00
6d37994c20
Added support for new configs for specifying image-pull secrets, executor core requests, and secretKeyRefs
2018-06-03 21:40:09 -07:00
Yinan Li
Shiqi Sun
jinxingwang
Jim Kleckner
Berkay Öztürk
Jiaxin Shan
kevin hogeland
shencheng
Jose Luis Pedrosa
Qiang.Kang
Chaoran Yu
Age Mooij
akhurana001
Rohit Menon
Sergey Samsonov
Piotr Mrowczynski
Dharmesh Kakadia