kubeflow
/
spark-operator
mirror of https://github.com/kubeflow/spark-operator.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated user guide on specification of extra Java options (#513)

This commit is contained in:
Yinan Li 2019-06-12 13:57:56 -07:00 committed by GitHub
parent 976e450bd7
commit 504fbb877f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/user-guide.md
View File

@ -11,6 +11,7 @@ The Kubernetes Operator for Apache Spark ships with a command-line tool called `
* [Specifying Hadoop Configuration](#specifying-hadoop-configuration)
* [Writing Driver Specification](#writing-driver-specification)
* [Writing Executor Specification](#writing-executor-specification)
* [Specifying Extra Java Options](#specifying-extra-java-options)
* [Requesting GPU Resources](#requesting-gpu-resources)
* [Host Network](#host-network)
* [Mounting Secrets](#mounting-secrets)
@ -147,6 +148,18 @@ spec:
version: 2.4.0
```
### Specifying Extra Java Options
A `SparkApplication` can specify extra Java options for the driver or executors, using the optional field `.spec.driver.javaOptions` for the driver and `.spec.executor.javaOptions` for executors. Below is an example:
```yaml
spec:
executor:
javaOptions: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
```
Values specified using those two fields get converted to Spark configuration properties `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions`, respectively. **Prefer using the above two fields over configuration properties `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions`** as the fields work well with other fields that might modify what gets set for `spark.driver.extraJavaOptions` or `spark.executor.extraJavaOptions`.
### Requesting GPU Resources
A `SparkApplication` can specify GPU resources for the driver or executor pod, using the optional field `.spec.driver.gpu` or `.spec.executor.gpu`. Below is an example: