# spark

![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.1.0](https://img.shields.io/badge/AppVersion-0.1.0-informational?style=flat-square)

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| file://../hopsworkslib | hopsworkslib | 0.1.0 |
| https://kubeflow.github.io/spark-operator/ | spark-operator | 1.1.27 |

This is the helm chart for installing spark services on kubernetes. This helm chart installs the following:
1. Uniffle Remote Shuffle Service (RSS) coordinator and server. You should first install the rss chart that installs the CRD, webhook and controller for RSS.
2. Spark history server
3. Config maps for spark

## FIX, TODO:

The spark operator stop wotking after an upgrade if the serviceAccount for the operator is annotated as `create`. The reason seems to be the created serviceAccount. Since it is refreshed and the old spark-operator is not rolled out. See the full issue here, https://github.com/kubeflow/spark-operator/issues/1554

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| dependencies.hive.consulServiceName | string | `"hive"` |  |
| dependencies.hive.consulServiceTag | string | `"metastore"` |  |
| dependencies.hive.port | int | `9083` |  |
| dependencies.namenode.consulServiceName | string | `"namenode"` |  |
| dependencies.namenode.consulServiceTag | string | `"rpc"` |  |
| dependencies.namenode.port | int | `8020` |  |
| dependencies.prometheusPushgateway.consulServiceName | string | `"prometheus"` |  |
| dependencies.prometheusPushgateway.consulServiceTag | string | `"pushgateway"` |  |
| dependencies.prometheusPushgateway.port | int | `9091` |  |
| dependencies.prometheusPushgateway.protocol | string | `"http"` |  |
| historyServer | object | `{"certsDir":"/srv/hops/super_crypto/spark","cleaner":{"enabled":true,"interval":"1d","maxAge":"7d"},"deploymentName":"spark-history-server-deployment","hadoopHome":"/srv/hops/hadoop","image":{"pullPolicy":"Always","repository":"sparkhistoryserver","tag":"3.5.5"},"name":"spark-history-server","nodeSelector":{},"probes":{"liveness":{"failureThreshold":10,"httpGet":{"path":"/","port":"http","scheme":"HTTPS"},"initialDelaySeconds":8,"periodSeconds":5,"timeoutSeconds":10},"readiness":{"failureThreshold":10,"httpGet":{"path":"/","port":"http","scheme":"HTTPS"},"initialDelaySeconds":8,"periodSeconds":5,"timeoutSeconds":10}},"replicaCount":1,"resources":{"limits":{"cpu":"2000m","memory":"3Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"service":{"annotations":{"consul.hashicorp.com/service-name":"sparkhistoryserver"},"externalPort":80,"internalPort":18080,"name":"sparkhistoryserver","type":"ClusterIP"},"sparkHome":"/srv/hops/spark","tolerations":[],"topologySpreadConstraint":{}}` | The configuration for the Spark history server |
| historyServer.nodeSelector | object | `{}` | node selector configuration |
| historyServer.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |
| hopsworkslib | object | `{}` | override hopsworkslib values |
| rss | object | `{"appName":"rss-hops","configDir":"/data/rssadmin/rss/conf","configmap":{"name":"rss-configuration"},"controller":{"containerPort":9876,"image":"rss-controller","replicas":1,"resources":{"limits":{"cpu":1,"memory":"512Mi"},"requests":{"cpu":"100m","memory":"150Mi"}},"serviceAccount":{"annotations":{}}},"coordinator":{"count":2,"dynamicClientConfigMapName":"rss-dynamic-client-configuration","dynamicClientConfigMountPath":"/tmp","httpPort":19996,"labels":{"role":"rss-hops-coordinator"},"replicas":1,"resources":{"limits":{"cpu":"500m","memory":"2Gi"},"requests":{"cpu":"500m"}},"rpcPort":19997,"xmxSizeMemoryExtraPercentage":30},"dashboard":{"enabled":false,"httpPort":19997,"name":"uniffle-dashboard","resources":{"limits":{"cpu":"500m","memory":"2Gi"},"requests":{"cpu":"250m"}},"xmxSizeMemoryExtraPercentage":30},"dynamicClient":{"readBufferSize":"14m","storageType":"MEMORY_LOCALFILE"},"fullnameOverride":null,"image":{"image":"rss","initImage":"hops-rss-init","initImageVersion":"0.9.1","pullPolicy":"Always"},"nodeSelector":{},"resources":{"jobs":{"limits":{"cpu":"200m"}},"rss":{"limits":{"cpu":"500m","memory":"1Gi"}}},"serviceAccount":{"annotations":{}},"shuffleServer":{"bufferCapacity":-1,"bufferCapacityRatio":0.7,"diskCapacity":-1,"diskCapacityRatio":0.8,"httpPort":19998,"nettyPort":20000,"readBufferCapacity":-1,"readBufferCapacityRatio":0.1,"replicas":3,"resources":{"limits":{"cpu":"1000m","memory":"3Gi"},"requests":{"cpu":"200m"}},"rpcPort":19999,"upgradeStrategy":"FullUpgrade"},"storage":{"size":"10Gi","storageClassName":null,"volumeNameTemplate":"rss-storage"},"tolerations":[],"topologySpreadConstraint":{},"version":"0.10.1","webhook":{"app":"rss-webhook","resources":{"limits":{"cpu":1,"memory":"512Mi"},"requests":{"cpu":"100m","memory":"150Mi"}},"service":{"name":"rss-webhook","port":443,"targetPort":9876}}}` | The configuration for the uniffle remote shuffle service |
| rss.controller.serviceAccount.annotations | object | `{}` | service account annotations |
| rss.coordinator.resources.requests | object | `{"cpu":"500m"}` | memory is set to be equal to the limit  |
| rss.dashboard.resources.requests | object | `{"cpu":"250m"}` | memory is set to be equal to the limit  |
| rss.fullnameOverride | string | `nil` | fullnameOverride |
| rss.nodeSelector | object | `{}` | node selector configuration |
| rss.resources.jobs | object | `{"limits":{"cpu":"200m"}}` | jobs resources |
| rss.resources.rss | object | `{"limits":{"cpu":"500m","memory":"1Gi"}}` | rss resources |
| rss.serviceAccount.annotations | object | `{}` | service account annotations |
| rss.shuffleServer.resources.requests | object | `{"cpu":"200m"}` | memory is set to be equal to the limit  |
| rss.storage.storageClassName | string | `nil` | storage class name to request for volumes attached to rss |
| rss.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |
| spark-operator | object | `{"batchScheduler":{"enable":false},"controllerThreads":10,"image":{"pullPolicy":"IfNotPresent","repository":"docker.hops.works/hopsworks/spark-operator","tag":"v1beta2-1.3.8-3.1.1"},"logLevel":2,"nodeSelector":{},"replicaCount":1,"resourceQuotaEnforcement":{"enable":false},"resources":{"requests":{"cpu":"300m","memory":"512Mi"}},"resyncInterval":30,"serviceAccounts":{"spark":{"annotations":{},"create":true,"name":""},"sparkoperator":{"annotations":{},"create":true,"name":""}},"tolerations":[],"uiService":{"enable":true},"volumes":[{"name":"certsecret","secret":{"optional":false,"secretName":"spark-history-server-deployment-sparkhistoryserver-crypto-material"}}],"webhook":{"cleanupAnnotations":{"helm.sh/hook":"pre-delete, pre-upgrade","helm.sh/hook-delete-policy":"hook-succeeded"},"cleanupPodLabels":{},"enable":true,"initAnnotations":{"helm.sh/hook":"pre-install, pre-upgrade","helm.sh/hook-weight":"50"},"port":8080,"portName":"webhook","timeout":30}}` | override spark operator values |
| spark-operator.batchScheduler.enable | bool | `false` | Enable batch scheduler for spark jobs scheduling. If enabled, users can specify batch scheduler name in spark application |
| spark-operator.resourceQuotaEnforcement.enable | bool | `false` | Whether to enable the ResourceQuota enforcement for SparkApplication resources. Requires the webhook to be enabled by setting `webhook.enable` to true. Ref: https://github.com/kubeflow/spark-operator/blob/master/docs/user-guide.md#enabling-resource-quota-enforcement. |
| spark-operator.serviceAccounts.spark.annotations | object | `{}` | Optional annotations for the spark service account |
| spark-operator.serviceAccounts.spark.name | string | `""` | Optional name for the spark service account |
| spark-operator.serviceAccounts.sparkoperator.annotations | object | `{}` | Optional annotations for the operator service account |
| spark-operator.serviceAccounts.sparkoperator.create | bool | `true` | Create a service account for the operator |
| spark-operator.serviceAccounts.sparkoperator.name | string | `""` | Optional name for the operator service account |
| spark-operator.uiService.enable | bool | `true` | Enable UI service creation for Spark application |
| spark-operator.webhook.cleanupAnnotations | object | `{"helm.sh/hook":"pre-delete, pre-upgrade","helm.sh/hook-delete-policy":"hook-succeeded"}` | The annotations applied to the cleanup job, required for helm lifecycle hooks |
| spark-operator.webhook.cleanupPodLabels | object | `{}` | The podLabels applied to the pod of the cleanup job |
| sparkJobDebugLevel | string | `"INFO"` |  |
| sparkOperatorRestartJob | object | `{"imagePullPolicy":"Always","name":"spark-operator-restart-job","nodeSelector":{},"serviceAccount":{"annotations":{}},"tolerations":[]}` | Job configuration to restart the spark operator deployment on upgrade so that it can use the new webhook certificates. These certificates are regenerated on upgrade. |
| sparkOperatorRestartJob.nodeSelector | object | `{}` | node selector configuration |
| sparkOperatorRestartJob.serviceAccount.annotations | object | `{}` | service account annotations |