# airflow

This helm chart starts two deployments: airflow webserver and airflow scheduler. These services need the namenode deployment to be ready and running.

![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.10.10.5](https://img.shields.io/badge/AppVersion-1.10.10.5-informational?style=flat-square)

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| file://../hopsworkslib | hopsworkslib | 0.1.0 |

## Terminating airflow containers

We use hopsfs-mount as a background process in the sidecar container. Therefore, the SIGINT signal of kubelet is not received well. We added a preStop hook to this container in order to prevent the sidecar container to hang in `TERMINATING` stage as follows:

```yaml
  lifecycle:
    preStop:
      exec:
        # Nuke and unmount the hdfs mount
        command: ["/bin/bash", "-c", "umount -l $MOUNTING_DIR && rm -rf $MOUNTING_DIR"]
```

Notice that the previous hook only works when the `grace-period` of the container is not zero.

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| appName | string | `"airflow"` | app name label |
| bidirectional_mount | string | `"mounted"` | the name of bidirectional mount |
| common.celery.broker_url | string | `"rdis://:6379/0"` |  |
| common.celery.celery_app_name | string | `"airflow.executors.celery_executor"` |  |
| common.celery.default_queue | string | `"default"` |  |
| common.celery.flower_port | int | `5555` |  |
| common.celery.flower_url_prefix | string | `"http://localhost/hopsworks-api/flower"` |  |
| common.celery.worker_concurrency | int | `8` |  |
| common.celery.worker_log_server_port | int | `8793` |  |
| common.core_config.airflow_home | string | `"/airflow"` |  |
| common.core_config.base_log_folder | string | `"/airflow/logs"` |  |
| common.core_config.dag_concurrency | int | `16` |  |
| common.core_config.dagbag_import_timeout | int | `60` |  |
| common.core_config.dags_are_paused_at_creation | bool | `true` |  |
| common.core_config.dags_folder | string | `"/airflow/dags"` |  |
| common.core_config.default_timezone | string | `"utc"` |  |
| common.core_config.donot_pickle | bool | `false` |  |
| common.core_config.executor | string | `"LocalExecutor"` |  |
| common.core_config.fernet_key | string | `"G3jB5--jCQpRYp7hwUtpfQ_S8zLRbRMwX8tr3dehnNU="` |  |
| common.core_config.load_examples | bool | `false` |  |
| common.core_config.logging_config_class | string | `"log_config.LOGGING_CONFIG"` |  |
| common.core_config.max_active_runs_per_dag | int | `16` |  |
| common.core_config.non_pooled_task_slot_count | int | `128` |  |
| common.core_config.parallelism | int | `32` |  |
| common.core_config.plugins_folder | string | `"/airflow/plugins"` |  |
| common.core_config.sql_alchemy_max_overflow | int | `30` | We set it high, not unlimited to account for memory leaks |
| common.core_config.sql_alchemy_pool_pre_ping | bool | `true` |  |
| common.core_config.sql_alchemy_pool_recycle | int | `3600` |  |
| common.core_config.sql_alchemy_pool_size | int | `10` |  |
| common.group | string | `"airflow"` |  |
| common.namenode_cluster_ip_name | string | `"namenode-cluster-ip"` |  |
| common.smtp.smtp_host | string | `"localhost"` |  |
| common.smtp.smtp_mail_from | string | `"admin@kth.se"` |  |
| common.smtp.smtp_password | string | `"admin"` |  |
| common.smtp.smtp_port | int | `25` |  |
| common.smtp.smtp_ssl | bool | `false` |  |
| common.smtp.smtp_starttls | bool | `true` |  |
| common.smtp.smtp_user | string | `"admin@kth.se"` |  |
| common.user | string | `"airflow"` |  |
| debug | bool | `false` | enable or disable debug logging for hopsfs |
| dependencies.hopsworks | object | `{"consulServiceName":"glassfish","consulServiceTag":"hopsworks","port":8182}` | hopsworks consul service |
| dependencies.mysql | object | `{"consulServiceName":"mysql","port":3306}` | mysql consul service |
| dependencies.namenode | object | `{"consulServiceName":"namenode","port":8020}` | namenode consul service. This uses a headless ClusterIP underneath with `publishNotReadyAddresses: true` |
| hopsfs_bin_url | string | `nil` | url to download a patched hopsfs mount for testing and development |
| hopsworkslib | object | `{}` | override hopsworkslib values |
| image | object | `{"pullPolicy":"IfNotPresent","registry":"docker.hops.works"}` | image configuration. The image tag is the .Chart.AppVersion  |
| migrationBackOffLimit | int | `10` | backoffLimit for airflow migration job |
| reset_db_if_error | bool | `false` | if true, will reset the database and create a new one in case of error |
| scheduler.config.dag_dir_list_interval | int | `40` |  |
| scheduler.config.job_heartbeat_sec | int | `5` |  |
| scheduler.config.max_threads | int | `2` |  |
| scheduler.config.min_file_process_interval | int | `10` |  |
| scheduler.config.print_stats_interval | int | `600` |  |
| scheduler.config.scheduler_zombie_task_threshold | int | `300` |  |
| scheduler.deployment.replicas | int | `1` |  |
| scheduler.is_tls | bool | `false` |  |
| scheduler.name | string | `"airflow-scheduler"` |  |
| scheduler.nodeSelector | object | `{}` | node selector configuration |
| scheduler.probe.failureThreshold | int | `20` |  |
| scheduler.probe.initialDelaySeconds | int | `30` |  |
| scheduler.probe.periodSeconds | int | `10` |  |
| scheduler.probe.timeoutSeconds | int | `10` |  |
| scheduler.probeCheckWebserver | bool | `true` |  |
| scheduler.probeCommand | string | `"set -e && pgrep -f \"airflow scheduler\""` | The probe here will be concatenated with sleeping for the heartbeat time and then trying to reach the webserver. If the webserver is not reachable the scheduler should be restarted  https://hopsworks.atlassian.net/browse/HWORKS-1915 The scheduler goes to a non consistent state, then the health check of the webserver returns scheduler non-healthy and it fails |
| scheduler.resources.limits | object | `{"memory":"2000Mi"}` | resources limits configuration |
| scheduler.resources.requests | object | `{"cpu":"1","memory":"1000Mi"}` | resources requests configuration |
| scheduler.securityContext.runAsGroup | int | `1508` |  |
| scheduler.securityContext.runAsNonRoot | bool | `true` |  |
| scheduler.securityContext.runAsUser | int | `1512` |  |
| scheduler.service.annotations."consul.hashicorp.com/service-name" | string | `"airflow"` |  |
| scheduler.service.annotations."consul.hashicorp.com/service-tags" | string | `"scheduler"` |  |
| scheduler.tolerations | list | `[]` |  |
| scheduler.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |
| serviceAccount.annotations | object | `{}` | service account annotations |
| serviceAccountName | string | `"airflow"` | service account name |
| tls | bool | `true` | enable or disable TLS |
| webserver.config.authenticate | bool | `true` |  |
| webserver.config.expose_config | bool | `true` |  |
| webserver.config.rbac | bool | `true` |  |
| webserver.config.secret_key | string | `"temporary_key"` |  |
| webserver.config.web_server_host | string | `"0.0.0.0"` |  |
| webserver.config.web_server_port | int | `12358` |  |
| webserver.config.web_server_worker_timeout | int | `120` |  |
| webserver.config.worker_class | string | `"sync"` |  |
| webserver.config.workers | int | `2` |  |
| webserver.configHelm.base_path | string | `"/hopsworks-api/airflow"` |  |
| webserver.deployment.replicas | int | `1` |  |
| webserver.heartbeat | int | `30` |  |
| webserver.is_tls | bool | `false` |  |
| webserver.name | string | `"airflow-webserver"` |  |
| webserver.nodeSelector | object | `{}` | node selector configuration |
| webserver.probe.failureThreshold | int | `10` |  |
| webserver.probe.initialDelaySeconds | int | `60` |  |
| webserver.probe.periodSeconds | int | `5` |  |
| webserver.probe.timeoutSeconds | int | `10` |  |
| webserver.resources.limits | object | `{"memory":"2000Mi"}` | resources limits configuration |
| webserver.resources.requests | object | `{"cpu":"100m","memory":"1000Mi"}` | resources requests configuration |
| webserver.securityContext.runAsGroup | int | `1508` |  |
| webserver.securityContext.runAsUser | int | `1512` |  |
| webserver.service.annotations."consul.hashicorp.com/service-name" | string | `"airflow"` |  |
| webserver.service.annotations."consul.hashicorp.com/service-tags" | string | `"ui"` |  |
| webserver.service.annotations."prometheus.io/path" | string | `"/hopsworks-api/airflow/admin/metrics"` |  |
| webserver.service.annotations."prometheus.io/port" | int | `12358` |  |
| webserver.service.annotations."prometheus.io/scheme" | string | `"http"` |  |
| webserver.service.annotations."prometheus.io/scrape" | string | `"true"` |  |
| webserver.tolerations | list | `[]` |  |
| webserver.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |