# kafka

![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.45.1](https://img.shields.io/badge/AppVersion-0.45.1-informational?style=flat-square)

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| file://../hopsworkslib | hopsworkslib | 0.1.0 |
| https://strimzi.io/charts/ | strimzi-kafka-operator | 0.45.1 |

## Basics

All configuration information is provided in `kafka_cluster.yaml`, it is used by the [Strimzi](https://strimzi.io/) operator for building the Kafka cluster and generating configMap (log4j properties can be provided in separate configMaps if desired).

## Helpful material

[Quick Start for Strimzi Kafka](https://strimzi.io/quickstarts/)

[Strimzi Kafka Github](https://github.com/strimzi/strimzi-kafka-operator)

[Strimzi documentation](https://strimzi.io/docs/operators/latest/configuring.html)

[Deploying Strimzi with certificates blog](https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/)

## Changes To Strimzi

-  Strimzi Kafka operator file `values.yaml` has been modified to use the image with [hops-kafka-authorizer](https://github.com/logicalclocks/hops-kafka-authorizer) dependency (to add dependencies to docker images read [developer guide](https://github.com/strimzi/strimzi-kafka-operator/blob/main/development-docs/DEV_GUIDE.md)).

Branch with the changes to strimzi: https://github.com/logicalclocks/strimzi-kafka-operator/tree/release-0.39.x

## To Run

For testing:
```bash
kubectl get pod -n hopsworks --watch
```

Basics in kafka:
```bash
bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.hopsworks.svc:9092 --list
bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.hopsworks.svc:9092 --create --topic test_onlinefs
```

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| appName | string | `"kafka"` |  |
| cluster.entityOperator | object | `{}` | entity operator configuration for the cluster  |
| cluster.kafka.authorizer.authorizerClass | string | `"io.hops.kafka.HopsAclAuthorizer"` |  |
| cluster.kafka.authorizer.principalBuilderClass | string | `"io.hops.kafka.HopsPrincipalBuilder"` |  |
| cluster.kafka.authorizer.superUsers | list | `[]` |  |
| cluster.kafka.authorizer.type | string | `"custom"` |  |
| cluster.kafka.config.database.name | string | `"hopsworks"` |  |
| cluster.kafka.config.log.retention.bytes | int | `-1` | The maximum size of the log before deleting it. |
| cluster.kafka.config.log.retention.checkIntervalMs | int | `300000` | The interval at which log segments are checked for deletion. |
| cluster.kafka.config.log.retention.hours | int | `168` | The number of hours to keep a log segment before deleting it. |
| cluster.kafka.dependencies.glassfish.consulServiceName | string | `"glassfish"` |  |
| cluster.kafka.dependencies.glassfish.consulServiceTag | string | `"hopsworks"` |  |
| cluster.kafka.dependencies.mysql.consulServiceName | string | `"mysql"` |  |
| cluster.kafka.dependencies.mysql.port | int | `3306` |  |
| cluster.kafka.dependencies.onlinefs.consulServiceName | string | `"onlinefs"` |  |
| cluster.kafka.externalLoadBalancer.annotations | object | `{}` | load balancer annotations |
| cluster.kafka.externalLoadBalancer.class | string | `nil` | load balancer class name |
| cluster.kafka.externalLoadBalancer.dns | string | `""` | In case of unmanaged Load Balancers this is the advertised host for the brokers |
| cluster.kafka.externalLoadBalancer.enabled | string | `nil` | Enable External Load Balancers for Kafka cluster. If not set the .global._hopsworks.externalLoadBalancers.enabled will be used instead |
| cluster.kafka.externalLoadBalancer.finalizers | list | `[]` | A list of finalizers which will be configured for the services created for the external listener. |
| cluster.kafka.externalLoadBalancer.managed | string | `nil` | Cloud provider provisions Load Balancers. If not set the .global._hopsworks.externalLoadBalancers.managed will be used instead |
| cluster.kafka.externalLoadBalancer.nodeSelector | object | `{}` | selector for nodes the load balancer can use to route traffic |
| cluster.kafka.externalLoadBalancer.startingAdvertisedPort | int | `9093` | In case of unmanaged Load Balancers this is the starting advertised port for the brokers   |
| cluster.kafka.jvmOptions | object | `{}` | jvm options |
| cluster.kafka.nodeSelector | object | `{}` | node selector configuration |
| cluster.kafka.podAntiAffinity.required | bool | `false` |  |
| cluster.kafka.replicas | int | `1` |  |
| cluster.kafka.resources | object | `{}` | resources configuration |
| cluster.kafka.services.brokers.annotations."consul.hashicorp.com/service-name" | string | `"kafka"` |  |
| cluster.kafka.services.brokers.annotations."consul.hashicorp.com/service-port" | int | `9092` |  |
| cluster.kafka.services.brokers.annotations."consul.hashicorp.com/service-tags" | string | `"broker"` |  |
| cluster.kafka.services.pods.annotations."prometheus.io/path" | string | `"/metrics"` |  |
| cluster.kafka.services.pods.annotations."prometheus.io/port" | int | `9404` |  |
| cluster.kafka.services.pods.annotations."prometheus.io/scheme" | string | `"http"` |  |
| cluster.kafka.services.pods.annotations."prometheus.io/scrape" | string | `"true"` |  |
| cluster.kafka.storageClassName | string | `nil` | storage class name |
| cluster.kafka.storageSize | string | `"30Gi"` |  |
| cluster.kafka.tolerations | list | `[]` |  |
| cluster.kafka.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |
| cluster.kafka.version | string | `"3.9.0"` |  |
| cluster.name | string | `"kafka-cluster"` |  |
| cluster.tlscerts.annotations."certs.hopsworks.ai/owned-by" | string | `"deployment-strimzi-cluster-operator"` |  |
| cluster.zookeeper.nodeSelector | object | `{}` | node selector configuration |
| cluster.zookeeper.podAntiAffinity.required | bool | `false` |  |
| cluster.zookeeper.replicas | int | `1` |  |
| cluster.zookeeper.resources | object | `{"limits":{"cpu":2,"memory":"3Gi"},"requests":{"cpu":"200m","memory":"1Gi"}}` | resources configuration |
| cluster.zookeeper.services.client.annotations."consul.hashicorp.com/service-name" | string | `"zookeeper"` |  |
| cluster.zookeeper.services.client.annotations."consul.hashicorp.com/service-tags" | string | `"client"` |  |
| cluster.zookeeper.storageClassName | string | `nil` | storage class name |
| cluster.zookeeper.storageSize | string | `"5Gi"` |  |
| cluster.zookeeper.tolerations | list | `[]` |  |
| cluster.zookeeper.topologySpreadConstraint | object | `{}` | The default topology spread constraint. If not defined the global topology spread constraint would be used instead. |
| configmap.name | string | `"kafka-configmap"` |  |
| hopsworkslib | object | `{}` | override hopsworkslib values |
| metrics.configMapKey | string | `"kafka-metrics-config"` |  |
| metrics.enabled | bool | `true` |  |
| metrics.type | string | `"jmxPrometheusExporter"` |  |
| strimzi-kafka-operator | object | `{"createGlobalResources":true,"defaultImageRegistry":"docker.hops.works","defaultImageRepository":"strimzi","defaultImageTag":"0.45.1","enabled":true,"extraEnvs":[{"name":"HOPSWORKS_FIRST_STRIMZICA_CA_CERT","valueFrom":{"secretKeyRef":{"key":"ca.crt","name":"kafka-cluster-cluster-ca-cert"}}},{"name":"HOPSWORKS_FIRST_STRIMZICA_CLIENT_CERT","valueFrom":{"secretKeyRef":{"key":"ca.crt","name":"kafka-cluster-clients-ca-cert"}}},{"name":"HOPSWORKS_FIRST_STRIMZICA_CLIENT_CA","valueFrom":{"secretKeyRef":{"key":"ca.key","name":"kafka-cluster-clients-ca"}}},{"name":"HOPSWORKS_FIRST_STRIMZICA_CA","valueFrom":{"secretKeyRef":{"key":"ca.key","name":"kafka-cluster-cluster-ca"}}}],"nodeSelector":{},"tolerations":[],"watchAnyNamespace":false}` | override values for strimzi operator |
| topics | list | `[]` | A list of topics to provision in the Kafka cluster. |
| users | list | `[]` | A list of user to provision in the Kafka cluster. |