# Helm chart for OLK (Filebat-LogStash-OpenSearch-OpenSearchDashboard)

{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }}

{{ template "chart.requirementsSection" . }}

This helm chart provides logs gathering. The filebeat daemonset harvest logs from file `inputs` (for some reason this is faster than autodiscover). By default we mount container log folders and collect stdout and stderr. We do a preprocessing of the logs in the filebeat instances, including some cleaing of the logs (See the **Filebeat processor** heading). For example, we remove ANSI color commands directly in the filebeat instance. Besides, we partially deal with multilines.
Once harvested we send the logs to logstash deployments for an extra processing. The pipeline is configured there to extract and map log fields. 

The logstash instances send the logs to opensearch `logstash-%DATE` indexes. Once there, the indexes can be queried by the opensearch dashboard service.

## Filebeat file inputs

In the `values.yaml`, we take the folders to look for logs from the `filebeat.logs_locations` field. Each entry of the field contains the host path to mount in the filebeat container and a glob pattern. The glob pattern is injected in the filebeat.yaml configuration in order to search correct log files. For example, mounting the path `/var/lib/docker/containers` with glob pattern `/**/*.log` will cover all log files recursively inside the path.

## Access the dashboard

- `kubectl port-forward -n hopsworks service/opensearch-dashboard 5601:5601`

## Delete the logs

**Note** this means data loss

- `kubectl port-forward -n hopsworks service/opensearch 9200:9200`
- `curl -X DELETE "http://localhost:9200/*"`

**Note** opensearch statefulsets and the dashboards share the certificates we generate with the certs-operator.

## Custom annotate logs

Labels and annotations from each pod are automatically injected in the log. For example, once in the opensearch dashboard, logs can be easily filtered by `kubernetes.labels.app` to see the logs of an specific annotated service.

## How we filter out logs

- Based on namespace right now. Everything outside the deployed namespace is filtered out.

### If we need another broader filter

- We can filter in any of the followinf fields:


## Improve logs collection

- JSON logs are way easier for filebeat and logstash to process. Try to make this happen in each service. For example,
In the case of Payara services, run the following line to make json logs instead of custom format:

`asadmin set-log-attributes com.sun.enterprise.server.logging.GFFileHandler.formatter='fish.payara.enterprise.server.logging.JSONLogFormatter'`

- Avoid ANSII colors. 

- Avoid multiline in general. 


## Installing opensearch plugins

To install custom plugins in opensearch just add entries to the `opensearch.plugins` values. This will inject `opensearch-plugin install --batch plugin-name ` for each entry.

## Installing logstash plugins

To install custom output plugins in logstash just add entries to the `logstash.plugins` values. This will inject `logstash-plugin install plugin-name ` for each entry.
If installing in an air-gapped environment the only supported plugins are `logstash-output-datadog`, `logstash-output-http`, `logstash-output-tcp`, `logstash-output-lumberjack` and `logstash-output-elasticsearch`

## Snapshots through repositories

If the flag `opensearch.backup.enabled` is `true`, we create one repository and its corresponding CronJob to make regular snapshots. For example, the snippet below sends a 24h shift snapshot to the internal minIO artifact storage (notice this one is the default enabled one). Notice that, each repository is a map entry. Thus you can change values by setting them directly with `--set opensearch.backup.repositories.minio.schedule.cron="5 0 * * *"`.

```yaml
opensearch:
# Resource for backup, we use s3 since fs does not make much sense in this env
  backup:
    enabled: true
    # We might have multiple repositories, so we use a list
    repositories:
      minio2:
        snapshot_name: "opensearch"


        # The following map will be added as a config for the opensearch.yaml file using the s3-repository plugin
        # The format would be s3.client.rootName.keyName
        s3:
          endpoint: "minio.service.consul:9000"
          path_style_access: true
          protocol: "http"
          region: "eu-west-1"
        # The following map will be transformed into a json payload to call the /_snapshot API
        # See https://opensearch.org/docs/1.3/api-reference/snapshots/create-repository/ for more information about the parameters
        # So far we only enable the local minIO S3 artifact store
        settings:
          base_path: "opensearch-backup"
          bucket: "hopsworks"
          client: "minio"
          endpoint: "http://minio.service.consul:9000"
          region: "eu-west-1"
          

        credentials:
            secret_from: "minio-secret"

        # Schedule of the backup
        schedule: 
            # Run every 5 mins
            cron: "5 0 * * *"

        # This map will be converted to a json payload to call the /_snapshot API when creating a snapshot
        backup_payload:
          indices: "*"
          ignore_unavailable: true
          include_global_state: false
          partial: false

```

{{ template "chart.valuesSection" . }}