fluentd-s3

This custom built chart enables the collection and forwarding of logs from various Kubernetes components (e.g., kubelet, apiserver, docker logs) to AWS S3, with support for filtering, parsing, and metadata enrichment.

It consists of 5 main components:

1. ConfigMap (fluentd-config):

   • Contains Fluentd configuration files (fluent.conf, kubernetes.conf, system.conf, systemd.conf, kubernetes-containerd.conf, and custom audit log formats).
   • Includes several @include statements to load various configuration files depending on environment variables (e.g., FLUENTD_SYSTEMD_CONF, FLUENTD_PROMETHEUS_CONF).
   • The log forwarding is configured to send Kubernetes logs to an S3 bucket, with S3 settings configured through environment variables (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).

2. ServiceAccount:

   • Defines a service account (fluentd-s3) for Fluentd to use in the awss3 namespace.

3. ClusterRole:

   • Grants fluentd-s3 service account access to specific resources like pods, namespaces, and ingresses across the cluster. The role allows actions like get, list, and watch.

4. ClusterRoleBinding:

   • Binds the fluentd-s3 ClusterRole to the fluentd-s3 service account, enabling Fluentd to access resources as defined in the ClusterRole.

5. DaemonSet:

   • Deploys Fluentd as a DaemonSet to run on all nodes in the awss3 namespace.
   • Ensures that Fluentd runs with the appropriate service account (fluentd-s3) and has access to the necessary environment variables for AWS credentials and other configurations.
   • The init container (copy-fluentd-config) is used to copy configuration files from a read-only ConfigMap to the Fluentd container's writable directory (/fluentd/etc), since Fluentd requires write access to its configuration files.

Requirements

• Environment Variable in Use

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY: Provided via Kubernetes Secret.

  • S3_LOG_BUCKET_NAME and S3_LOG_BUCKET_REGION: Ensure these are set correctly in your environment or ConfigMap.

Installation

As this is a helm chart, installation is done via helm install or helm upgrade

• Perform dry run

  helm upgrade --install fluentds3 fluentd-s3 -f fluentd-s3/values.yaml --namespace=kube-system --dry-run

• Install

  helm upgrade --install fluentds3 fluentd-s3 -f fluentd-s3/values.yaml --namespace=kube-system