Amazon EKS 0.56.3

EKS Container Logs Module

View SourceRelease Notes

This Terraform Module installs and configures aws-for-fluent-bit on an EKS cluster, so that each node runs fluent-bit to collect the logs and ship to CloudWatch Logs, Kinesis Streams, or Kinesis Firehose.

This module uses the community helm chart, with a set of best practices inputs.

This module is for setting up log aggregation for EKS Pods on EC2 workers (self-managed or managed node groups). For Fargate pods, take a look at the eks-fargate-container-logs module.

How does this work?

This module solves the problem of unifying the log streams in your Kubernetes cluster to be shipped to an aggregation service on AWS (CloudWatch Logs, Kinesis, or Firehose) so that you have a single interface to search and monitor your logs. To achieve this, the module installs a service (fluent-bit) that monitors the log files on the filesystem, parses custom log formats into a unified format, and ships the result to a centralized log aggregation service (CloudWatch).

fluent-bit is installed as a Kubernetes DaemonSet, which ensures that there is one fluent-bit Pod running per node. In this way, we are able to ensure that all workers in the cluster are running the fluent-bit service for shipping the logs into CloudWatch.

You can read more about fluent-bit in their official home page. You can also learn more about CloudWatch logging in the official AWS docs.

What is the difference with fluentd?

fluent-bit is an optimized version of fluentd that focuses on streaming and aggregating log files. fluentd has a larger ecosystem of plugins that enable various processing capabilities on top of the logs prior to aggregating in the data store.

For most EKS deployments, it is recommended to use this fluent-bit module for container log aggregation. Unless you have a specific need for a plugin only supported by fluentd, the superior performance and memory footprint of fluent-bit will ensure resources are available on your EKS workers for your Pods.

Log format

This module leverages native plugins for Kubernetes built into fluent-bit that extract additional metadata for each Pod that is reporting. Each log is shipped to the respective outputs in the following structure:

{
    "kubernetes": {
        "namespace_name": "NAMESPACE_WHERE_POD_LOCATED",
        "pod_name": "NAME_OF_POD_EMITTING_LOG",
        "pod_id": "ID_IN_KUBERNETES_OF_POD",
        "container_hash": "KUBERNETES_GENERATED_HASH_OF_CONTAINER_EMITTING_LOG",
        "container_name": "NAME_OF_CONTAINER_IN_POD_EMITTING_LOG",
        "docker_id": "ID_IN_DOCKER_OF_CONTAINER",
        "host": "NODE_NAME_OF_HOST_EMITTING_LOG",
        "labels": {
            "KEY": "VALUE",
        },
        "annotations": {
            "KEY": "VALUE"
        }
    },
    "log": "CONTENTS_OF_LOG_MESSAGE",
    "stream": "STDERR_OR_STDOUT",
    "time": "TIMESTAMP_OF_LOG"
}

This allows you to filter and search the logs by the respective attributes. For example, the following CloudWatch Insights Query can be used to search for all logs from Pods in the kube-system Namespace:

fields @timestamp, @message
| filter kubernetes.namespace_name = "kube-system"
| sort @timestamp desc
| limit 20

Sample Usage

Terraform

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S EKS-CONTAINER-LOGS MODULE
# ------------------------------------------------------------------------------------------------------

module "eks_container_logs" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-container-logs?ref=v0.56.3"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Configuration for using the IAM role with Service Accounts feature to provide
  # permissions to the helm charts. This expects a map with two properties:
  # `openid_connect_provider_arn` and `openid_connect_provider_url`. The
  # `openid_connect_provider_arn` is the ARN of the OpenID Connect Provider for EKS
  # to retrieve IAM credentials, while `openid_connect_provider_url` is the URL. Set
  # to null if you do not wish to use IAM role with Service Accounts.
  iam_role_for_service_accounts_config = <INPUT REQUIRED>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Can be used to add more inputs. This string should be formatted according to
  # Fluent Bit docs, as it will be injected directly into the fluent-bit.conf file.
  additional_inputs = ""

  # Configurations for forwarding logs to AWS managed Elasticsearch. Set to null if
  # you do not wish to forward the logs to ES.
  aws_elasticsearch_configuration = null

  # The version of the aws-for-fluent-bit helm chart to deploy. Note that this is
  # different from the app/container version (use var.aws_for_fluent_bit_version to
  # control the app/container version).
  aws_for_fluent_bit_chart_version = "0.1.15"

  # The Container repository to use for looking up the aws-for-fluent-bit Container
  # image when deploying the pods. When null, uses the default repository set in the
  # chart.
  aws_for_fluent_bit_image_repository = null

  # Which version of aws-for-fluent-bit to install. When null, uses the default
  # version set in the chart.
  aws_for_fluent_bit_version = null

  # The AWS partition used for default AWS Resources.
  aws_partition = "aws"

  # Configurations for forwarding logs to CloudWatch Logs. Set to null if you do not
  # wish to forward the logs to CloudWatch Logs.
  cloudwatch_configuration = null

  # Create a dependency between the resources in this module to the interpolated
  # values in this list (and thus the source resources). In other words, the
  # resources in this module will now depend on the resources backing the values in
  # this list such that those resources need to be created before the resources in
  # this module, and the resources in this module need to be destroyed before the
  # resources in the list.
  dependencies = []

  # Can be used to provide custom filtering of the log output. This string should be
  # formatted according to Fluent Bit docs, as it will be injected directly into the
  # fluent-bit.conf file.
  extra_filters = ""

  # Can be used to fan out the log output to multiple additional clients beyond the
  # AWS ones. This string should be formatted according to Fluent Bit docs, as it
  # will be injected directly into the fluent-bit.conf file.
  extra_outputs = ""

  # Configurations for forwarding logs to Kinesis Firehose. Set to null if you do
  # not wish to forward the logs to Firehose.
  firehose_configuration = null

  # Used to name IAM roles for the service account. Recommended when
  # var.iam_role_for_service_accounts_config is configured.
  iam_role_name_prefix = null

  # Configurations for forwarding logs to Kinesis stream. Set to null if you do not
  # wish to forward the logs to Kinesis.
  kinesis_configuration = null

  # Configure affinity rules for the Pod to control which nodes to schedule on. Each
  # item in the list should be a map with the keys `key`, `values`, and `operator`,
  # corresponding to the 3 properties of matchExpressions. Note that all expressions
  # must be satisfied to schedule on the node.
  pod_node_affinity = []

  # Specify the resource limits and requests for the fluent-bit pods. Set to null
  # (default) to use chart defaults.
  pod_resources = null

  # Configure tolerations rules to allow the Pod to schedule on nodes that have been
  # tainted. Each item in the list specifies a toleration rule.
  pod_tolerations = []

  # Optionally use a cri parser instead of the default Docker parser. This should be
  # used for EKS v1.24 and later.
  use_cri_parser_conf = true

  # When true, all IAM policies will be managed as dedicated policies rather than
  # inline policies attached to the IAM roles. Dedicated managed policies are
  # friendlier to automated policy checkers, which may scan a single resource for
  # findings. As such, it is important to avoid inline policies when targeting
  # compliance with various security standards.
  use_managed_iam_policies = true

}

Terragrunt

terraform.tfvars

# Coming soon!