Example blog

Kubernetes logging guide: Learn how to log in Kubernetes


Computer system logs are essential to help engineers perform the following tasks:

  • Troubleshooting application performance bottlenecks
  • Determine the root cause of availability issues
  • Debug software code
  • Investigate security vulnerabilities
  • Infrastructure capacity planning
  • Maintaining Audit Compliance Requirements
  • … and more.

Modern cloud-based distributed applications rely heavily on container technology. These applications often rely on Kubernetes for container orchestration which provides transparent scaling and robust fault tolerance. With this shift in deployment infrastructure from bare metal to containers, logging and monitoring techniques have changed.

Storing logs in containers or virtual machines is not practical because both are ephemeral. Therefore, a different approach is needed to capture logs from Kubernetes-hosted applications.

This article is the first part of a guide that covers the fundamentals of Kubernetes Logging. In Part 1, we’ll introduce Kubernetes logging architectures and explain node-level logging patterns in detail.

Kubernetes logging basics

In a production Kubernetes environment, hundreds or even thousands of containers may exist in different states (running, stopping, restarting, or terminating) at any one time.

Logging drivers

Kubernetes does not provide a native solution for storing logs. However, it comes with some logging drivers to facilitate log storage and aggregation. By default, Kubernetes uses the json-file driver, which formats all log entries into JSON and caches them internally. The syslog and journald logging drivers help write to Linux logging systems. Other logging drivers such as FluentD, AWS CloudWatch, and GCP Logs make it easy to write logs to external log aggregation services.

Kubernetes system component logs

Kubernetes logs can come from the container orchestration system and containerized applications.

Kubernetes system components include the Kubernetes scheduler, kube-proxy, kubelet, and container runtime. Kubernetes scheduler and kube-proxy run in a container and always write logs to local /var/log directory regardless of the driver used. Both the kubelet and the container runtime write logs to the systemd newspaper if present or at /var/log directory if not.

Node level logging

In a Kubernetes environment, the node or cluster maintains application logs. In node-level logging, the pod writes application logs to the node it is running on. The container engine redirects any message from the application log to the stdout Where stderr streams. The node logging driver collects messages and writes them to the appropriate log file. If no driver is configured, the default logging is json-file.

You can use the following command to check which logs are using the default driver.

kubectl logs 

Node-level logging, however, has a significant shortcoming: a pod’s logs are lost if the node goes down or if Kubernetes evicts that pod.

Cluster Level Logging

Cluster-level logging involves a centralized logging service that aggregates logs from all nodes. This solves the problem of losing node-level logs by forwarding all node-level logs to a backend service. There are two ways to achieve cluster level logging.

The first method uses a node-level agent configured as a DaemonSet. A DaemonSet is a Kubernetes feature where each node (or some of them) runs a copy of the pod. In a later section of this guide, we’ll show how to use this approach for efficient logging.

The second method uses a sidecar model, where each pod contains a sidecar container that captures logs and sends those logs to an external service.

Capturing logs using a node-level logging agent

A node-level logging agent is an always-on service that runs in a container in a separate pod.

A node-level logging agent is useful when your environment meets the following conditions:

  • There is a resource constraint in the nodes. A single node-level agent can capture logs from all application pods on that node, ensuring that resource consumption remains low.
  • There is a common log storage for all Pods.
  • There is a need for basic multi-tenant isolation without complex requirements. Separate resource allocation is not possible here because a logging agent takes care of all application pods.

Using a node-level logging agent has several advantages:

  • It is easy to install.
  • Resource usage is low.
  • Since the agent uses the same mechanism as the default logging configuration, logs are available through the kubectl ordered.
  • There is no need to modify anything in the app’s Pods.

Configure the node-level logging agent in Kubernetes

The Kubernetes node-level agent uses the DaemonSet pattern. A DaemonSet runs a copy of a specific Pod in each cluster node (or some of its nodes). Each time you add a new node to the cluster, this pod is also added to the new node. When you delete a node (or the cluster shrinks), Kubernetes automatically reclaims that pod. The DaemonSet feature lets you run a pod to collect logs from all pods on a node and send them to a log management service.

To capture logs from a specific directory, developers can use an open source log collection agent like Fluentd. Fluentd has a GitHub repository that contains example configurations for different logging backends.

Suppose we want to send the DaemonSet Pod logs collected by Fluentd to an Elasticsearch backend. We would use this example configuration from the GitHub repository.

The first step in deploying this configuration is to create a service account for the Fluentd logging agent. The code snippet below shows how to do this.

apiVersion: v1
kind: ServiceAccount
  name: fluentd-logger-daemonset
  namespace: logging
    app: fluentd-logger-daemonset

Save file as fluentd_service_account.yamlthen run the below command to create the service account.

kubectl apply -f fluentd_service_account.yaml

Next, create the configuration for the DaemonSet. You can use the following configuration file to do this.

apiVersion: extensions/v1beta1
kind: DaemonSet
  name: fluentd
  namespace: kube-system
    k8s-app: fluentd-elasticsearch-daemonset
        k8s-app: fluentd-elasticsearch-daemonset
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
          value: ""
          value: ""
        - name: logdir
          mountPath: /var/log
      - name: logdir
          path: /var/log

The important part of this specification is the kind attribute that specifies that the pod will run as DaemonSet. The spec The template field is mandatory and defines the base image for creating the DaemonSet. This configuration uses the default fluentd image as base image. Like all Kubernetes configuration files, it is necessary to specify the apiVersion and the metadata keys. Application container logs are mounted on the /var/log phone book.

You can save this configuration as fluentd_daemonset_account.yaml and run the below command to create the pod in all nodes of the cluster.

kubectl apply -f fluentd-daemonset.yaml

This will launch a node-level logging agent pod in all your cluster nodes. It is possible to run the pod only on specific nodes using the nodeSelector parameter (under spec.template.spec). Without this parameter, Kubeternetes creates the Pod in each node.


Troubleshooting containerized microservices can be tricky. Gone are the days when an engineer could log into each physical server and track logs from the command shell. Compared to logs from other clustered applications, Kubernetes logs, both from the orchestrator and from the application, are difficult to manage because pods are transient. This is why you should collect logs from each node and send them to a central location outside of the cluster for persistence and further analysis.

As we’ve seen in this guide, a DaemonSet template makes it quick and easy to implement node-level logging agents in Kubernetes. You don’t have to make any changes to the app and its resource usage is low. However, this solution does not provide separate log storage or full multi-tenant isolation capabilities for each application. For these features, you will need to use a sidecar-based approach.

In the second part of our guide, we will discuss sidecar implementation, specifically its pros and cons, and common use cases.

Record everything, answer everything – for free

Falcon LogScale Community Edition (formerly Humio) offers a modern, free log management platform for the cloud. Leverage streaming data ingestion to gain instant visibility into distributed systems and prevent and resolve incidents.

Falcon LogScale Community Edition, available instantly at no cost, includes the following:

  • Ingest up to 16 GB per day
  • 7 days retention
  • No credit card needed
  • Continuous access without trial period
  • Indexless logging, real-time alerts, and live dashboards
  • Access our marketplace and packages, including guides for creating new packages
  • Learn and collaborate with an active community

Start for free

Source link