Skip to content

Kubernetes Log Forwarding Configuration

This guide tells you how to forward logs from Kubernetes clusters to SenseOn. The examples focus on Amazon EKS. This guide is for customers who want to ingest logs from:

  • Amazon EKS (Elastic Kubernetes Service)
  • Google GKE (Google Kubernetes Engine)
  • Azure AKS (Azure Kubernetes Service)
  • Self-hosted Kubernetes

Deployment Requirements

For Kubernetes logs, SenseOn uses a FluentBit → external collector architecture. You do the configuration yourself, so you keep control of log collection. There are two paths. Select the path for your start point:


Deploying FluentBit From Scratch

Use this section if FluentBit, Fluentd or any other log collector does not run in your cluster yet. This section deploys FluentBit as its own DaemonSet, next to your existing applications.

Architecture:

Application Pods → FluentBit (new DaemonSet) → SenseOn Collector

Prerequisites

  • Cluster admin access to create namespaces, RBAC, ConfigMaps, and DaemonSets
  • Outbound HTTPS (443) access to *.senseon.io
  • Outbound access to Docker Hub, to pull the fluent/fluent-bit image
  • Applications that log JSON to stdout

Step 1: Deploy the namespace, RBAC, ConfigMap, and DaemonSet

Save the YAML that follows as fluentbit-deployment.yaml. Replace <your-cluster-name> and <your-code> with your own values:

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 5
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Kube_Tag_Prefix kube.var.log.containers.
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off

    [FILTER]
        Name modify
        Match kube.*
        Add cluster_name <your-cluster-name>
        Add environment production
        Add source kubernetes

    [OUTPUT]
        Name            http
        Match           kube.*
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2.3
          resources:
            requests:
              memory: "128Mi"
              cpu: "150m"
            limits:
              memory: "128Mi"
              cpu: "150m"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluent-bit-config

Then deploy it:

kubectl apply -f fluentbit-deployment.yaml

kubectl get pods -n logging
kubectl -n logging rollout status daemonset/fluent-bit

No /var/lib/docker/containers mount: This DaemonSet mounts only /var/log. On containerd-based clusters (the default for EKS, GKE and AKS today), /var/lib/docker/containers does not exist. Also, the FluentBit tail input never reads that directory directly. It follows /var/log/containers/*.log. The kubelet keeps symlinks there to the location where the container runtime actually stores the logs.

To learn why the ConfigMap has this shape, refer to Understanding Kubernetes Log Parsing. Pay special attention to the note that Tag must match Kube_Tag_Prefix.

Step 2: Verify FluentBit is running and forwarding

kubectl -n logging logs -l app=fluent-bit --tail=100 | grep "HTTP status"

# Should see: HTTP status=200

Step 3: Test end-to-end with a sample application

Deploy a small pod that writes JSON logs on a timer. Then make sure that FluentBit collects them:

kubectl run log-generator --image=busybox --restart=Never -- sh -c \
  'while true; do
    echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"event\":\"heartbeat\",\"level\":\"info\",\"service\":\"log-generator\"}";
    sleep 5;
  done'

kubectl logs log-generator --tail=10

# FluentBit's own logs report HTTP status, not record content - use the
# same check as Step 2 to confirm this test app's events round-tripped
kubectl -n logging logs -l app=fluent-bit --tail=50 | grep "HTTP status"

When you see clean HTTP status=200 responses for these test events, delete the test pod:

kubectl delete pod log-generator

New applications usually show as Unclassified in SenseOn until the schema of your log source is onboarded. First make sure that FluentBit forwards the logs correctly. Then send a sanitised sample. SenseOn does the work that remains. Send the sample to support@senseon.io.

For help with parser problems, TLS errors and high memory use, refer to Troubleshooting. That section covers both deployment paths.


Configuring Log Forwarding

Overview

If you already deployed FluentBit (or Fluentd) in your cluster, add only a new OUTPUT block to forward logs to SenseOn.

Architecture:

Application Pods → FluentBit (already present) → Add OUTPUT → SenseOn Collector

Prerequisites

  • Existing FluentBit or Fluentd deployment
  • Cluster admin access to modify ConfigMaps
  • Outbound HTTPS (443) access to *.senseon.io (your collector endpoint is <your-code>-logs.senseon.io)
  • Applications that log JSON to stdout (or configured FluentBit parsers)

Step 1: Identify Your FluentBit Configuration

Find your existing FluentBit ConfigMap:

# Find FluentBit namespace
kubectl get pods --all-namespaces | grep fluent

# Get the ConfigMap
kubectl get configmap -n <fluent-namespace>
kubectl get configmap fluent-bit-config -n <fluent-namespace> -o yaml

Step 2: Add SenseOn OUTPUT Block

Edit your FluentBit ConfigMap to add a new OUTPUT section:

kubectl edit configmap fluent-bit-config -n <fluent-namespace>

Add a new OUTPUT block to the outputs: section:

outputs: |
    # Your existing outputs (keep these)
    [OUTPUT]
        Name          forward
        Match         *
        Host          your-internal-aggregator.example.com
        Port          24224

    # NEW: Add this OUTPUT for SenseOn
    [OUTPUT]
        Name            http
        Match           *
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On

settings explained:

  • Match * - Forwards all logs. To forward logs from specific apps only, you can filter with Match kube.*app-name*.
  • Format json - Sends each flush as a single JSON array. The SenseOn collector accepts one JSON value per request. It does not accept newline-delimited json_lines. With newline-delimited JSON, a request stops with an HTTP 400 error intermittently, when a flush has more than one record.
  • json_date_key false - Let SenseOn add timestamps server-side
  • tls On - Encrypt traffic (mandatory)

Step 3: Restart FluentBit Pods

# Restart FluentBit to apply configuration
kubectl rollout restart daemonset/fluent-bit -n <fluent-namespace>

# Verify pods are running
kubectl get pods -n <fluent-namespace>

# Check pod logs for errors
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=50

Step 4: Verify Logs Are Being Sent

# Check FluentBit logs for HTTP status
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=100 | grep "HTTP status"

# Should see: HTTP status=200 (logs accepted successfully)

Example successful output:

[2025/10/21 10:00:05] [ info] [output:http:http.1] <your-code>-logs.senseon.io:443, HTTP status=200

Step 5: Test with Sample Application

To do an end-to-end check, deploy a test pod that generates JSON logs:

kubectl run test-logger --image=busybox --restart=Never -- sh -c \
  'while true; do echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test log\",\"source\":\"test-pod\"}"; sleep 5; done'

# Check test pod logs
kubectl logs test-logger

# Verify FluentBit picked them up
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=20 | grep test-pod

Understanding Kubernetes Log Parsing

The Problem: Nested JSON in Kubernetes

Kubernetes wraps container logs in a Docker/containerd format. As a result, the application JSON is nested and escaped.

What your application outputs:

{"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}

What Kubernetes writes to disk:

2025-10-21T10:00:00.123456789Z stdout F {"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}

If the logs are not parsed correctly, SenseOn receives:

{
  "log": "{\"timestamp\": \"2025-10-21T10:00:00Z\", \"level\": \"info\", \"message\": \"user login\", \"user_id\": \"123\"}",
  "stream": "stdout",
  "time": "2025-10-21T10:00:00.123456789Z"
}

Your application JSON is escaped in the log field. ❌

The Solution: Multi-Stage Parsing

FluentBit parses Kubernetes logs in more than one stage to unwrap them:

Stage 1: Parse Container Runtime Format

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  cri
    Tag               kube.*

The cri parser processes the Docker/containerd format, and it extracts the actual log message.

Stage 2: Parse Application JSON

[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Keep_Log Off

The kubernetes filter with Merge_Log On parses JSON from the extracted message and merges fields.

Stage 3: Clean Up and Enrich

[FILTER]
    Name modify
    Match kube.*
    Add source kubernetes
    Add cluster_name production-eks

Add metadata to show the source.

Final result sent to SenseOn:

{
  "timestamp": "2025-10-21T10:00:00Z",
  "level": "info",
  "message": "user login",
  "user_id": "123",
  "source": "kubernetes",
  "cluster_name": "production-eks",
  "stream": "stdout",
  "kubernetes": {
    "pod_name": "api-server-abc123",
    "namespace_name": "production",
    "container_name": "api-server",
    "labels": { "app": "api-server" }
  }
}

The Kubernetes filter puts its metadata in a kubernetes object, for example kubernetes.pod_name and kubernetes.namespace_name. It does not flatten the metadata into top-level kubernetes_* fields. Real forwarded records show this behaviour.


Example Configuration Reference

Complete FluentBit Configuration for Kubernetes

This complete example configuration shows best practices for Kubernetes log collection:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 5
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    # Collect logs from nginx ingress controller
    [INPUT]
        Name tail
        Path /var/log/containers/ingress-nginx-controller*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    # Collect logs from application pods
    [INPUT]
        Name tail
        Path /var/log/containers/api-server*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    # Enrich with Kubernetes metadata and parse JSON logs
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Kube_Tag_Prefix kube.var.log.containers.
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off

    # Add cluster identification
    [FILTER]
        Name modify
        Match kube.*
        Add cluster_name production-eks-01
        Add environment production
        Add source kubernetes

    # Forward to SenseOn collector
    [OUTPUT]
        Name            http
        Match           kube.*
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On

Tag must match Kube_Tag_Prefix: Both [INPUT] blocks in this example use Tag kube.*. Thus FluentBit puts the path of each tailed file into its tag. This tag matches Kube_Tag_Prefix kube.var.log.containers. in the filter. You can give an INPUT a fixed tag (for example, Tag kube.api) that does not contain the path of the tailed file. In this case, the tag does not match Kube_Tag_Prefix. The Kubernetes filter then silently skips each record from that INPUT.

The result is no JSON merge and no kubernetes.* metadata. The collector gives no error. It still returns 200, because at that point the record is an ordinary JSON object. If the metadata does not show as you expect, look for skip filter warnings in the logs of FluentBit.

Use_Kubelet needs hostNetwork: Use_Kubelet On queries the kubelet API of the node directly. This works only if the FluentBit pod runs with hostNetwork: true. If not, you see kubelet upstream connection error in the logs of FluentBit. If your DaemonSet does not set hostNetwork: true (the example that follows does not), do not use Use_Kubelet/Kubelet_Port. The Kubernetes filter then queries the API server, with the ServiceAccount token of the pod. A special network configuration is not necessary for this method.

DaemonSet Deployment Example

If FluentBit does not run in your cluster yet, refer to the deployment of a new FluentBit near the start of this page. That section has a complete, tested example of the namespace, RBAC, ConfigMap and DaemonSet.


Verification and Testing

Check FluentBit is Running

# List FluentBit pods (one per node)
kubectl get pods -n logging -l app=fluent-bit

# Check pod status
kubectl describe pod -n logging <fluent-bit-pod-name>

# View logs
kubectl logs -n logging <fluent-bit-pod-name> --tail=100

Verify Log Collection

# Check FluentBit is reading container logs
kubectl logs -n logging -l app=fluent-bit --tail=100 | grep "inotify_fs_add"

# Should see lines like:
# [ info] [input:tail:tail.0] inotify_fs_add(): inode=12345 name=/var/log/containers/api-server-xyz.log

Verify Forwarding to SenseOn

# Check for successful HTTP POSTs
kubectl logs -n logging -l app=fluent-bit --tail=200 | grep "HTTP status"

# Should see:
# [ info] [output:http:http.0] <your-code>-logs.senseon.io:443, HTTP status=200

Generate Test Logs

Deploy a test application that generates JSON logs:

# Create test pod
kubectl run log-generator --image=busybox --restart=Never -- sh -c \
  'while true; do
    echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test event\",\"request_id\":\"$(date +%s)\"}";
    sleep 10;
  done'

# Check test pod logs
kubectl logs log-generator --tail=10

# Verify FluentBit collected them
kubectl logs -n logging -l app=fluent-bit --tail=50 | grep "test event"

Troubleshooting

FluentBit Pods Not Starting

Examine the pod events:

kubectl describe pod -n logging <fluent-bit-pod-name>

Common issues:

  1. RBAC permissions missing

    • Make sure that the ServiceAccount, ClusterRole and ClusterRoleBinding exist
    • Make sure that the DaemonSet refers to the ServiceAccount
  2. ConfigMap not mounted

    • Make sure that the ConfigMap exists: kubectl get configmap -n logging
    • Examine the volume mount in the DaemonSet spec
  3. Resource constraints

    • Examine the node resources: kubectl describe node
    • Adjust the memory/CPU limits in the DaemonSet

No Logs Being Collected

Examine the INPUT configuration:

kubectl logs -n logging -l app=fluent-bit --tail=100 | grep -i "input"

Make sure that the log paths exist:

# Exec into a FluentBit pod
kubectl exec -it -n logging <fluent-bit-pod-name> -- sh

# Check log directory
ls -la /var/log/containers/

# Should see *.log files

Common issues:

  1. Wrong log path in INPUT
    • The path must match the actual locations of the container logs
    • Usually /var/log/containers/*.log
  2. Permissions issues
    • FluentBit must have read access to /var/log
    • Examine the hostPath volume mounts

Logs Not Reaching SenseOn

Examine the OUTPUT configuration:

kubectl logs -n logging -l app=fluent-bit --tail=200 | grep -i "output"

Test the collector endpoint:

# From within cluster
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -v -X POST https://<your-code>-logs.senseon.io \
  -H "Content-Type: application/json" \
  -d '{"test":"connection"}'

# Should see HTTP 200

Common issues:

  1. Network policy blocks egress
    • Examine the network policies: kubectl get networkpolicy --all-namespaces
    • Make sure that the policies allow HTTPS (443) egress
  2. Wrong collector URL
    • Ask SenseOn support to do a check of the URL
    • Your endpoint is <your-code>-logs.senseon.io. Look for typos
  3. TLS certificate issues
    • Look for TLS errors in the FluentBit logs
    • Make sure that tls.verify On is set

Logs Are Nested/Escaped JSON

This means that the logs are not parsed correctly.

Look for these items in your configuration:

  1. INPUT must use multiline.parser cri:

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser cri    # ← This is required
    

  2. Kubernetes filter must have Merge_Log On:

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On    # ← This parses and merges JSON
        Keep_Log Off     # ← This removes the raw log field
    

  3. Tag must match Kube_Tag_Prefix: Your INPUT can use a fixed tag (for example, Tag kube.api) and not Tag kube.*. If that tag does not contain the path of the tailed file, the Kubernetes filter silently skips each record. To find this problem, look for skip filter warnings in the logs of FluentBit. Merge_Log On never runs. Thus the logs stay nested, and the collector gives no error. It still returns 200.

  4. If logs are still nested, add an explicit JSON parser:

    [FILTER]
        Name parser
        Match kube.*
        Key_Name log
        Parser json
        Reserve_Data True
    

High Memory Usage

Examine the FluentBit resource usage:

kubectl top pods -n logging

Reduce memory usage:

  1. Lower buffer limits in INPUT:

    [INPUT]
        Mem_Buf_Limit 5MB    # Default is 32MB
    

  2. Increase flush frequency:

    [SERVICE]
        Flush 5    # Flush every 5 seconds (default is 1)
    

  3. Filter out verbose logs:

    [FILTER]
        Name grep
        Match kube.*
        Exclude log debug|trace
    

Performance Tuning

For high-volume environments:

[SERVICE]
    Flush 1              # Flush every second
    Grace 5              # Wait 5 seconds on shutdown
    Log_Level warning    # Reduce FluentBit's own logging

[INPUT]
    Mem_Buf_Limit 32MB   # Increase buffer size
    Skip_Long_Lines On   # Skip lines > 32KB

[OUTPUT]
    Workers 2            # Parallel HTTP connections
    Retry_Limit 3        # Max retry attempts
    net.keepalive On     # Reuse connections

Cloud-Specific Notes

Amazon EKS

Cluster access:

# Configure kubectl
aws eks update-kubeconfig --region <region> --name <cluster-name>

# Verify access
kubectl get nodes

EKS-specific considerations:

  • Make sure that IAM roles allow pod-to-pod communication
  • Make sure that VPC security groups allow outbound HTTPS
  • Make sure that the nodes have internet access (direct or through a NAT gateway)

Google GKE

Cluster access:

# Configure kubectl
gcloud container clusters get-credentials <cluster-name> --region=<region>

# Verify access
kubectl get nodes

GKE-specific considerations:

  • Workload Identity can be necessary for private clusters
  • Make sure that firewall rules allow egress to *.senseon.io
  • Binary Authorization can block the FluentBit image. If it does, add an exception

Azure AKS

Cluster access:

# Configure kubectl
az aks get-credentials --resource-group <rg> --name <cluster-name>

# Verify access
kubectl get nodes

AKS-specific considerations:

  • Make sure that Network Security Groups allow outbound HTTPS
  • Azure Policy can make specific pod security standards necessary
  • Make sure that DNS resolution for *.senseon.io works

Need help?

Contact SenseOn support at support@senseon.io

When you contact support, include this information:

  • Your organisation name
  • Collector endpoint URL
  • Example log entries (sanitised)
  • Relevant error messages (if applicable)
  • Log source type
  • Configuration files (if applicable)