Skip to content

Kubernetes Log Forwarding Configuration

This guide covers log forwarding from Kubernetes clusters to SenseOn, with examples focused on Amazon EKS. This is designed to help customers who seek log ingestion from:

  • Amazon EKS (Elastic Kubernetes Service)
  • Google GKE (Google Kubernetes Engine)
  • Azure AKS (Azure Kubernetes Service)
  • Self-hosted Kubernetes

Deployment Requirements

SenseOn's Kubernetes log forwarding uses a FluentBit → external collector architecture. Self-service configuration, so you keep control of log collection. There are two paths, depending on your starting point:

Deploying FluentBit From Scratch

Use this section if you don't have FluentBit, Fluentd, or any other log collector running in your cluster yet. It deploys FluentBit as its own DaemonSet, alongside your existing applications.

Architecture: 

Application Pods → FluentBit (new DaemonSet) → SenseOn Collector

Prerequisites

  • Cluster admin access to create namespaces, RBAC, ConfigMaps, and DaemonSets
  • Outbound HTTPS (443) access to *.senseon.io
  • Outbound access to Docker Hub, to pull the fluent/fluent-bit image
  • Your applications logging JSON to stdout

Step 1: Deploy the namespace, RBAC, ConfigMap, and DaemonSet

Save the following as fluentbit-deployment.yaml, replacing <your-cluster-name> and <your-code> with your own values:

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 5
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Kube_Tag_Prefix kube.var.log.containers.
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off

    [FILTER]
        Name modify
        Match kube.*
        Add cluster_name <your-cluster-name>
        Add environment production
        Add source kubernetes

    [OUTPUT]
        Name            http
        Match           kube.*
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2.3
          resources:
            requests:
              memory: "128Mi"
              cpu: "150m"
            limits:
              memory: "128Mi"
              cpu: "150m"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluent-bit-config

Then deploy it:

kubectl apply -f fluentbit-deployment.yaml

kubectl get pods -n logging
kubectl -n logging rollout status daemonset/fluent-bit

No /var/lib/docker/containers mount: this DaemonSet only mounts /var/log. On containerd-based clusters (the default for EKS, GKE, and AKS today) /var/lib/docker/containers doesn't exist, and FluentBit's tail input never reads it directly regardless — it follows /var/log/containers/*.log, which the kubelet keeps symlinked to wherever the container runtime actually stores logs.

See Understanding Kubernetes Log Parsing below for why the ConfigMap is shaped this way, particularly the note on Tag needing to match Kube_Tag_Prefix.

Step 2: Verify FluentBit is running and forwarding

kubectl -n logging logs -l app=fluent-bit --tail=100 | grep "HTTP status"

# Should see: HTTP status=200

Step 3: Test end-to-end with a sample application

Deploy a small pod that emits JSON logs on a timer, and confirm FluentBit picks it up:

kubectl run log-generator --image=busybox --restart=Never -- sh -c \
  'while true; do
    echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"event\":\"heartbeat\",\"level\":\"info\",\"service\":\"log-generator\"}";
    sleep 5;
  done'

kubectl logs log-generator --tail=10

# FluentBit's own logs report HTTP status, not record content - use the
# same check as Step 2 to confirm this test app's events round-tripped
kubectl -n logging logs -l app=fluent-bit --tail=50 | grep "HTTP status"

Once you see clean HTTP status=200 responses for these test events, tear down the test pod:

kubectl delete pod log-generator

New applications typically show as Unclassified in SenseOn until we've onboarded your log source's schema — contact support@senseon.io with a sanitised sample once you've confirmed forwarding is working, and we'll take it from there.

For troubleshooting (parsing issues, TLS errors, high memory usage), see Troubleshooting below — it covers both deployment paths.

Configuring Log Forwarding

Overview

If you already have FluentBit (or Fluentd) deployed in your cluster, you just need to add a new OUTPUT block to forward logs to SenseOn.

Architecture: 

Application Pods → FluentBit (already present) → Add OUTPUT → SenseOn Collector

Prerequisites

  • Existing FluentBit or Fluentd deployment
  • Cluster admin access to modify ConfigMaps
  • Outbound HTTPS (443) access to *.senseon.io (your collector endpoint is <your-code>-logs.senseon.io)
  • Your applications logging JSON to stdout (or FluentBit parsers configured)

Step 1: Identify Your FluentBit Configuration

Find your existing FluentBit ConfigMap:

# Find FluentBit namespace
kubectl get pods --all-namespaces | grep fluent

# Get the ConfigMap
kubectl get configmap -n <fluent-namespace>
kubectl get configmap fluent-bit-config -n <fluent-namespace> -o yaml

Step 2: Add SenseOn OUTPUT Block

Edit your FluentBit ConfigMap to add a new OUTPUT section:

kubectl edit configmap fluent-bit-config -n <fluent-namespace>

Add a new OUTPUT block to the outputs: section:

outputs: |
    # Your existing outputs (keep these)
    [OUTPUT]
        Name          forward
        Match         *
        Host          your-internal-aggregator.example.com
        Port          24224

    # NEW: Add this OUTPUT for SenseOn
    [OUTPUT]
        Name            http
        Match           *
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On

settings explained:

  • Match * - Forwards all logs (you can filter with Match kube.*app-name* for specific apps)
  • Format json - Sends each flush as a single JSON array (SenseOn's collector accepts one JSON value per request; it does not accept newline-delimited json_lines, which will intermittently fail with an HTTP 400 as soon as a flush batches more than one record)
  • json_date_key false - Let SenseOn add timestamps server-side
  • tls On - Encrypt traffic (required)

Step 3: Restart FluentBit Pods

# Restart FluentBit to apply configuration
kubectl rollout restart daemonset/fluent-bit -n <fluent-namespace>

# Verify pods are running
kubectl get pods -n <fluent-namespace>

# Check pod logs for errors
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=50

Step 4: Verify Logs Are Being Sent

# Check FluentBit logs for HTTP status
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=100 | grep "HTTP status"

# Should see: HTTP status=200 (logs accepted successfully)

Example successful output: 

[2025/10/21 10:00:05] [ info] [output:http:http.1] <your-code>-logs.senseon.io:443, HTTP status=200

Step 5: Test with Sample Application

If you want to verify end-to-end, deploy a test pod that generates JSON logs:

kubectl run test-logger --image=busybox --restart=Never -- sh -c \
  'while true; do echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test log\",\"source\":\"test-pod\"}"; sleep 5; done'

# Check test pod logs
kubectl logs test-logger

# Verify FluentBit picked them up
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=20 | grep test-pod

Understanding Kubernetes Log Parsing

The Problem: Nested JSON in Kubernetes

Kubernetes wraps container logs in a Docker/containerd format, which causes application JSON to be nested and escaped.

What your application outputs: 

{"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}

What Kubernetes writes to disk: 

2025-10-21T10:00:00.123456789Z stdout F {"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}

Without proper parsing, SenseOn would receive: 

{
  "log": "{\"timestamp\": \"2025-10-21T10:00:00Z\", \"level\": \"info\", \"message\": \"user login\", \"user_id\": \"123\"}",
  "stream": "stdout",
  "time": "2025-10-21T10:00:00.123456789Z"
}

Your application JSON is escaped inside the log field! ❌

The Solution: Multi-Stage Parsing

FluentBit uses multi-stage parsing to unwrap Kubernetes logs:

Stage 1: Parse Container Runtime Format 

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  cri
    Tag               kube.*

The cri parser handles Docker/containerd format, extracting the actual log message.

Stage 2: Parse Application JSON 

[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Keep_Log Off

The kubernetes filter with Merge_Log On parses JSON from the extracted message and merges fields.

Stage 3: Clean Up and Enrich 

[FILTER]
    Name modify
    Match kube.*
    Add source kubernetes
    Add cluster_name production-eks

Add metadata to identify the source.

Final result sent to SenseOn: 

{
  "timestamp": "2025-10-21T10:00:00Z",
  "level": "info",
  "message": "user login",
  "user_id": "123",
  "source": "kubernetes",
  "cluster_name": "production-eks",
  "stream": "stdout",
  "kubernetes": {
    "pod_name": "api-server-abc123",
    "namespace_name": "production",
    "container_name": "api-server",
    "labels": { "app": "api-server" }
  }
}

The Kubernetes filter nests its metadata under a kubernetes object (kubernetes.pod_name, kubernetes.namespace_name, and so on) rather than flattening it to top-level kubernetes_* fields — as confirmed directly against real forwarded records.

Example Configuration Reference

Complete FluentBit Configuration for Kubernetes

This is a complete example configuration showing best practices for Kubernetes log collection:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 5
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    # Collect logs from nginx ingress controller
    [INPUT]
        Name tail
        Path /var/log/containers/ingress-nginx-controller*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    # Collect logs from application pods
    [INPUT]
        Name tail
        Path /var/log/containers/api-server*.log
        multiline.parser cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    # Enrich with Kubernetes metadata and parse JSON logs
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Kube_Tag_Prefix kube.var.log.containers.
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off

    # Add cluster identification
    [FILTER]
        Name modify
        Match kube.*
        Add cluster_name production-eks-01
        Add environment production
        Add source kubernetes

    # Forward to SenseOn collector
    [OUTPUT]
        Name            http
        Match           kube.*
        Host            <your-code>-logs.senseon.io
        Port            443
        URI             /
        Format          json
        json_date_key   false
        tls             On
        tls.verify      On
        Header          Content-Type application/json
        Retry_Limit     3
        net.keepalive   On

Tag must match Kube_Tag_Prefix: both [INPUT] blocks above use Tag kube.* so FluentBit embeds each tailed file's own path into its tag, matching Kube_Tag_Prefix kube.var.log.containers. below. If you instead give an INPUT a fixed tag (e.g. Tag kube.api) that doesn't carry the tailed file's path, it won't match Kube_Tag_Prefix and the Kubernetes filter will silently skip every record from it: no JSON merging, no kubernetes.* metadata, with no error from the collector (it still returns 200, since by then it's just an ordinary JSON object). Check FluentBit's own logs for skip filter warnings if metadata isn't showing up as expected.

Use_Kubelet needs hostNetwork: Use_Kubelet On queries the node's kubelet API directly, which only works if the FluentBit pod runs with hostNetwork: true. Without it, you'll see kubelet upstream connection error in FluentBit's logs. If your DaemonSet doesn't set hostNetwork: true (the example below doesn't), omit Use_Kubelet/Kubelet_Port entirely — the Kubernetes filter then queries the API server instead, using the pod's own ServiceAccount token, which needs no special networking.

DaemonSet Deployment Example

If you don't have FluentBit running yet, see Deploying FluentBit From Scratch near the top of this page for a complete, tested namespace/RBAC/ConfigMap/ DaemonSet example.

Verification and Testing

Check FluentBit is Running

# List FluentBit pods (one per node)
kubectl get pods -n logging -l app=fluent-bit

# Check pod status
kubectl describe pod -n logging <fluent-bit-pod-name>

# View logs
kubectl logs -n logging <fluent-bit-pod-name> --tail=100

Verify Log Collection

# Check FluentBit is reading container logs
kubectl logs -n logging -l app=fluent-bit --tail=100 | grep "inotify_fs_add"

# Should see lines like:
# [ info] [input:tail:tail.0] inotify_fs_add(): inode=12345 name=/var/log/containers/api-server-xyz.log

Verify Forwarding to SenseOn

# Check for successful HTTP POSTs
kubectl logs -n logging -l app=fluent-bit --tail=200 | grep "HTTP status"

# Should see:
# [ info] [output:http:http.0] <your-code>-logs.senseon.io:443, HTTP status=200

Generate Test Logs

Deploy a test application that generates JSON logs:

# Create test pod
kubectl run log-generator --image=busybox --restart=Never -- sh -c \
  'while true; do
    echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test event\",\"request_id\":\"$(date +%s)\"}";
    sleep 10;
  done'

# Check test pod logs
kubectl logs log-generator --tail=10

# Verify FluentBit collected them
kubectl logs -n logging -l app=fluent-bit --tail=50 | grep "test event"

Troubleshooting

FluentBit Pods Not Starting

Check pod events: 

kubectl describe pod -n logging <fluent-bit-pod-name>

Common issues:

  1. RBAC permissions missing

    • Ensure ServiceAccount, ClusterRole, ClusterRoleBinding exist
    • Verify ServiceAccount is referenced in DaemonSet

  2. ConfigMap not mounted

    • Check ConfigMap exists: kubectl get configmap -n logging
    • Verify volume mount in DaemonSet spec

  3. Resource constraints

    • Check node resources: kubectl describe node
    • Adjust memory/CPU limits in DaemonSet

No Logs Being Collected

Check INPUT configuration: 

kubectl logs -n logging -l app=fluent-bit --tail=100 | grep -i "input"

Verify log paths exist: 

# Exec into a FluentBit pod
kubectl exec -it -n logging <fluent-bit-pod-name> -- sh

# Check log directory
ls -la /var/log/containers/

# Should see *.log files

Common issues:

  1. Wrong log path in INPUT
    • Path must match actual container log locations
    • Usually /var/log/containers/*.log
  2. Permissions issues
    • FluentBit needs read access to /var/log
    • Check hostPath volume mounts

Logs Not Reaching SenseOn

Check OUTPUT configuration: 

kubectl logs -n logging -l app=fluent-bit --tail=200 | grep -i "output"

Test collector endpoint: 

# From within cluster
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -v -X POST https://<your-code>-logs.senseon.io \
  -H "Content-Type: application/json" \
  -d '{"test":"connection"}'

# Should see HTTP 200

Common issues:

  1. Network policy blocking egress
    • Check network policies: kubectl get networkpolicy --all-namespaces
    • Ensure HTTPS (443) egress is allowed
  2. Wrong collector URL
    • Verify URL with SenseOn support
    • Your endpoint is <your-code>-logs.senseon.io — check for typos
  3. TLS certificate issues
    • Check FluentBit logs for TLS errors
    • Ensure tls.verify On is set

Logs Are Nested/Escaped JSON

This means parsing isn't working correctly.

Check for these in your configuration:

  1. INPUT must use multiline.parser cri: 

    [INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser cri    # ← This is required

  2. Kubernetes filter must have Merge_Log On: 

    [FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On    # ← This parses and merges JSON
    Keep_Log Off     # ← This removes the raw log field

  3. Tag must match Kube_Tag_Prefix — if your INPUT uses a fixed tag (e.g. Tag kube.api) rather than Tag kube.*, and it doesn't carry the tailed file's path, the Kubernetes filter silently skips every record (check FluentBit's own logs for skip filter warnings). Merge_Log On never runs, so logs stay nested with no error at all from the collector — it still returns 200.

  4. If logs are still nested, add explicit JSON parser: 

    [FILTER]
    Name parser
    Match kube.*
    Key_Name log
    Parser json
    Reserve_Data True

High Memory Usage

Check FluentBit resource usage: 

kubectl top pods -n logging

Reduce memory usage:

  1. Lower buffer limits in INPUT: 

    [INPUT]
    Mem_Buf_Limit 5MB    # Default is 32MB

  2. Increase flush frequency: 

    [SERVICE]
    Flush 5    # Flush every 5 seconds (default is 1)

  3. Filter out verbose logs: 

    [FILTER]
    Name grep
    Match kube.*
    Exclude log debug|trace

Performance Tuning

For high-volume environments:

[SERVICE]
    Flush 1              # Flush every second
    Grace 5              # Wait 5 seconds on shutdown
    Log_Level warning    # Reduce FluentBit's own logging

[INPUT]
    Mem_Buf_Limit 32MB   # Increase buffer size
    Skip_Long_Lines On   # Skip lines > 32KB

[OUTPUT]
    Workers 2            # Parallel HTTP connections
    Retry_Limit 3        # Max retry attempts
    net.keepalive On     # Reuse connections

Cloud-Specific Notes

Amazon EKS

Cluster access: 

# Configure kubectl
aws eks update-kubeconfig --region <region> --name <cluster-name>

# Verify access
kubectl get nodes

EKS-specific considerations:

  • Ensure IAM roles allow pod-to-pod communication
  • Check VPC security groups allow outbound HTTPS
  • Verify nodes have internet access (direct or via NAT gateway)

Google GKE

Cluster access: 

# Configure kubectl
gcloud container clusters get-credentials <cluster-name> --region=<region>

# Verify access
kubectl get nodes

GKE-specific considerations:

  • Workload Identity may be required for private clusters
  • Check firewall rules allow egress to *.senseon.io
  • Binary Authorization may block FluentBit image (add exception)

Azure AKS

Cluster access: 

# Configure kubectl
az aks get-credentials --resource-group <rg> --name <cluster-name>

# Verify access
kubectl get nodes

AKS-specific considerations:

  • Check Network Security Groups allow outbound HTTPS
  • Azure Policy may require specific pod security standards
  • Verify DNS resolution for *.senseon.io

Need help?

Contact SenseOn support at support@senseon.io

When contacting support, include:

  • Your organisation name
  • Collector endpoint URL
  • Example log entries (sanitised)
  • Relevant error messages (if applicable)
  • Log source type
  • Configuration files (if applicable)