Kubernetes Log Forwarding Configuration
This guide tells you how to forward logs from Kubernetes clusters to SenseOn. The examples focus on Amazon EKS. This guide is for customers who want to ingest logs from:
- Amazon EKS (Elastic Kubernetes Service)
- Google GKE (Google Kubernetes Engine)
- Azure AKS (Azure Kubernetes Service)
- Self-hosted Kubernetes
Deployment Requirements
For Kubernetes logs, SenseOn uses a FluentBit → external collector architecture. You do the configuration yourself, so you keep control of log collection. There are two paths. Select the path for your start point:
- FluentBit or Fluentd already runs in your cluster: Add a new OUTPUT block to your existing configuration. Refer to Configuring Log Forwarding.
- You do not have FluentBit yet: Deploy a new FluentBit next to your applications. Refer to the deployment of a new FluentBit.
Deploying FluentBit From Scratch
Use this section if FluentBit, Fluentd or any other log collector does not run in your cluster yet. This section deploys FluentBit as its own DaemonSet, next to your existing applications.
Architecture:
Application Pods → FluentBit (new DaemonSet) → SenseOn Collector
Prerequisites
- Cluster admin access to create namespaces, RBAC, ConfigMaps, and DaemonSets
- Outbound HTTPS (443) access to
*.senseon.io - Outbound access to Docker Hub, to pull the
fluent/fluent-bitimage - Applications that log JSON to stdout
Step 1: Deploy the namespace, RBAC, ConfigMap, and DaemonSet
Save the YAML that follows as fluentbit-deployment.yaml. Replace <your-cluster-name> and <your-code> with your own values:
apiVersion: v1
kind: Namespace
metadata:
name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit
rules:
- apiGroups: [""]
resources: ["pods", "namespaces"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 5
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Kube_Tag_Prefix kube.var.log.containers.
K8S-Logging.Parser Off
K8S-Logging.Exclude Off
[FILTER]
Name modify
Match kube.*
Add cluster_name <your-cluster-name>
Add environment production
Add source kubernetes
[OUTPUT]
Name http
Match kube.*
Host <your-code>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
net.keepalive On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
labels:
app: fluent-bit
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: fluent/fluent-bit:2.2.3
resources:
requests:
memory: "128Mi"
cpu: "150m"
limits:
memory: "128Mi"
cpu: "150m"
volumeMounts:
- name: varlog
mountPath: /var/log
- name: config
mountPath: /fluent-bit/etc/fluent-bit.conf
subPath: fluent-bit.conf
volumes:
- name: varlog
hostPath:
path: /var/log
- name: config
configMap:
name: fluent-bit-config
Then deploy it:
kubectl apply -f fluentbit-deployment.yaml
kubectl get pods -n logging
kubectl -n logging rollout status daemonset/fluent-bit
No /var/lib/docker/containers mount: This DaemonSet mounts only
/var/log. On containerd-based clusters (the default for EKS, GKE and
AKS today), /var/lib/docker/containers does not exist. Also, the FluentBit
tail input never reads that directory directly. It follows
/var/log/containers/*.log. The kubelet keeps symlinks there to the
location where the container runtime actually stores the logs.
To learn why the ConfigMap has this shape, refer to
Understanding Kubernetes Log Parsing.
Pay special attention to the note that Tag must match Kube_Tag_Prefix.
Step 2: Verify FluentBit is running and forwarding
kubectl -n logging logs -l app=fluent-bit --tail=100 | grep "HTTP status"
# Should see: HTTP status=200
Step 3: Test end-to-end with a sample application
Deploy a small pod that writes JSON logs on a timer. Then make sure that FluentBit collects them:
kubectl run log-generator --image=busybox --restart=Never -- sh -c \
'while true; do
echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"event\":\"heartbeat\",\"level\":\"info\",\"service\":\"log-generator\"}";
sleep 5;
done'
kubectl logs log-generator --tail=10
# FluentBit's own logs report HTTP status, not record content - use the
# same check as Step 2 to confirm this test app's events round-tripped
kubectl -n logging logs -l app=fluent-bit --tail=50 | grep "HTTP status"
When you see clean HTTP status=200 responses for these test events, delete the test pod:
kubectl delete pod log-generator
New applications usually show as Unclassified in SenseOn until the schema of your log source is onboarded. First make sure that FluentBit forwards the logs correctly. Then send a sanitised sample. SenseOn does the work that remains. Send the sample to support@senseon.io.
For help with parser problems, TLS errors and high memory use, refer to Troubleshooting. That section covers both deployment paths.
Configuring Log Forwarding
Overview
If you already deployed FluentBit (or Fluentd) in your cluster, add only a new OUTPUT block to forward logs to SenseOn.
Architecture:
Application Pods → FluentBit (already present) → Add OUTPUT → SenseOn Collector
Prerequisites
- Existing FluentBit or Fluentd deployment
- Cluster admin access to modify ConfigMaps
- Outbound HTTPS (443) access to
*.senseon.io(your collector endpoint is<your-code>-logs.senseon.io) - Applications that log JSON to stdout (or configured FluentBit parsers)
Step 1: Identify Your FluentBit Configuration
Find your existing FluentBit ConfigMap:
# Find FluentBit namespace
kubectl get pods --all-namespaces | grep fluent
# Get the ConfigMap
kubectl get configmap -n <fluent-namespace>
kubectl get configmap fluent-bit-config -n <fluent-namespace> -o yaml
Step 2: Add SenseOn OUTPUT Block
Edit your FluentBit ConfigMap to add a new OUTPUT section:
kubectl edit configmap fluent-bit-config -n <fluent-namespace>
Add a new OUTPUT block to the outputs: section:
outputs: |
# Your existing outputs (keep these)
[OUTPUT]
Name forward
Match *
Host your-internal-aggregator.example.com
Port 24224
# NEW: Add this OUTPUT for SenseOn
[OUTPUT]
Name http
Match *
Host <your-code>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
net.keepalive On
settings explained:
Match *- Forwards all logs. To forward logs from specific apps only, you can filter withMatch kube.*app-name*.Format json- Sends each flush as a single JSON array. The SenseOn collector accepts one JSON value per request. It does not accept newline-delimitedjson_lines. With newline-delimited JSON, a request stops with an HTTP 400 error intermittently, when a flush has more than one record.json_date_key false- Let SenseOn add timestamps server-sidetls On- Encrypt traffic (mandatory)
Step 3: Restart FluentBit Pods
# Restart FluentBit to apply configuration
kubectl rollout restart daemonset/fluent-bit -n <fluent-namespace>
# Verify pods are running
kubectl get pods -n <fluent-namespace>
# Check pod logs for errors
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=50
Step 4: Verify Logs Are Being Sent
# Check FluentBit logs for HTTP status
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=100 | grep "HTTP status"
# Should see: HTTP status=200 (logs accepted successfully)
Example successful output:
[2025/10/21 10:00:05] [ info] [output:http:http.1] <your-code>-logs.senseon.io:443, HTTP status=200
Step 5: Test with Sample Application
To do an end-to-end check, deploy a test pod that generates JSON logs:
kubectl run test-logger --image=busybox --restart=Never -- sh -c \
'while true; do echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test log\",\"source\":\"test-pod\"}"; sleep 5; done'
# Check test pod logs
kubectl logs test-logger
# Verify FluentBit picked them up
kubectl logs -n <fluent-namespace> -l app=fluent-bit --tail=20 | grep test-pod
Understanding Kubernetes Log Parsing
The Problem: Nested JSON in Kubernetes
Kubernetes wraps container logs in a Docker/containerd format. As a result, the application JSON is nested and escaped.
What your application outputs:
{"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}
What Kubernetes writes to disk:
2025-10-21T10:00:00.123456789Z stdout F {"timestamp": "2025-10-21T10:00:00Z", "level": "info", "message": "user login", "user_id": "123"}
If the logs are not parsed correctly, SenseOn receives:
{
"log": "{\"timestamp\": \"2025-10-21T10:00:00Z\", \"level\": \"info\", \"message\": \"user login\", \"user_id\": \"123\"}",
"stream": "stdout",
"time": "2025-10-21T10:00:00.123456789Z"
}
Your application JSON is escaped in the log field. ❌
The Solution: Multi-Stage Parsing
FluentBit parses Kubernetes logs in more than one stage to unwrap them:
Stage 1: Parse Container Runtime Format
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser cri
Tag kube.*
The cri parser processes the Docker/containerd format, and it extracts the actual log message.
Stage 2: Parse Application JSON
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
The kubernetes filter with Merge_Log On parses JSON from the extracted message and merges fields.
Stage 3: Clean Up and Enrich
[FILTER]
Name modify
Match kube.*
Add source kubernetes
Add cluster_name production-eks
Add metadata to show the source.
Final result sent to SenseOn:
{
"timestamp": "2025-10-21T10:00:00Z",
"level": "info",
"message": "user login",
"user_id": "123",
"source": "kubernetes",
"cluster_name": "production-eks",
"stream": "stdout",
"kubernetes": {
"pod_name": "api-server-abc123",
"namespace_name": "production",
"container_name": "api-server",
"labels": { "app": "api-server" }
}
}
The Kubernetes filter puts its metadata in a kubernetes object, for
example kubernetes.pod_name and kubernetes.namespace_name. It does not
flatten the metadata into top-level kubernetes_* fields. Real forwarded
records show this behaviour.
Example Configuration Reference
Complete FluentBit Configuration for Kubernetes
This complete example configuration shows best practices for Kubernetes log collection:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 5
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
# Collect logs from nginx ingress controller
[INPUT]
Name tail
Path /var/log/containers/ingress-nginx-controller*.log
multiline.parser cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
# Collect logs from application pods
[INPUT]
Name tail
Path /var/log/containers/api-server*.log
multiline.parser cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
# Enrich with Kubernetes metadata and parse JSON logs
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Kube_Tag_Prefix kube.var.log.containers.
K8S-Logging.Parser Off
K8S-Logging.Exclude Off
# Add cluster identification
[FILTER]
Name modify
Match kube.*
Add cluster_name production-eks-01
Add environment production
Add source kubernetes
# Forward to SenseOn collector
[OUTPUT]
Name http
Match kube.*
Host <your-code>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
net.keepalive On
Tag must match Kube_Tag_Prefix: Both [INPUT] blocks in this example use
Tag kube.*. Thus FluentBit puts the path of each tailed file into its tag.
This tag matches Kube_Tag_Prefix kube.var.log.containers. in the filter. You
can give an INPUT a fixed tag (for example, Tag kube.api) that does not contain
the path of the tailed file. In this case, the tag does not match Kube_Tag_Prefix.
The Kubernetes filter then silently skips each record from that INPUT.
The result is no JSON merge and no kubernetes.* metadata. The collector
gives no error. It still returns 200, because at that point the record is an
ordinary JSON object. If the metadata does not show as you expect, look for
skip filter warnings in the logs of FluentBit.
Use_Kubelet needs hostNetwork: Use_Kubelet On queries the kubelet API
of the node directly. This works only if the FluentBit pod runs with
hostNetwork: true. If not, you see kubelet upstream connection
error in the logs of FluentBit. If your DaemonSet does not set
hostNetwork: true (the example that follows does not), do not use
Use_Kubelet/Kubelet_Port. The Kubernetes filter then queries the API
server, with the ServiceAccount token of the pod. A special network
configuration is not necessary for this method.
DaemonSet Deployment Example
If FluentBit does not run in your cluster yet, refer to the deployment of a new FluentBit near the start of this page. That section has a complete, tested example of the namespace, RBAC, ConfigMap and DaemonSet.
Verification and Testing
Check FluentBit is Running
# List FluentBit pods (one per node)
kubectl get pods -n logging -l app=fluent-bit
# Check pod status
kubectl describe pod -n logging <fluent-bit-pod-name>
# View logs
kubectl logs -n logging <fluent-bit-pod-name> --tail=100
Verify Log Collection
# Check FluentBit is reading container logs
kubectl logs -n logging -l app=fluent-bit --tail=100 | grep "inotify_fs_add"
# Should see lines like:
# [ info] [input:tail:tail.0] inotify_fs_add(): inode=12345 name=/var/log/containers/api-server-xyz.log
Verify Forwarding to SenseOn
# Check for successful HTTP POSTs
kubectl logs -n logging -l app=fluent-bit --tail=200 | grep "HTTP status"
# Should see:
# [ info] [output:http:http.0] <your-code>-logs.senseon.io:443, HTTP status=200
Generate Test Logs
Deploy a test application that generates JSON logs:
# Create test pod
kubectl run log-generator --image=busybox --restart=Never -- sh -c \
'while true; do
echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"info\",\"message\":\"test event\",\"request_id\":\"$(date +%s)\"}";
sleep 10;
done'
# Check test pod logs
kubectl logs log-generator --tail=10
# Verify FluentBit collected them
kubectl logs -n logging -l app=fluent-bit --tail=50 | grep "test event"
Troubleshooting
FluentBit Pods Not Starting
Examine the pod events:
kubectl describe pod -n logging <fluent-bit-pod-name>
Common issues:
-
RBAC permissions missing
- Make sure that the ServiceAccount, ClusterRole and ClusterRoleBinding exist
- Make sure that the DaemonSet refers to the ServiceAccount
-
ConfigMap not mounted
- Make sure that the ConfigMap exists:
kubectl get configmap -n logging - Examine the volume mount in the DaemonSet spec
- Make sure that the ConfigMap exists:
-
Resource constraints
- Examine the node resources:
kubectl describe node - Adjust the memory/CPU limits in the DaemonSet
- Examine the node resources:
No Logs Being Collected
Examine the INPUT configuration:
kubectl logs -n logging -l app=fluent-bit --tail=100 | grep -i "input"
Make sure that the log paths exist:
# Exec into a FluentBit pod
kubectl exec -it -n logging <fluent-bit-pod-name> -- sh
# Check log directory
ls -la /var/log/containers/
# Should see *.log files
Common issues:
- Wrong log path in INPUT
- The path must match the actual locations of the container logs
- Usually
/var/log/containers/*.log
- Permissions issues
- FluentBit must have read access to /var/log
- Examine the hostPath volume mounts
Logs Not Reaching SenseOn
Examine the OUTPUT configuration:
kubectl logs -n logging -l app=fluent-bit --tail=200 | grep -i "output"
Test the collector endpoint:
# From within cluster
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
curl -v -X POST https://<your-code>-logs.senseon.io \
-H "Content-Type: application/json" \
-d '{"test":"connection"}'
# Should see HTTP 200
Common issues:
- Network policy blocks egress
- Examine the network policies:
kubectl get networkpolicy --all-namespaces - Make sure that the policies allow HTTPS (443) egress
- Examine the network policies:
- Wrong collector URL
- Ask SenseOn support to do a check of the URL
- Your endpoint is
<your-code>-logs.senseon.io. Look for typos
- TLS certificate issues
- Look for TLS errors in the FluentBit logs
- Make sure that
tls.verify Onis set
Logs Are Nested/Escaped JSON
This means that the logs are not parsed correctly.
Look for these items in your configuration:
-
INPUT must use
multiline.parser cri:[INPUT] Name tail Path /var/log/containers/*.log multiline.parser cri # ← This is required -
Kubernetes filter must have
Merge_Log On:[FILTER] Name kubernetes Match kube.* Merge_Log On # ← This parses and merges JSON Keep_Log Off # ← This removes the raw log field -
Tagmust matchKube_Tag_Prefix: Your INPUT can use a fixed tag (for example,Tag kube.api) and notTag kube.*. If that tag does not contain the path of the tailed file, the Kubernetes filter silently skips each record. To find this problem, look forskip filterwarnings in the logs of FluentBit.Merge_Log Onnever runs. Thus the logs stay nested, and the collector gives no error. It still returns200. -
If logs are still nested, add an explicit JSON parser:
[FILTER] Name parser Match kube.* Key_Name log Parser json Reserve_Data True
High Memory Usage
Examine the FluentBit resource usage:
kubectl top pods -n logging
Reduce memory usage:
-
Lower buffer limits in INPUT:
[INPUT] Mem_Buf_Limit 5MB # Default is 32MB -
Increase flush frequency:
[SERVICE] Flush 5 # Flush every 5 seconds (default is 1) -
Filter out verbose logs:
[FILTER] Name grep Match kube.* Exclude log debug|trace
Performance Tuning
For high-volume environments:
[SERVICE]
Flush 1 # Flush every second
Grace 5 # Wait 5 seconds on shutdown
Log_Level warning # Reduce FluentBit's own logging
[INPUT]
Mem_Buf_Limit 32MB # Increase buffer size
Skip_Long_Lines On # Skip lines > 32KB
[OUTPUT]
Workers 2 # Parallel HTTP connections
Retry_Limit 3 # Max retry attempts
net.keepalive On # Reuse connections
Cloud-Specific Notes
Amazon EKS
Cluster access:
# Configure kubectl
aws eks update-kubeconfig --region <region> --name <cluster-name>
# Verify access
kubectl get nodes
EKS-specific considerations:
- Make sure that IAM roles allow pod-to-pod communication
- Make sure that VPC security groups allow outbound HTTPS
- Make sure that the nodes have internet access (direct or through a NAT gateway)
Google GKE
Cluster access:
# Configure kubectl
gcloud container clusters get-credentials <cluster-name> --region=<region>
# Verify access
kubectl get nodes
GKE-specific considerations:
- Workload Identity can be necessary for private clusters
- Make sure that firewall rules allow egress to *.senseon.io
- Binary Authorization can block the FluentBit image. If it does, add an exception
Azure AKS
Cluster access:
# Configure kubectl
az aks get-credentials --resource-group <rg> --name <cluster-name>
# Verify access
kubectl get nodes
AKS-specific considerations:
- Make sure that Network Security Groups allow outbound HTTPS
- Azure Policy can make specific pod security standards necessary
- Make sure that DNS resolution for *.senseon.io works
Need help?
Contact SenseOn support at support@senseon.io
When you contact support, include this information:
- Your organisation name
- Collector endpoint URL
- Example log entries (sanitised)
- Relevant error messages (if applicable)
- Log source type
- Configuration files (if applicable)