Detailed Configuration Examples
Before You Start
You must have your ingestion hostname, <code-name>-logs.senseon.io, from SenseOn. You must also have outbound access to it, with no TLS inspection on the path. This guide covers syslog over TLS on TCP 6514 (on request) and JSON over HTTPS on TCP 443. For OpenTelemetry, refer to Claude Code Telemetry. Always connect by hostname, not by IP address. The Configuration Overview page gives the full list of rules.
The examples on this page are patterns that you can adapt. They are not fixed recipes. Change the log paths, parsers and filters for your source. Test the endpoint first. Then make sure that the log of your forwarder shows successful sends (HTTP 200 for JSON over HTTPS).
Table of Contents
Worked Examples
- GKE Nginx - Nginx on Google Kubernetes Engine
- GCE VM Nginx - Nginx on Google Compute Engine VM
- Cloud Run Nginx - Nginx on Cloud Run (serverless)
- Cloud Run Application - Custom application on Cloud Run
Generic Patterns
Testing Your Endpoint
Before you configure a log forwarder, test your ingestion endpoint.
JSON over HTTPS:
# Replace <code-name> with your code name
curl -X POST https://<code-name>-logs.senseon.io/ \
-H "Content-Type: application/json" \
-d '{"timestamp": "2025-10-17T10:00:00Z", "source": "test", "message": "Connection test"}'
Expected response: HTTP 200.
Syslog over TLS. Run this test after SenseOn tells you that syslog is on for your organisation:
openssl s_client -connect <code-name>-logs.senseon.io:6514 \
-servername <code-name>-logs.senseon.io \
-verify_hostname <code-name>-logs.senseon.io -verify_return_error -brief </dev/null
Expected result: Verification: OK and Verified peername: *.senseon.io. This test must use OpenSSL 1.1 or later. The LibreSSL build that comes with macOS does not have -verify_hostname. If one of the tests does not work, refer to Troubleshooting.
GCE VM Nginx with FluentBit
Configuration Overview
This example shows nginx on a Google Compute Engine VM. FluentBit forwards the logs. This same approach works for: - Any Linux VM (AWS EC2, Azure VMs, on-premises) - Physical servers - Any environment where you can install FluentBit as a service
What you build:
HTTP Request → Nginx (JSON logs) → FluentBit (local service) → SenseOn
Prerequisites
- Linux VM with nginx installed
- Root/sudo access to install packages
- Outbound HTTPS (443) access
Step 1: Configure Nginx JSON Logging
Create or edit /etc/nginx/conf.d/senseon-logging.conf:
log_format senseon_json escape=json '{'
'"timestamp":"$time_iso8601",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"status":"$status",'
'"body_bytes_sent":$body_bytes_sent,'
'"request_time":$request_time,'
'"http_referer":"$http_referer",'
'"http_user_agent":"$http_user_agent",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"upstream_addr":"$upstream_addr",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_status":"$upstream_status"'
'}';
access_log /var/log/nginx/access.log senseon_json;
What each field captures:
- timestamp - When the request occurred (ISO8601 format)
- remote_addr - Client IP address
- request - Full HTTP request line (method, URI, protocol)
- status - HTTP response status code
- body_bytes_sent - Size of response body
- request_time - Total request processing time (seconds)
- upstream_* fields - For reverse proxy scenarios (backend server info)
Note: When nginx serves web content directly, the upstream fields are empty. In reverse proxy scenarios, the log contains them.
Step 2: Restart Nginx
# Test nginx config
sudo nginx -t
# Restart to apply changes
sudo systemctl restart nginx
# Verify logs are JSON
sudo tail -f /var/log/nginx/access.log
# Generate a test request
curl http://localhost/
You see JSON logs such as:
{"timestamp":"2025-10-17T10:00:00+00:00","remote_addr":"127.0.0.1","request":"GET / HTTP/1.1","status":"200"...}
Step 3: Install FluentBit
# For Debian/Ubuntu
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
# Or using package manager (secure method for Ubuntu 20.04+)
wget -qO /usr/share/keyrings/fluentbit-archive-keyring.gpg https://packages.fluentbit.io/fluentbit.key
echo "deb [signed-by=/usr/share/keyrings/fluentbit-archive-keyring.gpg] https://packages.fluentbit.io/ubuntu/focal focal main" | sudo tee /etc/apt/sources.list.d/fluent-bit.list
sudo apt-get update
sudo apt-get install fluent-bit
# Verify installation
fluent-bit --version
# Should show: Fluent Bit v2.2.x
Step 4: Configure FluentBit
Create /etc/fluent-bit/fluent-bit.conf:
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/nginx/access.log
Parser json
Tag nginx.access
Refresh_Interval 5
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name modify
Match nginx.access
Add source nginx-gce-vm
Add environment production
Add vm_name nginx-log-vm
[OUTPUT]
Name http
Match nginx.access
Host <code-name>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
net.keepalive On
Notable settings functionalities:
- Mem_Buf_Limit 5MB - Limits memory buffer per input (prevents OOM)
- Skip_Long_Lines On - Skips lines > 32KB (prevents issues with huge log entries)
- vm_name nginx-log-vm - Adds VM identifier (useful when you have multiple VMs)
Step 5: Start FluentBit
# Start the service
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
# Check status
sudo systemctl status fluent-bit
# Should show: active (running)
Step 6: Verify Log Forwarding
# Check FluentBit is reading nginx logs
sudo journalctl -u fluent-bit -f | grep "HTTP status"
# Should see: HTTP status=200
# Generate test traffic
curl http://localhost/
# Check FluentBit logs show forwarding
sudo journalctl -u fluent-bit --since "1 minute ago" | grep "HTTP status"
Example Log Output Sent to SenseOn
{
"timestamp": "2025-10-14T15:30:45+00:00",
"remote_addr": "203.0.113.42",
"remote_user": "",
"request": "GET /api/users HTTP/1.1",
"status": "200",
"body_bytes_sent": 1247,
"http_referer": "https://example.com/dashboard",
"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"request_time": 0.234,
"upstream_response_time": "0.156"
}
Syslog (rsyslog)
Configuration Overview
This example forwards the system logs of a Linux host to SenseOn as syslog over TLS on port 6514. It uses the GnuTLS driver of rsyslog and a disk queue. If you add network inputs to the same configuration, the host becomes a relay for other devices. Refer to Ingesting Syslog from Network Appliances.
Before you start, ask SenseOn to turn on syslog for your organisation.
Step 1: Install rsyslog with TLS support
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install rsyslog rsyslog-gnutls
# RHEL, Rocky, Alma
sudo dnf install rsyslog rsyslog-gnutls
Step 2: Configure rsyslog
Create /etc/rsyslog.d/10-senseon.conf. Set the CA file for your distribution:
- Debian and Ubuntu:
/etc/ssl/certs/ca-certificates.crt - RHEL family:
/etc/pki/tls/certs/ca-bundle.crt
# SenseOn forwarder. Logs are sent over TLS with a disk queue
# (up to 2 GB) that holds messages while SenseOn is unreachable.
global(defaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt")
ruleset(name="to_senseon") {
action(
type="omfwd"
target="<code-name>-logs.senseon.io"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="*.senseon.io"
TCP_Framing="octet-counted"
template="RSYSLOG_SyslogProtocol23Format"
queue.type="linkedList"
queue.filename="senseon_fwd"
queue.spoolDirectory="/var/spool/rsyslog"
queue.maxdiskspace="2g"
queue.saveonshutdown="on"
action.resumeRetryCount="-1"
)
}
# Choose what to send. Adjust the severity as needed
# (debug, info, notice, warning, err, crit).
*.info call to_senseon
Important points:
- Use
StreamDriver="gtls"exactly as shown. The OpenSSL driver (ossl) in many rsyslog releases sends no SNI, and the endpoint resets each connection. StreamDriverPermittedPeersmust be the wildcard*.senseon.io. The name check of rsyslog rejects the full hostname.TCP_Framing="octet-counted"is the framing SenseOn recommends.RSYSLOG_SyslogProtocol23Formatsends RFC 5424. Do not use a custom template that removes the<priority>header, because the endpoint then rejects the messages. If this host relays logs from devices with unusual timestamps, useRSYSLOG_ForwardFormatas an alternative.
Step 3: Restart rsyslog
# Test configuration
sudo rsyslogd -N1
# Restart service
sudo systemctl restart rsyslog
# Send a test message and check for errors
logger -p user.info "senseon connection test from $(hostname)"
sudo journalctl -u rsyslog --since "5 minutes ago"
A healthy forwarder shows no omfwd or gtls errors in the journal, and the queue files under /var/spool/rsyslog stay small.
| Symptom in the journal | Cause | Fix |
|---|---|---|
peer name not authorized |
PermittedPeers set to the hostname | Set it to *.senseon.io |
warning: certificate file is not set or key file is not set |
rsyslog notes that it has no client certificate | No action is necessary. SenseOn does not use client certificates. |
SSL_ERROR_SYSCALL or ECONNRESET on every attempt |
No server name sent, the CA file is missing, or syslog is not yet turned on for your organisation | Use gtls. Make sure that the defaultNetstreamDriverCAFile path exists. Ask SenseOn if syslog is on for your organisation |
cannot resolve |
The hostname is wrong | It is <code-name>-logs.senseon.io, with a hyphen |
| Queue files grow | SenseOn is unreachable | Examine outbound 6514. Make sure that no proxy inspects TLS on the path |
Example Log Output
Each message arrives as RFC 5424 syslog, for example:
<14>1 2025-10-14T15:30:45.123456+00:00 web-server-01 backup-job 4242 - - Nightly backup finished in 312 seconds
FluentBit Log Forwarder
Configuration Overview
FluentBit is a lightweight log forwarder. It can collect logs from many sources and forward them to SenseOn.
Installation
# Download and install
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
# Or using package manager (Ubuntu/Debian)
wget -qO /usr/share/keyrings/fluentbit-archive-keyring.gpg https://packages.fluentbit.io/fluentbit.key
echo "deb [signed-by=/usr/share/keyrings/fluentbit-archive-keyring.gpg] https://packages.fluentbit.io/ubuntu/focal focal main" | sudo tee /etc/apt/sources.list.d/fluent-bit.list
sudo apt-get update
sudo apt-get install fluent-bit
Configuration File
Create /etc/fluent-bit/fluent-bit.conf:
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
# Example: Tail multiple log files
[INPUT]
Name tail
Path /var/log/nginx/access.log
Parser json
Tag nginx
Refresh_Interval 5
[INPUT]
Name tail
Path /var/log/app/application.log
Parser json
Tag application
Refresh_Interval 5
# Example: Collect systemd journal logs
[INPUT]
Name systemd
Tag systemd
Read_From_Tail On
Strip_Underscores On
# Add hostname to all logs
[FILTER]
Name modify
Match *
Add hostname ${HOSTNAME}
# Forward all logs to SenseOn
[OUTPUT]
Name http
Match *
Host <code-name>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
Start FluentBit
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
# Check status
sudo systemctl status fluent-bit
# View logs
sudo journalctl -u fluent-bit -f
GKE Nginx with FluentBit
Configuration Overview
This example shows nginx on Google Kubernetes Engine (GKE). A FluentBit DaemonSet forwards the logs. This same approach works for: - Any Kubernetes cluster (EKS, AKS, self-hosted) - Any K8s application that logs JSON to stdout (not only nginx)
What you build:
HTTP Request → Nginx Pod (JSON logs to stdout) →
Docker wrapper → FluentBit DaemonSet (parse & forward) → SenseOn
The challenge: Kubernetes wraps logs in Docker/containerd format. As a result, the JSON logs of your application are nested and escaped in a log field. This section shows you how to parse Kubernetes logs correctly, so that you send clean JSON to SenseOn.
The Problem: Nested JSON
If you forward Kubernetes logs without the correct parser configuration, you get this result:
{
"log": "2025-10-17T10:00:00.123456789Z stdout F {\"timestamp\":\"2025-10-17T10:00:00Z\",\"status\":\"200\"}",
"stream": "stdout",
"time": "2025-10-17T10:00:00.123456789Z"
}
Your application JSON is escaped and nested in the log field. This is not what you want.
The Solution: Two-Stage Parsing
Parse the log in two stages: 1. Strip the Docker wrapper (timestamp, stream, tag) 2. Parse your application JSON from the extracted message
The DaemonSet configuration that follows does this.
Prerequisites
- Kubernetes cluster (any type: GKE, EKS, AKS, self-hosted)
kubectlaccess with admin permissions- Application containers that log JSON to stdout
Step 1: Create Namespace
kubectl create namespace nginx-logs
Step 2: Create FluentBit DaemonSet
Save this as fluentbit-daemonset.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: nginx-logs
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit
rules:
- apiGroups: [""]
resources: ["pods", "namespaces"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: nginx-logs
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: nginx-logs
data:
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 5
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*nginx-logs_nginx-*.log
Tag nginx
Refresh_Interval 5
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Parser docker
# Stage 1: Parse Docker wrapper to extract the JSON message
[FILTER]
Name parser
Match nginx
Key_Name log
Parser docker_strip
Reserve_Data On
# Stage 2: Parse the application JSON from the message field
[FILTER]
Name parser
Match nginx
Key_Name message
Parser nginx_json
Reserve_Data On
# Remove Docker wrapper fields, keep application data
[FILTER]
Name modify
Match nginx
Remove log
Remove stream
Remove time
Remove message
Remove logtag
Add source nginx-gke
Add environment production
[OUTPUT]
Name http
Match nginx
Host <code-name>-logs.senseon.io
Port 443
URI /
Format json
json_date_key false
tls On
tls.verify On
Header Content-Type application/json
Retry_Limit 3
net.keepalive On
parsers.conf: |
# Parser for Docker JSON format
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep Off
# Parser to strip Docker wrapper and extract message
[PARSER]
Name docker_strip
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep Off
# Parser for nginx JSON logs
[PARSER]
Name nginx_json
Format json
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: nginx-logs
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: fluent/fluent-bit:2.2.3
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
Step 3: Deploy to Kubernetes
# Deploy FluentBit (namespace is already created in Step 1)
kubectl apply -f fluentbit-daemonset.yaml
# Verify deployment
kubectl get pods -n nginx-logs
# Should show one fluent-bit pod per node, all Running
# NAME READY STATUS RESTARTS AGE
# fluent-bit-xxxxx 1/1 Running 0 30s
# fluent-bit-yyyyy 1/1 Running 0 30s
Step 4: Verify Log Forwarding
# Check FluentBit logs for successful forwarding
kubectl logs -n nginx-logs -l app=fluent-bit --tail=20 | grep "HTTP status"
# Should see: HTTP status=200 (logs accepted by SenseOn)
# Example output:
# [2025/10/17 10:00:05] [ info] [output:http:http.0] <code-name>-logs.senseon.io:443, HTTP status=200
Step 5: Generate Test Logs
If you have nginx pods that run in the nginx-logs namespace:
# Generate a test request to your nginx service
kubectl exec -n nginx-logs deployment/nginx -- curl localhost
# Check FluentBit picked up the log
kubectl logs -n nginx-logs -l app=fluent-bit --tail=50 | grep "HTTP status"
How the Parsing Works
Input (from Kubernetes):
2025-10-17T10:00:00.123456789Z stdout F {"timestamp":"2025-10-17T10:00:00+00:00","remote_addr":"10.154.15.237","request":"GET / HTTP/1.1","status":"200"}
After Stage 1 (docker_strip):
- Extracts: message = {"timestamp":"2025-10-17T10:00:00+00:00","remote_addr":"10.154.15.237","request":"GET / HTTP/1.1","status":"200"}
- Removes: Docker timestamp, stream, tag
After Stage 2 (nginx_json):
- Parses the JSON from the message field
- Moves fields to the top level, for example timestamp, remote_addr, request and status
After modify filter:
- Removes: log, stream, time, message, logtag (Docker wrapper fields)
- Adds: source, environment (to show the log source)
Final output sent to SenseOn:
{
"timestamp": "2025-10-17T10:00:00+00:00",
"remote_addr": "10.154.15.237",
"request": "GET / HTTP/1.1",
"status": "200",
"body_bytes_sent": 45,
"request_time": 0.000,
"http_user_agent": "curl/7.88.1",
"source": "nginx-gke",
"environment": "production"
}
Troubleshooting Kubernetes Logs
Problem: Logs still nested/escaped
Make sure that your FluentBit configuration has these items:
1. Two parser filters (docker_strip, then nginx_json or app_json)
2. Reserve_Data On on both parsers
3. A modify filter that removes the wrapper fields (log, stream, time, message, logtag)
4. Correct parser names in parsers.conf that match the filter references
Problem: No logs forwarded
# Check FluentBit is reading container logs
kubectl logs -n nginx-logs -l app=fluent-bit --tail=100 | grep "inotify_fs_add"
# Should see: inotify_fs_add(): inode=... name=/var/log/containers/nginx-logs_nginx-...
# Check for errors
kubectl logs -n nginx-logs -l app=fluent-bit --tail=100 | grep -i error
Problem: FluentBit pods do not start
# Check pod status
kubectl describe pod -n nginx-logs -l app=fluent-bit
# Common issues:
# - Missing permissions (check RBAC)
# - Config syntax error (check configmap)
# - Resource limits too low
Notes
Best practice configuration details:
- Use Format json in OUTPUT. This setting sends each batch as a JSON array. json_lines sends newline-delimited JSON, and the endpoint rejects it with HTTP 400
- Set json_date_key false - Let SenseOn add timestamps server-side
- Use Reserve_Data On. This setting merges parsed fields and does not replace them
Cloud Run Nginx with Cloud Logging
Configuration Overview
This example shows nginx on Cloud Run (serverless). Cloud Logging forwards the logs. This approach is ideal for: - Serverless nginx (no VM management) - Low-to-medium traffic websites - Dev/test environments - Quick deployments
What you build:
HTTP Request → Nginx (Cloud Run container) → JSON logs to stdout →
Cloud Logging → Pub/Sub → Cloud Function → SenseOn
Why this is different from VM/GKE: - FluentBit is not necessary (Cloud Logging does the collection) - Serverless (auto-scaling, no infrastructure) - Pay-per-request pricing - Simpler deployment (only container + function)
Prerequisites
- GCP project with billing enabled
gcloudCLI installed and authenticated- Docker installed locally (to build the image)
Step 1: Create Nginx Configuration Files
Create nginx.conf (log format definition):
log_format senseon_json escape=json '{'
'"timestamp":"$time_iso8601",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"status":"$status",'
'"body_bytes_sent":$body_bytes_sent,'
'"request_time":$request_time,'
'"http_referer":"$http_referer",'
'"http_user_agent":"$http_user_agent",'
'"http_x_forwarded_for":"$http_x_forwarded_for"'
'}';
Create default.conf (server configuration):
server {
listen 8080;
server_name localhost;
access_log /dev/stdout senseon_json;
error_log /dev/stderr warn;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
Create Dockerfile:
FROM nginx:alpine
# Copy log format definition first (loaded alphabetically before default.conf)
COPY nginx.conf /etc/nginx/conf.d/00-log-format.conf
# Copy server configuration
COPY default.conf /etc/nginx/conf.d/default.conf
# Disable default access log in main nginx.conf
RUN sed -i 's/access_log .*;/access_log off;/' /etc/nginx/nginx.conf
# Create a simple index page
RUN echo '<html><body><h1>Cloud Run Nginx</h1><p>Real nginx logs to SenseOn</p></body></html>' > /usr/share/nginx/html/index.html
EXPOSE 8080
Step 2: Deploy to Cloud Run
# Set variables
export PROJECT_ID="your-project-id"
export REGION="europe-west2"
export SERVICE_NAME="nginx-cloudrun"
# Build and submit image
gcloud builds submit --tag gcr.io/$PROJECT_ID/$SERVICE_NAME
# Deploy to Cloud Run
gcloud run deploy $SERVICE_NAME \
--image gcr.io/$PROJECT_ID/$SERVICE_NAME \
--region=$REGION \
--platform=managed \
--allow-unauthenticated \
--port=8080
# Get the service URL
SERVICE_URL=$(gcloud run services describe $SERVICE_NAME --region=$REGION --format='value(status.url)')
echo "Nginx URL: $SERVICE_URL"
# Test it
curl $SERVICE_URL
Step 3: Set Up Log Forwarding
Create Pub/Sub topic:
gcloud pubsub topics create senseon-nginx-logs
Create log sink (captures nginx JSON logs):
gcloud logging sinks create senseon-nginx-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/senseon-nginx-logs \
--log-filter='resource.type="cloud_run_revision"
AND resource.labels.service_name="'$SERVICE_NAME'"
AND jsonPayload.timestamp!=""'
Grant permissions:
SERVICE_ACCOUNT=$(gcloud logging sinks describe senseon-nginx-sink --format='value(writerIdentity)')
gcloud pubsub topics add-iam-policy-binding senseon-nginx-logs \
--member=$SERVICE_ACCOUNT \
--role=roles/pubsub.publisher
Step 4: Deploy Log Forwarder Function
Create main.py:
import base64
import json
import requests
import functions_framework
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
COLLECTOR_URL = "https://<code-name>-logs.senseon.io/"
@functions_framework.cloud_event
def forward_logs(cloud_event):
"""
Forward real GCP logs to SenseOn collector.
Forwards RAW JSON logs without wrapping - exactly as customers would.
"""
try:
# Decode Pub/Sub message
pubsub_message = base64.b64decode(cloud_event.data["message"]["data"]).decode('utf-8')
log_entry = json.loads(pubsub_message)
# Extract service name for logging
resource = log_entry.get("resource", {})
resource_labels = resource.get("labels", {})
service_name = (
resource_labels.get("service_name") or
resource_labels.get("function_name") or
"unknown"
)
# Forward the raw log data directly (not wrapped)
if "jsonPayload" in log_entry:
# This is structured JSON log (like our nginx JSON)
senseon_log = log_entry["jsonPayload"]
# Add minimal source tag (customers would do this)
senseon_log["source"] = f"gcp-{resource.get('type', 'unknown')}"
senseon_log["_gcp_service"] = service_name
elif "textPayload" in log_entry:
# Plain text log - wrap minimally
senseon_log = {
"message": log_entry["textPayload"],
"timestamp": log_entry.get("timestamp"),
"source": f"gcp-{resource.get('type', 'unknown')}",
"_gcp_service": service_name
}
else:
# Skip logs without payload
logger.info(f"Skipping log without payload from {service_name}")
return
# Forward to SenseOn collector
response = requests.post(
COLLECTOR_URL,
json=senseon_log,
headers={"Content-Type": "application/json"},
timeout=10
)
if response.ok:
logger.info(f"Log forwarded: {service_name} (HTTP {response.status_code})")
else:
logger.warning(f"Unexpected status: {response.status_code} for {service_name}")
except Exception as e:
logger.error(f"Error forwarding log: {e}")
raise
Create requirements.txt:
functions-framework==3.*
requests==2.*
Deploy function:
gcloud functions deploy senseon-nginx-forwarder \
--gen2 \
--runtime=python311 \
--region=$REGION \
--source=. \
--entry-point=forward_logs \
--trigger-topic=senseon-nginx-logs \
--timeout=60s \
--memory=256MB
Step 5: Verify End-to-End
Generate traffic:
# Multiple requests to generate logs
for i in {1..10}; do curl $SERVICE_URL; done
Make sure that Cloud Run writes logs:
gcloud logging read \
'resource.type="cloud_run_revision"
AND resource.labels.service_name="'$SERVICE_NAME'"' \
--limit=5 \
--format=json
Make sure that the function forwards logs:
gcloud functions logs read senseon-nginx-forwarder \
--region=$REGION \
--limit=20
# Should see: "Log forwarded (HTTP 200)"
What Gets Sent to SenseOn
Nginx logs this:
{
"timestamp": "2025-10-17T10:00:00+00:00",
"remote_addr": "169.254.169.126",
"request": "GET / HTTP/1.1",
"status": "200",
"body_bytes_sent": 84,
"http_user_agent": "curl/7.88.1",
"http_x_forwarded_for": "203.0.113.42"
}
SenseOn receives:
{
"timestamp": "2025-10-17T10:00:00+00:00",
"remote_addr": "169.254.169.126",
"request": "GET / HTTP/1.1",
"status": "200",
"body_bytes_sent": 84,
"http_user_agent": "curl/7.88.1",
"http_x_forwarded_for": "203.0.113.42",
"source": "gcp-cloud_run_revision",
"_gcp_service": "nginx-cloudrun"
}
Cloud Run Application Logs
Configuration Overview
This example shows a custom Python application on Cloud Run. Cloud Logging forwards the logs. This same pattern works for: - Cloud Run services - Cloud Functions - App Engine - Compute Engine (with the Cloud Logging agent) - Any GCP service that writes to Cloud Logging
Architecture:
Application → Cloud Logging → Pub/Sub → Cloud Function → SenseOn
Prerequisites
- GCP project with billing enabled
gcloudCLI installed and authenticated- Cloud Run service that makes JSON logs (or any GCP service)
- Permissions to create Pub/Sub topics, sinks, and functions
Step 1: Configure Application to Log JSON
Your application must log JSON to stdout. This is a Python Flask example:
Create app.py:
"""
Real Data Processing Application
This is a realistic customer application that:
- Processes data requests
- Validates inputs
- Performs calculations
- Generates authentic application logs
"""
from flask import Flask, request, jsonify
import logging
import json
import time
import random
from datetime import datetime
app = Flask(__name__)
# Configure JSON logging (what customers would do)
class JSONFormatter(logging.Formatter):
def format(self, record):
log_data = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': record.levelname,
'logger': record.name,
'message': record.getMessage(),
}
# Add extra fields if present
if hasattr(record, 'duration_ms'):
log_data['duration_ms'] = record.duration_ms
if hasattr(record, 'status'):
log_data['status'] = record.status
if hasattr(record, 'user_id'):
log_data['user_id'] = record.user_id
if hasattr(record, 'request_id'):
log_data['request_id'] = record.request_id
return json.dumps(log_data)
# Set up JSON logging to stdout
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger = logging.getLogger('data_processor')
logger.addHandler(handler)
logger.setLevel(logging.INFO)
# Remove default Flask logger
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
def generate_request_id():
"""Generate a unique request ID"""
return f"req-{int(time.time())}-{random.randint(1000, 9999)}"
@app.route('/')
def home():
"""Health check endpoint"""
logger.info('Health check requested')
return {'status': 'healthy', 'service': 'data-processor'}, 200
@app.route('/process', methods=['POST'])
def process_data():
"""
Process data endpoint - this is what a real customer app does.
Generates authentic logs from real processing steps.
"""
request_id = generate_request_id()
start_time = time.time()
try:
# Log request received (REAL log)
logger.info(
'Processing request received',
extra={
'request_id': request_id,
'user_id': request.headers.get('X-User-ID', 'anonymous')
}
)
# Get and validate data (REAL operation)
data = request.get_json() or {}
items = data.get('items', [])
operation = data.get('operation', 'sum')
if not items:
# Log validation failure (REAL log)
logger.warning(
'Validation failed: no items provided',
extra={'request_id': request_id, 'status': 'invalid_input'}
)
return {'error': 'No items provided'}, 400
# Log processing start (REAL log)
logger.info(
f'Processing {len(items)} items with operation: {operation}',
extra={'request_id': request_id}
)
# Perform actual processing (REAL operation)
result = 0
if operation == 'sum':
result = sum(items)
elif operation == 'avg':
result = sum(items) / len(items) if items else 0
elif operation == 'max':
result = max(items) if items else 0
else:
# Log unsupported operation (REAL log)
logger.warning(
f'Unsupported operation: {operation}',
extra={'request_id': request_id}
)
return {'error': f'Unsupported operation: {operation}'}, 400
# Calculate duration
duration_ms = int((time.time() - start_time) * 1000)
# Log successful processing (REAL log)
logger.info(
'Processing completed successfully',
extra={
'request_id': request_id,
'status': 'success',
'duration_ms': duration_ms
}
)
return {
'request_id': request_id,
'result': result,
'items_processed': len(items),
'operation': operation,
'duration_ms': duration_ms
}, 200
except Exception as e:
# Log errors (REAL log)
duration_ms = int((time.time() - start_time) * 1000)
logger.error(
f'Processing failed: {str(e)}',
extra={
'request_id': request_id,
'status': 'error',
'duration_ms': duration_ms
}
)
return {'error': 'Internal server error', 'request_id': request_id}, 500
@app.route('/batch', methods=['POST'])
def batch_process():
"""
Batch processing endpoint - simulates batch jobs.
Generates multiple logs as it processes items.
"""
request_id = generate_request_id()
start_time = time.time()
# Log batch start (REAL log)
logger.info(
'Batch processing started',
extra={'request_id': request_id}
)
data = request.get_json() or {}
batches = data.get('batches', 5)
processed = 0
for i in range(batches):
# Simulate processing each batch item
time.sleep(0.1) # Simulate work
processed += 1
# Log progress (REAL log)
if i % 2 == 0: # Log every other item
logger.info(
f'Batch progress: {processed}/{batches} items processed',
extra={'request_id': request_id}
)
duration_ms = int((time.time() - start_time) * 1000)
# Log batch completion (REAL log)
logger.info(
'Batch processing completed',
extra={
'request_id': request_id,
'status': 'success',
'items_processed': processed,
'duration_ms': duration_ms
}
)
return {
'request_id': request_id,
'items_processed': processed,
'duration_ms': duration_ms
}, 200
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)
Create requirements.txt:
Flask==3.0.0
gunicorn==21.2.0
Create Dockerfile:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "2", "--threads", "4", "--timeout", "60", "--access-logfile", "-", "--error-logfile", "-", "--log-level", "error", "app:app"]
Step 2: Deploy to Cloud Run
# Set your project and region
export PROJECT_ID="your-project-id"
export REGION="europe-west2"
export SERVICE_NAME="data-processor"
gcloud config set project $PROJECT_ID
# Build and deploy
gcloud builds submit --tag gcr.io/$PROJECT_ID/$SERVICE_NAME
gcloud run deploy $SERVICE_NAME \
--image gcr.io/$PROJECT_ID/$SERVICE_NAME \
--region=$REGION \
--platform=managed \
--allow-unauthenticated \
--port=8080
# Get the service URL
SERVICE_URL=$(gcloud run services describe $SERVICE_NAME --region=$REGION --format='value(status.url)')
echo "Service URL: $SERVICE_URL"
Step 3: Create Pub/Sub Topic
gcloud pubsub topics create senseon-logs
Step 4: Create Log Sink
# Create sink that captures structured JSON logs from your Cloud Run service
gcloud logging sinks create senseon-log-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/senseon-logs \
--log-filter='resource.type="cloud_run_revision"
AND resource.labels.service_name="'$SERVICE_NAME'"
AND jsonPayload.level!=""'
Filter explanation:
- resource.type="cloud_run_revision" - Only Cloud Run logs
- resource.labels.service_name="data-processor" - Only your specific service
- jsonPayload.level!="" - Only structured JSON logs (filters out HTTP request logs)
Step 5: Grant Pub/Sub Permissions
# Get the service account from the sink
SERVICE_ACCOUNT=$(gcloud logging sinks describe senseon-log-sink --format='value(writerIdentity)')
# Grant publisher permission
gcloud pubsub topics add-iam-policy-binding senseon-logs \
--member=$SERVICE_ACCOUNT \
--role=roles/pubsub.publisher
Step 6: Create Log Forwarder Cloud Function
Create main.py:
import base64
import json
import requests
import functions_framework
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Replace with your SenseOn collector endpoint
COLLECTOR_URL = "https://<code-name>-logs.senseon.io/"
@functions_framework.cloud_event
def forward_logs(cloud_event):
"""
Forward real GCP logs to SenseOn collector.
Forwards RAW JSON logs without wrapping - exactly as customers would.
"""
try:
# Decode Pub/Sub message
pubsub_message = base64.b64decode(cloud_event.data["message"]["data"]).decode('utf-8')
log_entry = json.loads(pubsub_message)
# Extract service name for logging
resource = log_entry.get("resource", {})
resource_labels = resource.get("labels", {})
service_name = (
resource_labels.get("service_name") or
resource_labels.get("function_name") or
"unknown"
)
# Forward the raw log data directly (not wrapped)
if "jsonPayload" in log_entry:
# This is structured JSON log (like our nginx JSON)
senseon_log = log_entry["jsonPayload"]
# Add minimal source tag (customers would do this)
senseon_log["source"] = f"gcp-{resource.get('type', 'unknown')}"
senseon_log["_gcp_service"] = service_name
elif "textPayload" in log_entry:
# Plain text log - wrap minimally
senseon_log = {
"message": log_entry["textPayload"],
"timestamp": log_entry.get("timestamp"),
"source": f"gcp-{resource.get('type', 'unknown')}",
"_gcp_service": service_name
}
else:
# Skip logs without payload
logger.info(f"Skipping log without payload from {service_name}")
return
# Forward to SenseOn collector
response = requests.post(
COLLECTOR_URL,
json=senseon_log,
headers={"Content-Type": "application/json"},
timeout=10
)
if response.ok:
logger.info(f"Log forwarded: {service_name} (HTTP {response.status_code})")
else:
logger.warning(f"Unexpected status: {response.status_code} for {service_name}")
except Exception as e:
logger.error(f"Error forwarding log: {e}")
raise
Create requirements.txt:
functions-framework==3.*
requests==2.*
Step 7: Deploy Cloud Function
gcloud functions deploy senseon-log-forwarder \
--gen2 \
--runtime=python311 \
--region=$REGION \
--source=. \
--entry-point=forward_logs \
--trigger-topic=senseon-logs \
--timeout=60s \
--memory=256MB
Step 8: Verify End-to-End
Generate test logs:
# Test the /process endpoint
curl -X POST $SERVICE_URL/process \
-H "Content-Type: application/json" \
-H "X-User-ID: user-123" \
-d '{"items": [10, 20, 30], "operation": "sum"}'
# Test the /batch endpoint
curl -X POST $SERVICE_URL/batch \
-H "Content-Type: application/json" \
-d '{"batches": 5}'
Examine the Cloud Run logs:
gcloud logging read \
'resource.type="cloud_run_revision"
AND resource.labels.service_name="'$SERVICE_NAME'"' \
--limit=10 \
--format=json
Make sure that the Cloud Function forwarded the logs:
gcloud functions logs read senseon-log-forwarder \
--region=$REGION \
--limit=10
# Should see: "Log forwarded: data-processor (HTTP 200)"
What Gets Forwarded
Your application logs this (from a /process request):
{
"timestamp": "2025-10-17T10:00:00.123456Z",
"level": "INFO",
"logger": "data_processor",
"message": "Processing request received",
"request_id": "req-1697542800-1234",
"user_id": "user-123"
}
SenseOn receives this:
{
"timestamp": "2025-10-17T10:00:00.123456Z",
"level": "INFO",
"logger": "data_processor",
"message": "Processing request received",
"request_id": "req-1697542800-1234",
"user_id": "user-123",
"source": "gcp-cloud_run_revision",
"_gcp_service": "data-processor"
}
Multiple logs from a single request:
// 1. Request received
{
"timestamp": "2025-10-17T10:00:00.123456Z",
"level": "INFO",
"message": "Processing request received",
"request_id": "req-1697542800-1234",
"user_id": "user-123"
}
// 2. Processing
{
"timestamp": "2025-10-17T10:00:00.234567Z",
"level": "INFO",
"message": "Processing 3 items with operation: sum",
"request_id": "req-1697542800-1234"
}
// 3. Completion
{
"timestamp": "2025-10-17T10:00:00.345678Z",
"level": "INFO",
"message": "Processing completed successfully",
"request_id": "req-1697542800-1234",
"status": "success",
"duration_ms": 2
}
The result is clean, parsed JSON with minimal added metadata.
Troubleshooting
No logs forwarded:
# Check log sink is active
gcloud logging sinks describe senseon-log-sink
# Check Pub/Sub has messages
gcloud pubsub subscriptions list --filter="topic:senseon-logs"
# Check Cloud Function errors
gcloud functions logs read senseon-log-forwarder --region=$REGION --limit=50
Function returns errors:
- Make sure that COLLECTOR_URL in main.py is correct
- Make sure that the function has internet access. Gen2 functions must have a VPC connector for private networks
- Test the collector URL: curl -X POST $COLLECTOR_URL -H "Content-Type: application/json" -d '{"test":"connection"}'
Adapting for Other GCP Services
Cloud Functions:
- Use the same approach. Change the filter to resource.type="cloud_function"
Compute Engine:
- Install the Cloud Logging agent on the VMs
- Change the filter to resource.type="gce_instance"
GKE: - GKE automatically forwards to Cloud Logging - Use this pattern, or deploy a FluentBit DaemonSet (refer to the Kubernetes Configuration Guide) - Cloud Logging is easier but FluentBit gives more control
Why This Pattern Is Realistic
For GCP customers: - Uses native GCP services (no additional infrastructure) - Serverless (fully managed, no VMs to maintain) - Auto-scaling (handles any log volume) - Cost-effective for low-to-medium volumes - Simple code (< 50 lines of Python)
Common in production: - Microservices on Cloud Run - Serverless applications - Cloud-native organisations - Teams that want minimal operational overhead
Verification and Testing
Test Your Configuration
After you configure your log source, test the integration.
JSON over HTTPS:
curl -X POST https://<code-name>-logs.senseon.io/ \
-H "Content-Type: application/json" \
-d '{
"timestamp": "2025-10-14T15:30:45Z",
"source": "test",
"message": "Configuration test log"
}'
Syslog over TLS, from a host where you configured rsyslog as in Syslog (rsyslog):
logger -p user.info "senseon configuration test from $(hostname)"
Monitor Log Flow
- Look for errors in the logs of your log forwarder
- Make sure that the forwarder reads logs from the source files
- Monitor network connectivity to the ingestion endpoint
- Ask SenseOn support if the logs arrive
Troubleshooting
Common Issues
Logs are not forwarded
- Make sure that the forwarder service runs
- Make sure that the sender connects to <code-name>-logs.senseon.io by hostname, not by IP address
- Examine the error logs of the forwarder
- For JSON, make sure that the body is valid JSON and not newline-delimited JSON
- For syslog, make sure that you use octet-counted framing and a standard syslog header
Connection reset straight away - The sender does not send SNI, or syslog is not on for your organisation yet - Refer to Configuration Overview
Connection timeouts - Make sure that the hostname is correct - Make sure that the firewall rules allow outbound TCP 443 or 6514 - Examine the DNS resolution
Getting Help
If you have problems, do these steps:
- Read the main ingestion guide for the steps to troubleshoot common problems
- Examine the error logs of your forwarder
- Test the ingestion endpoint with
curloropenssl - Contact SenseOn support with configuration details and error messages
Need help?
Contact SenseOn support at support@senseon.io
When you contact support, include this information:
- Your organisation name
- Your ingestion endpoint hostname
- Example log entries (sanitised)
- Relevant error messages (if applicable)
- Log source type