Skip to content

Streaming Logs from AWS CloudTrail and CloudWatch

This guide is for customers who want to send AWS CloudTrail events and CloudWatch Logs (such as application or API logs) into SenseOn. Both use the same mechanism: a small forwarder Lambda in your own AWS account, subscribed to whichever CloudWatch Log Groups you want ingested. CloudTrail rides this same path by enabling its CloudWatch Logs delivery, rather than a separate S3-based route.

How it works

Your log producers (CloudTrail, applications)
        -> CloudWatch Log Group (your account)
        -> Subscription filter
        -> Forwarder Lambda (your account)
        -> HTTPS POST -> SenseOn ingestion endpoint

You deploy the Lambda and point it at the ingestion endpoint SenseOn provides for your environment, attach a subscription filter per log group you want forwarded, and grant CloudWatch Logs permission to invoke it. Everything runs in your own account, nothing is pulled from it, and no cross-account IAM role or access into your account is required for this path.

Prerequisites

  • Permissions in your AWS account to create a Lambda function, an IAM role/permission for it, and CloudWatch Logs subscription filters.
  • If you want CloudTrail included: an existing CloudTrail trail (or create a new one).
  • The ingestion endpoint URL for your environment, provided by your SenseOn contact.

Step 1: Enable CloudWatch Logs delivery on your CloudTrail trail

If you want CloudTrail events ingested, open your trail and enable CloudWatch Logs delivery, pointing at a dedicated log group.

This is additive: your trail keeps writing to its existing S3 destination exactly as before, this only adds a second, parallel delivery target. Nothing about your current CloudTrail setup needs to change.

If you're only sending CloudWatch application logs and not CloudTrail, skip this step.

Step 2: Identify your CloudWatch log groups

Note the name of every CloudWatch Log Group you want SenseOn to receive, your application/API logs, and (if applicable) the CloudTrail-destined log group from Step 1.

Step 3: Deploy the forwarder Lambda

Create a new Lambda function:

  • Name: SenseOn-Forwarder
  • Runtime: Python 3.12
  • Timeout: Under Basic settings, increase the execution timeout to 1 minute. The default of 3 seconds is too short for the code's own 10 second HTTP timeout to ever take effect, so ordinary network latency alone could get the function killed and trigger unnecessary CloudWatch Logs retries.

Paste the following directly into the console's default code editor (it already matches AWS's default filename and handler, lambda_function.py / lambda_handler, so no renaming or Handler configuration is needed):

import base64
import gzip
import json
import os
import urllib.request
from datetime import datetime, timezone

INGESTION_ENDPOINT_URL = os.environ["INGESTION_ENDPOINT_URL"]
DEBUG_LOG_ENVELOPES = os.environ.get("DEBUG_LOG_ENVELOPES", "") == "true"


def _to_iso8601(epoch_ms):
    return datetime.fromtimestamp(epoch_ms / 1000, tz=timezone.utc).strftime(
        "%Y-%m-%dT%H:%M:%S.%f"
    )[:-3] + "Z"


def _envelope(log_group, log_stream, event):
    return {
        "metadata": {
            "original_log_group": log_group,
            "resource_id": log_stream,
        },
        "log_event": {
            "eventId": event["id"],
            "logStreamName": log_stream,
            "timestamp_utc": _to_iso8601(event["timestamp"]),
            "message": event["message"],
        },
    }


def lambda_handler(event, _context):
    payload = json.loads(gzip.decompress(base64.b64decode(event["awslogs"]["data"])))
    log_group = payload["logGroup"]
    log_stream = payload["logStream"]

    envelopes = [_envelope(log_group, log_stream, e) for e in payload["logEvents"]]
    body = json.dumps(envelopes).encode("utf-8")

    if DEBUG_LOG_ENVELOPES:
        print(json.dumps(envelopes[:1]))

    headers = {"Content-Type": "application/json"}
    request = urllib.request.Request(
        INGESTION_ENDPOINT_URL, data=body, headers=headers, method="POST"
    )
    with urllib.request.urlopen(request, timeout=10) as response:
        response.read()

    return {"forwarded": len(envelopes)}

Environment variables

Variable Required Purpose
INGESTION_ENDPOINT_URL Yes The SenseOn ingestion endpoint provided for your environment.
DEBUG_LOG_ENVELOPES No Set to exactly true temporarily to have the function print the payload it sends, viewable in its own CloudWatch Logs group. Leave unset otherwise.

Step 4: Grant CloudWatch Logs permission to invoke the Lambda

CloudWatch Logs needs explicit permission to call this Lambda for a given log group, it isn't automatic.

Console: if you create the subscription filter (Step 5) through the CloudWatch console and choose this Lambda as the destination, the required permission is added for you automatically. Nothing else to do.

AWS CLI: if you use aws logs put-subscription-filter directly, add the permission first, or the command fails with an access error:

aws lambda add-permission \
  --function-name SenseOn-Forwarder \
  --statement-id AllowCloudWatchLogsInvoke-<log-group-short-name> \
  --action lambda:InvokeFunction \
  --principal logs.<region>.amazonaws.com \
  --source-arn arn:aws:logs:<region>:<account-id>:log-group:<log-group-name>:* \
  --source-account <account-id>

Give each log group's permission a unique --statement-id if you're forwarding more than one (CloudTrail's and an application log group both need their own).

Step 5: Create the subscription filter

One per log group you want ingested. Name: SenseOn-Forwarder-SF, this name only needs to be unique within its own log group, so it can be reused across every log group you subscribe.

Point it at the Lambda from Step 3. A blank filter pattern forwards every event from that log group.

⚠ Think before excluding events by type. If you're sending CloudTrail for security monitoring, avoid filtering out read-only calls (Describe*/List*/Get*). These are exactly how AWS account enumeration and reconnaissance activity shows up, filtering them out for volume reasons can blind detection to the activity it's meant to catch. If volume genuinely needs managing, filtering by identity type (for example excluding calls made by AWS service principals) is a safer lever than filtering by verb.

Step 6: Verify it's working

Trigger an event in the source log group (any AWS API call generates a CloudTrail event; a test log line works for application log groups), then check the Lambda's own CloudWatch Logs group (/aws/lambda/SenseOn-Forwarder). A clean run with no exceptions means it reached the ingestion endpoint successfully.

Troubleshooting

[ERROR] Runtime.HandlerNotFound: Handler 'lambda_handler' missing on module 'lambda_function' This shouldn't come up given the code in Step 3 already matches AWS's default filename and handler, but if the source was renamed or pasted into a differently-named file, go to the Code tab's Runtime settings panel and confirm Handler reads lambda_function.lambda_handler, and that the source file is named lambda_function.py.

Task timed out after 3.00 seconds The function's execution timeout wasn't raised in Step 3. Go to Basic settings and confirm Timeout is set to at least 1 minute, not the AWS default of 3 seconds.

Good practice

  • Additive, not disruptive: enabling CloudWatch Logs delivery on an existing CloudTrail trail never changes or replaces its existing S3 delivery.
  • No cross-account access required: this is a push model from your account, SenseOn never needs a role or credentials into your AWS account for this path.
  • Filter deliberately, not by default: see the warning in Step 5, unfiltered is the safer starting point for a source used for detection.
  • Turn off DEBUG_LOG_ENVELOPES when done: it's a diagnostic aid, not something to leave on, since it prints real log content into CloudWatch.