Ingesting Syslog from Network Appliances
This guide covers firewalls, switches, routers, VPN concentrators and other network appliances that send syslog. SenseOn receives syslog over TLS on TCP 6514 at <code-name>-logs.senseon.io. For the requirements, refer to Configuration Overview. The setup that you build depends on the capabilities of each appliance.
Choose a route per appliance
| The appliance can | Route |
|---|---|
| Send syslog over TLS to a hostname, with SNI. For example FortiOS 7.4. | Direct. Set the appliance to send to <code-name>-logs.senseon.io, TCP 6514, TLS. A relay is not necessary. |
| Send syslog over TLS, but only to an IP address, or without SNI. For example FortiOS 7.2. Also use this row if your firewall allows outbound rules by IP address only. | Dedicated port, reachable only from your egress IP addresses. Refer to Dedicated port route. A relay also works. |
| Only send plain syslog over UDP or TCP to an IP address. Most switches, older firewalls, UPS network cards, and hypervisor hosts. | Relay. Refer to Relay route. |
A common setup is that firewalls send directly, and all other appliances send through a relay. If the documentation of an appliance does not mention SNI, assume that the appliance does not send SNI. Keep this assumption until a test shows that the appliance sends SNI.
Direct route
On the appliance, set these values:
| Setting | Value |
|---|---|
| Server | <code-name>-logs.senseon.io (the hostname, not an IP address) |
| Port | 6514 |
| Protocol | TCP with TLS. Some vendors call this "reliable" or "encrypted" syslog. |
| Format | The default format of the appliance. RFC 5424, if the appliance offers it. |
| Certificate validation | On, against the public CA store. A client certificate is not necessary. |
Allow outbound TCP 6514 from the log interface of the appliance to the hostname. Make sure that no proxy on that path inspects TLS. Then do a check of the flow.
Dedicated port route
This route is for appliances that support TLS but cannot use the hostname as their destination.
- Tell SenseOn support the public IP addresses that the logs of the appliances leave your network from. These addresses must be fixed. Only these addresses can get to the port.
- SenseOn allocates the port, and gives you the IP addresses and the port number. Before you change the appliances, wait until SenseOn tells you that the port is ready.
- On each appliance, set these values:
- Server: one of those IP addresses, and the port. If the appliance allows more than one destination, you can set several.
- Protocol: TCP with TLS.
- Format: the default format of the appliance.
- Set the appliance to do a check of the certificate against
<code-name>-logs.senseon.io. Usually, the setting has the name peer name, server name or common name. The certificate does not contain an IP address. Thus a check against the IP address does not work. - Allow outbound TCP on that port from the appliance to those IP addresses, with no TLS inspection.
Some appliances cannot set a name for the certificate check, and can only turn the check off. If you turn the check off, the appliance still connects. But the appliance can no longer make sure that it communicates with SenseOn. For such an appliance, a relay is the better option.
Relay route
A relay is a small Linux host in your network. It receives plain syslog from your appliances. It forwards the text of each message to SenseOn over TLS without changes. A disk queue of up to 2 GB holds messages while SenseOn is unreachable. The relay does not convert messages to JSON, because SenseOn recognises each product by its native format.
Firewalls, switches, VPN concentrators
-> plain syslog over UDP or TCP 514, to the relay's IP address
-> relay in your estate (rsyslog)
-> syslog over TLS, TCP 6514, with SNI and a disk queue
-> <code-name>-logs.senseon.io
Only the relay must have access to the Internet. For log forwarding, the appliances must have access to the relay, not to the Internet.
Prerequisites
| Item | Value |
|---|---|
| Operating system | Debian 12 or Ubuntu 22.04 and later, minimal install. The RHEL family also works. Refer to the notes after the configuration. |
| Resources | 2 vCPU, 4 GB memory, 40 GB disk. The queue uses up to 2 GB under /var/spool/rsyslog. |
| Network | A static IP address reachable from the appliance management network. Outbound TCP 6514 to <code-name>-logs.senseon.io by hostname, not through a proxy that inspects TLS. DNS resolution for that hostname. |
| Per site | One relay for each site keeps appliance traffic local, and continues to operate during a WAN outage. The configuration is the same at each site. |
| Access | Administrative access to each appliance to change its log destination. |
Before you start, ask SenseOn to turn on syslog for your organisation. Until SenseOn turns it on, port 6514 resets each connection.
1. Configure the relay
Install rsyslog with its GnuTLS driver:
sudo apt-get update
sudo apt-get install rsyslog rsyslog-gnutls
On the RHEL family, use sudo dnf install rsyslog rsyslog-gnutls.
Create /etc/rsyslog.d/10-senseon-relay.conf. In the file, replace <code-name> with your code name:
# SenseOn relay. Receives plain syslog from the LAN and forwards it to
# SenseOn over TLS with a disk queue. The relay's own system logs are
# not forwarded: only the two network inputs feed this ruleset.
global(defaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt")
ruleset(name="forward_to_senseon") {
action(
type="omfwd"
target="<code-name>-logs.senseon.io"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="*.senseon.io"
TCP_Framing="octet-counted"
template="RSYSLOG_ForwardFormat"
queue.type="linkedList"
queue.filename="senseon_relay_fwd"
queue.spoolDirectory="/var/spool/rsyslog"
queue.maxdiskspace="2g"
queue.saveonshutdown="on"
action.resumeRetryCount="-1"
)
}
module(load="imudp")
input(type="imudp" port="514" ruleset="forward_to_senseon")
module(load="imtcp")
input(type="imtcp" port="514" ruleset="forward_to_senseon")
Keep these settings as shown:
StreamDriver="gtls". The OpenSSL driver (ossl) in many rsyslog releases sends no SNI. As a result, the endpoint resets each connection.StreamDriverPermittedPeers="*.senseon.io". The name check of rsyslog rejects the full hostname.TCP_Framing="octet-counted"andtemplate="RSYSLOG_ForwardFormat". A custom template that removes the<priority>header stops ingestion.RSYSLOG_ForwardFormatdoes not carry RFC 5424 structured data. A device can put fields that are important to you in structured data. In this case, useRSYSLOG_SyslogProtocol23Formatfor the traffic from that device. Then test it with a real message from the device.- The dedicated ruleset. With it, the relay forwards only the messages that arrive from the network, not its own system logs.
On the RHEL family, set defaultNetstreamDriverCAFile to /etc/pki/tls/certs/ca-bundle.crt. To listen on a high port such as 5140, change both inputs. Then set the appliances to send to that port, and use that port in place of 514 in the firewall rule and the checks that follow.
Start the service. Then make sure that it listens:
sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo systemctl status rsyslog
sudo ss -lntup | grep 514
2. Allow the traffic
- On the relay host, allow inbound UDP and TCP 514 from your appliance management addresses only.
- On the network path, allow traffic from the appliances to the relay on that port.
- Allow outbound traffic from the relay to
<code-name>-logs.senseon.ioon TCP 6514, by hostname, with no TLS inspection.
3. Point each appliance at the relay
Each vendor uses a different name for the setting. But you always look for the same four items:
- A remote syslog server address
- A port
- A protocol
- The facilities or severity levels to send
Look for a section with a name such as remote logging, syslog server, log forwarding or log settings. For the menu path, the documentation of the vendor is the authority.
On appliances such as Fortinet, Palo Alto, Cisco, Meraki, Cato, WatchGuard and SonicWall devices, set these values:
| Setting | Value |
|---|---|
| Server address | The IP address of the relay host |
| Port | 514, or the port that you selected |
| Protocol | TCP if the appliance supports it. If not, UDP |
| Format | The default of the appliance. Do not change to JSON. |
| Facility and severity | Start with the defaults. If the volume is higher than you want, narrow the severity later. |
Send from one appliance first. Make sure that it works end to end. Then add the other appliances.
Verify the flow
Do the checks in sequence, from the appliance outwards.
Do a check of the endpoint from the relay, or from a host on the same path as a direct appliance:
openssl s_client -connect <code-name>-logs.senseon.io:6514 \
-servername <code-name>-logs.senseon.io \
-verify_hostname <code-name>-logs.senseon.io -verify_return_error -brief </dev/null
If the endpoint works, the output shows Verification: OK and Verified peername: *.senseon.io, with a certificate from Let's Encrypt. This test must use OpenSSL 1.1 or later. The LibreSSL build that comes with macOS does not have -verify_hostname.
A reset with no certificate means one of two conditions. Syslog is not on for your organisation yet, or something on the path removed the server name. A certificate from a different issuer means that a proxy inspects TLS.
For the relay route, make sure that messages arrive at the relay. Monitor the port directly:
sudo tcpdump -n -i any port 514
You can also send a test message from any Linux host that can get to the relay. This test shows that the listener works, independently of the appliance:
# UDP
logger -n <relay-ip> -P 514 -d "senseon relay connection test"
# TCP
logger -n <relay-ip> -P 514 -T "senseon relay connection test"
Make sure that the relay forwards the messages. If the relay is healthy, it shows no omfwd or gtls errors, and the queue files under /var/spool/rsyslog stay small:
sudo journalctl -u rsyslog --since "5 minutes ago"
ls -lh /var/spool/rsyslog
Make sure that the data arrived. When you send a test message, tell SenseOn support. SenseOn then tells you if the message arrived.
Reliability and volume
- UDP loses messages silently. UDP has no delivery guarantee and no back pressure. Thus a busy firewall can overrun the listener, and neither end reports an error. If the appliance supports TCP, use TCP between the appliance and the relay.
- The relay is a single point of failure. If the relay is down, the appliances do not try again. For high-value sources, run two relays. If an appliance allows more than one destination, set it to send to both relays.
- The disk queue is bounded. It holds up to 2 GB while SenseOn is unreachable. When the path is available again, the queue empties. If an outage is sufficiently long to fill the queue, the relay loses messages.
- Firewall logs have a high volume. To decrease the volume at the lowest cost, narrow the severity on the appliance.
For information about which pipeline each source must feed, refer to Architecture.
Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
Nothing in tcpdump on the relay |
The appliance does not send, or something blocks the traffic | Make sure that the syslog destination and port of the appliance are correct. Examine the firewall rules on the path and on the relay host. |
SSL_ERROR_SYSCALL or ECONNRESET on every attempt in the relay's journal |
The relay sends no SNI, the CA file is missing, syslog is not on for your organisation yet, or outbound 6514 is blocked | Make sure that StreamDriver="gtls" is set and that the defaultNetstreamDriverCAFile path exists. Do the openssl check in Verify the flow |
peer name not authorized |
StreamDriverPermittedPeers is set to the full hostname |
Set it to *.senseon.io |
warning: certificate file is not set or key file is not set in the journal |
rsyslog notes that it has no client certificate | No action is necessary. SenseOn does not use client certificates. |
| The connection drops, or nothing arrives, when messages flow | A custom template removed the <priority> header |
Restore template="RSYSLOG_ForwardFormat" |
cannot resolve |
The hostname is wrong, or DNS on the relay does not work | The hostname is <code-name>-logs.senseon.io, with a hyphen before logs |
Queue files grow under /var/spool/rsyslog |
SenseOn is unreachable from the relay | Examine outbound 6514. Make sure that no proxy inspects TLS on the path |
| A direct appliance never connects | The appliance sends no SNI, or connects to an IP address | Use the dedicated port route or the relay route |
| An appliance on a dedicated port times out | Its logs leave from an IP address that SenseOn does not have, or the port is not ready yet | Ask your network team for the egress address. Send the address to SenseOn support |
| Certificate checks do not pass on an appliance with a dedicated port | The appliance does a check of the certificate against the IP address | Set its peer name or server name to <code-name>-logs.senseon.io |
Need help?
Contact support@senseon.io and send this information:
- Your organisation name
- Your ingestion hostname
- The make, model and version of the appliance
- If the appliance sends directly or through a relay
- A sanitised raw message
- For a relay: its configuration file, the output of
rsyslogd -vand the relevant journal lines
Additional Documentation
- Configuration Overview - Receive, Relay, and Retrieve modes
- Configuration Examples - Detailed setup guides for other sources
- Architecture - Deployment models and log pipelines
- FAQ & Troubleshooting - Quick answers and common patterns