Log Ingestion Overview
Background
SenseOn makes a data fabric from the telemetry of its Universal Sensor, Integrations and network probes. The data in this fabric is highly curated and correlated. The data fabric gives immediate threat detection and response of high value. You can extend this visibility with the ingestion of logs. Logs give more context and more coverage of your digital estate.
Why?
Compliance and Auditing: Regulations, legal obligations or internal policies can make it mandatory for you to keep specific logs. A separate archive of raw logs lets you comply with these obligations. It also lets you give auditors the exact data that they ask for.
Bespoke Application Coverage: Some applications do not have a direct integration with SenseOn. Legacy and custom-built systems are frequently in this group. If you collect raw logs from these applications, you continue to have visibility into their activity. You can also correlate their events with other data sources.
Comprehensive Coverage: The SenseOn Universal Sensor and integrations cover a large range of data sources. But it is possible that SenseOn cannot integrate directly with some systems. If you collect raw logs from these systems, no part of your digital estate is a complete blind spot. Advanced users can use SQL to get access to all data in the data lakehouse that powers the SenseOn Intelligence Cloud. SQL lets them do deep, customised queries on this comprehensive dataset. With this capability, users can do complex analyses and get specific insights for their unique needs.
Choosing the right logs to ingest
SenseOn does not recommend that you send every possible log to the SenseOn platform. Start with the most valuable data sources. Then you get visibility and actionable intelligence quickly, and too much data does not overwhelm you. This guide gives the key log sources to prioritise, why they are important and the usual methods to collect them.
1. What are the gaps in your current visibility?
Before you start to ingest logs, you must know the gaps in your current visibility. To find these gaps, examine the visibility that you have now. The SenseOn Universal Sensor, your current integrations and your other security tools can give this visibility. Find the critical systems, applications or network segments that do not have sufficient coverage. Think about the types of attack that are probable for you. Then think about which logs give the most relevant data to detect and respond to these threats.
2. Is there an opportunity to reduce costs by sending logs to SenseOn instead of other platforms?
Examine the logs and tools that you have now. Find out if you can consolidate your log management in SenseOn. When you consolidate, you do not have to maintain more than one log management solution. This can decrease the costs of infrastructure, licences and operations. Base this analysis on financial and operational factors. Compare the costs and benefits of your current products, people and processes with those of SenseOn.
3. How will you get these logs to SenseOn?
SenseOn has many methods to ingest logs. Each method has its own advantages and considerations. The most common method is to host a log collector in your environment. For smaller deployments or specific use cases, it is possible that you can also use a cloud-hosted collector or a direct API integration. The Architecture document gives more details of the methods.
4. What is the estimated volume of logs?
You must know two things for each source: the log volume that you expect to ingest, and the business value that you want. High log volumes can increase costs. Estimate the daily log volume for each source that you plan to ingest. If necessary, SenseOn solution engineers can help you calculate this volume. This estimate helps you find the correct size for your deployment. It also helps you make sure that your log ingestion process is efficient and cost-effective.
5. Prioritise what logs need to have what level of service.
When the SenseOn platform ingests logs, it sends them to different log pipelines. The pipeline for a log depends on the processing that the log must have. Each pipeline has a different price. Thus you can give the highest level of capability to your most important logs, and you can manage costs effectively.
For example, you can send logs for audit to cold storage only. But send logs for real-time detection to pipelines where analytics can run on the data. The Architecture document gives more details about the pipelines and their capabilities.
Monitoring Log Ingestion in the Platform
When the configuration of your log sources is complete, you can monitor their status in SenseOn. Select Integrations in the main side menu.
The installed table shows your custom log sources together with your data connectors. Each source has its own row. A custom log row shows Custom log in the Type column. Its Category column is empty. To show only your custom logs, use the search box or the Type column filter. The Status column of each row shows one of these values:
- Connected: The logs arrive. Latest data received shows how recent the logs are.
- No logs received recently: The source has a configuration, but no logs arrived recently. Examine the forwarder or the source system. For more help, refer to Troubleshooting.
- Draft: The setup of the source started, but it is not complete.
Note: Custom log sources share the Integrations table with your data connectors. There is no longer a separate Log ingestion tab. For the full layout of the table, refer to Data Connectors.
To add a new custom log source, select Set up a new integration on the Integrations page. Scroll to the Can't find the integration you're looking for? tile at the end of the catalogue. Select Enquire about log ingestion. The Request for ingesting custom logs form opens. In the form, tell SenseOn about the source that you want to add.
SenseOn sets up new sources for you. To add a source or change an existing source, you can also contact your Customer Success Manager or support@senseon.io.