Creating a dedicated log management layer
Action | Key |
---|---|
Play / Pause | K or space |
Mute / Unmute | M |
Select next subtitles | C |
Select next audio track | A |
Show slide in full page or toggle automatic source change | V |
Seek 5s backward | left arrow |
Seek 5s forward | right arrow |
Seek 10s backward | shift + left arrow or J |
Seek 10s forward | shift + right arrow or L |
Seek 60s backward | control + left arrow |
Seek 60s forward | control + right arrow |
Decrease volume | shift + down arrow |
Increase volume | shift + up arrow |
Decrease playback rate | shift + comma |
Increase playback rate | shift + dot or shift + semicolon |
Seek to end | end |
Seek to beginning | beginning |
Share this media
HLS video stream
You can use an external player to play this stream (like VLC).
HLS video streamWhen subscribed to notifications, an email will be sent to you for all added annotations.
Your user account has no email address.
Information on this media
Event logging is a central source of information both for IT security and operations, but different teams use different tools to collect and analyze log messages. The same log message is often collected by multiple applications. Having each team using different tools is complex, inefficient and makes a system less secure. Using a single application to create a dedicated log management layer independent of analytics has multiple benefits.
Collecting system logs with one application locally, forwarding the logs with another one, collecting audit logs with a different app, buffering logs with a dedicated server, and processing logs with yet another app centrally means installing several different applications on your infrastructure. And you might need a different set of applications for different log analysis software. Using multiple software solutions makes a system more complex, difficult to update and needs more computing, network and storage resources as well.
All of these features can be implemented using a single application which in the end can feed multiple log analysis software. A single app to learn and to follow in bug & CVE trackers. A single app to push through the security and operations teams, instead of many. Less resources needed both on the human and technical side.
In my talk I show you how to implement a log management layer using syslog-ng, as this is what I know best, but other applications have similar functionality. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis.
In an ideal world, all log messages come in a structured format, ready to be used for log analysis, alerting or dashboards. But in a real world only part of the logs belong to this category. Traditionally, most of the log messages come as free format text messages. These are easy to be read by humans, which was the original use of log messages. However, today logs are rarely processed by the human eye. Fortunately syslog-ng has several tools to turn unstructured and many of the structured message formats into name-value pairs, and thus delivers the benefits of structured log messages.
Once you have name-value pairs, log messages can be further enriched with additional information in real-time, which helps responding to security events faster. One way is adding geo-location based on IP addresses. Another way is adding contextual data from external files, like the role of a server based on the IP address or the role of the user based on the name. Data from external files can also be used to filter messages, for example to check firewall logs to determine whether certain IP addresses are contained in various black lists for malware command centers, spammers, and so on.
Logging is subject to an increasing number of compliance regulations. PCI-DSS or many European privacy laws require removing sensitive data from log messages. I will demonstrate how logs can be anonymized in a way that they are still useful for security analitics.
With log messages parsed and enriched you can now make informed decisions where to store or forward log messages. You can do basic alerting already in syslog-ng and receive critical log messages on a Slack channel. There are many ready to use destinations within syslog-ng, like Kafka, MongoDB or Elasticsearch. And you can easily create your own based on the generic network or HTTP destinations and using templates to log in a format as required by a SIEM or a Logging as a Service solution.
Speaker
Peter Czanik (One Identity)
Other media in the channel "2020"
- 17 viewsConclusion talkJuly 2nd, 2020
- 199 views, 11 this yearPique curiosity, not diabetic fingersJuly 2nd, 2020
- 37 viewsWars of the machines: build your own Seek and Destroy RobotJuly 2nd, 2020
- 51 viewsTackling security issues in virtualizationJuly 2nd, 2020
- 29 views, 3 this yearEnarx - secured, attested execution on any cloudJuly 2nd, 2020
- 55 views, 2 this yearRemote Forensic Investigations For The WinJuly 2nd, 2020