Creating a dedicated log management layer
Key | Action |
---|---|
K or space | Play / Pause |
M | Mute / Unmute |
C | Select next subtitles |
A | Select next audio track |
V | Show slide in full page or toggle automatic source change |
left arrow | Seek 5s backward |
right arrow | Seek 5s forward |
shift + left arrow or J | Seek 10s backward |
shift + right arrow or L | Seek 10s forward |
control + left arrow | Seek 60s backward |
control + right arrow | Seek 60s forward |
shift + down arrow | Decrease volume |
shift + up arrow | Increase volume |
shift + comma | Decrease playback rate |
shift + dot or shift + semicolon | Increase playback rate |
end | Seek to end |
beginning | Seek to beginning |
Share this media
HLS video stream
You can use an external player to play this stream (like VLC).
HLS video streamInformation on this media
Links:
Number of views:
22Creation date:
June 30, 2020Speakers:
Peter CzanikLicense:
CC BY-SA v4Description
Event logging is a central source of information both for IT security and operations, but different teams use different tools to collect and analyze log messages. The same log message is often collected by multiple applications. Having each team using different tools is complex, inefficient and makes a system less secure. Using a single application to create a dedicated log management layer independent of analytics has multiple benefits.
Collecting system logs with one application locally, forwarding the logs with another one, collecting audit logs with a different app, buffering logs with a dedicated server, and processing logs with yet another app centrally means installing several different applications on your infrastructure. And you might need a different set of applications for different log analysis software. Using multiple software solutions makes a system more complex, difficult to update and needs more computing, network and storage resources as well.
All of these features can be implemented using a single application which in the end can feed multiple log analysis software. A single app to learn and to follow in bug & CVE trackers. A single app to push through the security and operations teams, instead of many. Less resources needed both on the human and technical side.
In my talk I show you how to implement a log management layer using syslog-ng, as this is what I know best, but other applications have similar functionality. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis.
In an ideal world, all log messages come in a structured format, ready to be used for log analysis, alerting or dashboards. But in a real world only part of the logs belong to this category. Traditionally, most of the log messages come as free format text messages. These are easy to be read by humans, which was the original use of log messages. However, today logs are rarely processed by the human eye. Fortunately syslog-ng has several tools to turn unstructured and many of the structured message formats into name-value pairs, and thus delivers the benefits of structured log messages.
Once you have name-value pairs, log messages can be further enriched with additional information in real-time, which helps responding to security events faster. One way is adding geo-location based on IP addresses. Another way is adding contextual data from external files, like the role of a server based on the IP address or the role of the user based on the name. Data from external files can also be used to filter messages, for example to check firewall logs to determine whether certain IP addresses are contained in various black lists for malware command centers, spammers, and so on.
Logging is subject to an increasing number of compliance regulations. PCI-DSS or many European privacy laws require removing sensitive data from log messages. I will demonstrate how logs can be anonymized in a way that they are still useful for security analitics.
With log messages parsed and enriched you can now make informed decisions where to store or forward log messages. You can do basic alerting already in syslog-ng and receive critical log messages on a Slack channel. There are many ready to use destinations within syslog-ng, like Kafka, MongoDB or Elasticsearch. And you can easily create your own based on the generic network or HTTP destinations and using templates to log in a format as required by a SIEM or a Logging as a Service solution.
Speaker
Peter Czanik (One Identity)
Other media in the channel "2020"
17 viewsConclusion talkJuly 2nd, 2020
173 views, 3 this monthPique curiosity, not diabetic fingersJuly 2nd, 2020
37 views, 1 this monthWars of the machines: build your own Seek and Destroy RobotJuly 2nd, 2020
50 viewsTackling security issues in virtualizationJuly 2nd, 2020
26 views, 1 this monthEnarx - secured, attested execution on any cloudJuly 2nd, 2020
50 views, 1 this monthRemote Forensic Investigations For The WinJuly 2nd, 2020