Security Information and Event Monitoring (SIEM)

Part of the information security function involves the monitoring of our systems and networks. To be effective, this monitoring must be both “reactive”, i.e., reviewing log data to investigate circumstances surrounding a suspected system or information compromise, and “proactive”, i.e., continually collecting, correlating and evaluating events as they occur to detect an attack in its early stages so that it can be addressed before any significant damage is done.

Systems, applications, databases, networking devices and security products can generate millions of log records daily. While log records that are generated for each hardware device or software product can be stored independently on each device, there are mechanisms that can be configured to direct log record data that is generated by multiple hardware and software products to a single, centrally managed file system.

Unfortunately, storing log record data from many systems in a central system for easy retrieval is no small task. Each hardware and software product uses a different method of formatting its data, so the location of a user ID, an IP address or any other of the many data elements held will vary from one device type to another. So, for any product to be effective in linking log records together, it must provide a robust ability to “parse” records, i.e., to extract data element values from the raw log data.

But parsing is only the beginning. Once the log record data is parsed, data related to specific incidents across systems must be correlated in ways that help you analyze what is occurring.  For example, firewall log records may indicate that a certain network address is making unusual connections to many devices across your network. Once you have identified the suspicious device, you probably want to know who is logged into the device, so you need to match the network address from the firewall record to your authentication log records. Once you have found the user ID that is logged into the suspicious system, you may want to determine what other devices that user has accessed during the same time period, so you would match the user ID that is logged into the suspicious system against authentication and activity records for all of your other systems during the time frame being researched, and so on.

It should be noted that this is actually a very simple example. The kinds of correlations needed to uncover attacks in progress could become very complex. Add to that the fact that your hardware and software are generating millions of log records, and you can see that monitoring everything proactively from raw log records using a manual process is virtually impossible.

Automated SIEM systems

Since their arrival in the early 1990s, automated SIEM systems have provided functionality to:

• Collect log data from many different types of operating systems, databases, network security devices and applications
• Parse the log data to extract pertinent information, such as information about the reporting device, the ID of the user involved, source and target device addresses and ports, event descriptions, etc.
• Correlate each entry in the parsed SIEM logs to pull together related events
• Allow users to use standard log data correlations or create custom ones
• Run standard queries and reports “out-of-the-box” or allow users to create ad hoc queries and reports with a built in query tool and report writer
• Show the state of the data processing environment through dashboards that can be customized by each authorized user
• Issue alerts to appropriate personnel when certain event types are detected

There are numerous SIEM products in the marketplace, each with its own strengths and weaknesses. When selecting a SIEM solution, consider how each product has implemented each of the components described in the next section.

SIEM Components

A SIEM system is an application program that typically is installed on a dedicated server capable of handling a large amount of network traffic and data storage. SIEM systems usually are comprised of multiple components that run in parallel to collect log records, parse them, store extracted data elements, correlate related events, interpret correlation results, alert appropriate individuals and groups and create and run queries and reports.

Logging

The logging function of a SIEM system is a powerful collection engine to which all participating devices are configured to send their log records. There are three primary ways that the records are sent to the SIEM:

• Some devices have the ability to send (“push”) log records to the SIEM one-by-one as they are generated by the system
• Some require that an agent be installed on the participating device to send (“push”) the latest batch of log records after a configurable time period, e.g., every minute
• Some require the SIEM to log into the device and “pull” the log records from the participating system’s own security logs. This would require the SIEM to store a valid user ID and password that has the authority to read the necessary log files

The method used is based on the source device type. Some devices can support both push and pull methods of log record collection.

Parsing

As devices are defined to the SIEM, they are categorized by device type so that the SIEM’s parsing engine knows how to interpret the data, i.e., what kinds of log records are generated by the device type, and for each log record type, the data elements that are contained in the record, how the data are held, each data element's position in the record or its associated keyword, etc. Some log records hold their data positionally, e.g., the user name starts in position 10 for a maximum length of 30 characters, and some organize their data in a freeform format using key words, e.g., username=whatever, IPaddress=123.45.67.89.

Storing

As each record is parsed, the data values are added to the appropriate SIEM database tables in a consistent format based on device type.

Correlating

Up to this point, the steps being performed are fairly straightforward to configure.  Correlation is the first step that requires significant forethought. Basically, the correlation engine joins log records together to provide a big picture of the events upon which you may want to focus, e.g., what a specific user account is doing at a given point in time, what devices he or she is connected to, what errors are being generated across all the systems for that user. There are many types of correlations that can be made, and SIEM systems generally provide tools that can help administrators build correlation rules. Fortunately, SIEM systems also provide a number of predefined correlations that cover many potential attack scenarios.

Interpreting

As specific correlated situations occur, the SIEM keeps track of how many times certain correlations are being made across specified periods of time, either by a specific user account, a specific device or set of devices, or across the University as a whole. Once certain thresholds are met the correlated conditions are noted in the database.

Alerting

Detected correlation conditions are reviewed to determine whether or not the conditions require the system to alert specific individuals to address them. To accomplish this, a SIEM administrator enters contact information for the individual or group of individuals who should be alerted when the specific correlation or set of correlations are detected.

Reporting

Since all of the log data is stored in a set of relational database tables, queries and reports can be easily generated against the basic and correlated data using the SIEM’s reporting tool. Most SIEM systems provide a robust set of predefined queries and reports to facilitate the implementation, and a mechanism for building customized queries and reports.

The reporting subsystem also allows each administrator to have his or her own customizable dashboard that provides summary information that is presented in both textual and graphical form. This feature is widely used by network operations center staff who are continually monitoring network and server health.