As the Security Information and Event Management (SIEM) market has matured, products within the market have lost the ability to quickly respond to threatening situations, and no longer meet the requirement to be a real-time decision support system (RTDSS).
The root cause of the problem is the woefully inadequate scalability and performance characteristics of the underlying data management technologies used within these products; technologies that were not designed to address the requirements of this domain. The ever-growing types, volumes and rates of relevant security information have exposed these fundamental design shortcomings.
Simply put, as the SIEM market has evolved, the market's products have devolved - they no longer function as viable solutions for information security, and instead are limited to the role of compliance reporting tools that can only generate actionable intelligence in a few hours or days.
The next evolution of SIEM overcomes the performance limitations of its predecessors. More systems must be monitored. All activity must be examined, and in greater detail-all the way into the contents of applications and protocols. Most importantly, all of this information and the context around it must be readily available to the analyst, in order to provide real-time decision support. The new SIEM must be content-aware, highly scalable, and lightening fast. What is this new SIEM? It's called: NitroView
All Security Information & Event Management systems (SIEMs) work in the same basic fashion: they collect information from a variety of sources, store that information, and provide a layer of analytics and reporting against it.
Why is database technology so important to SIEM? All security information managed by SIEM needs to be collected stored and managed ... and the database is the weak link that has prevented most SIEMs from evolving.
As more data is collected, the reaction time of the database decreases in every possible way.
The keystone of the next evolution in SIEM technology, therefore, is the database. Without first overcoming the fundamental challenges of highly scalable, rapid data I/O and analysis, no SIEM can hope to operate effectively as a security operations tool.
When a SIEM collects an "event" (within this document, the term "event" will be used to identify any relevant piece of data obtained from a device notification, log, direct monitoring, or any other source), that event must be stored. Most SIEMs use either a commercial database (MySQL, Oracle, etc.), a commercially-derived database (an SQL variant), or a flat-file (a text-based data store). Upon collection, events are initially analyzed in order to identify the most important indicators of a threat. Such indications are presented via some sort of threat notification, by email, SNMP, or other means, and are often separated within the SIEM console to facilitate security management. Dashboards are often used to display summaries of the threats, although these dashboards are almost always static snapshots of very specific periods of time.
Upon collection, the information source may either be parsed and indexed, or kept in its raw format. Raw storage is faster, yet provides limited analytical capabilities; indexed storage is typically slower, yet allows for correlation, data filtering, pivoting, and any number of other statistical and analytical operations. Because of this, most SIEM solutions parse and index information upon collection, while most Log Management solutions store raw log files. As the market evolved, Log Management and SIEM products converged, although because of this disparity no truly cohesive solution was found.
While unstructured data management is suitable for time-insensitive functions such as Log Management, the value of SIEM lies in the ability to manage structured information. The value of a SIEM therefore increases as the detail and granularity of the information being managed increases. Information is easier to manage if common data points-IP address, user names, normalized event descriptions, etc.-are organized and readily available. When an event source is parsed upon collection, the important information from that event is separated into defined indices within the database.
However, all commercially available database and flat-file storage systems degrade in performance as the depth of indexing increases. Most SIEMs use thin event indexing to provide balance of manageability and performance. Indices consisting of: a timestamp; event identifiers; source and destination information; and a normalized event identifier (if supported) are adequate for high- level event and threat management, yet put less strain on the underlying system. Some more ambitious SIEMs opt to provide "thick" event indexing, expanding the number of indexes to include as many as a dozen relevant data-points, willingly sacrificing speed for the sake of more robust (yet much less responsive) analysis. Others index minimally-or not at all-for the sake of performance, and sacrifice data manageability instead.
The evolution of SIEM, not surprisingly, is highly dependent upon the ability to manage structured and unstructured data simultaneously. For structured analysis, data indexing is extremely important, as it is the fundamental mechanism used to search, compare, and analyze data in a meaningful manner. Without indexing, each analytical operation would require multiple, complete full-text search of all collected information, which would render most security functions inoperable.
Unfortunately, even using optimized SQL or flat-file systems, performance of most functions begins to slow after the data store has either:
When even small networks can generate events at a rate of tens-of-thousands per secondError! Bookmark not defined., this fundamental performance limitation has relegated the SIEM to the primary role of a post-incident reporting tool, providing limited value (if any) to real-time decision-making. To evolve, the SIEM must grow in both its breadth (scale) and depth (granularity) ... without losing its ability to analyze information in real time.
These underlying architectural deficiencies result in several inherent limitations. While these limitations may be problematic on their own, they also impede the evolution to more recent requirements of security information and event management systems (see 'Evolving SIEM Requirements' below).
Most legacy SIEM architectures are unable to scale beyond a few 10,000's of events per second. Very high-end systems, using highly distributed database back-ends, may be able to achieve 100,000 events per second, but at elevated costs due to extreme hardware and processing requirements. While typical event rates in small to mid sized networks can easily reach 15,000 events per second, these limitations prevent the legacy SIEM from expanding visibility into deeper monitoring of database activity, applications, protocols, and network activity. These limitations also present deployment challenges in larger enterprise networks.
Most legacy SIEM architectures will begin to show performance degradation after collecting just a million total events. However, event rates could potentially reach hundreds-of-millions of events per day, even in small to mid-sized networks.
Most legacy SIEM architectures, due to the above limitation of total event volumes, are only capable of analyzing short periods of time: often less than thirty days, and in some cases only 24 hours. However, compliance requirements mandate a minimum of 90 days, and up to seven years of data retention; and forensics investigations often require the analysis of months or years of information.
According to Forrester's 2009 Market Overview or Security Information & Event Management, SIEMs need to perform the basic functions of collecting information; distilling it; storing it; hardening the logs (for compliance); and producing reports.
However, the use of SIEM as an operational tool only occurs when the collection and distillation of information (getting data into a SIEM) and the reporting capabilities (getting information out of a SIEM) converge into an ongoing, real-time analysis. Not surprisingly, many legacy SIEM solutions are unable to achieve the simultaneous performance required to collect information and process it at high rates, and provide detailed reports and dashboards in real-time, so that any information or assessment stored within the SIEM can be produced as needed for the purposes of threat investigations, incident management, or remediation.
This evolution ultimately requires improved performance and scalability of the entire SIEM architecture. Next-generation SIEMs such as NitroView Enterprise Security Manager must be capable of scaling well beyond the limitations of current systems. Next-generation SIEMs must be able to support hundreds of thousands of events per second, without impacting the analysis and reporting capabilities of the solution. These volumes of events must be kept available for analysis for longer periods of time-from at least 90 days, to up to several years.
Likewise, analysis and reporting performance must increase to the point where full historical information queries, relational lookups, and statistical operations occur in near-real-time-in less than a minute, regardless of the total amount of information being stored, and without impacting the systems ability to collect new information.
Expanding the underlying scalability and performance capabilities of the SIEM allows the SIEM to evolve, providing added value from each of its core functions:
Legacy information management systems relied almost exclusively on device logs, including server logs, host logs, application logs, logs from firewalls, VPNs, intrusion prevention systems, etc.
However, as SIEM evolves logs alone are no longer sufficient. Information that is necessary for security and compliance practices may not be available from logs, and so dedicated monitoring of critical assets is necessary. This is evident in the upsurge in Database Activity Monitoring (DAM) solutions, which monitor and track each and every database transaction. Likewise, Deep Packet Inspection (DPI) is increasingly used to monitor the contents of applications, documents, and protocols in an effort to gain a better insight into how data is being accessed and used.
Threat detection in SIEM has occurred through "event correlation," or the examination of event data to determine patterns, which in turn might indicate a larger threat. In this way, the SIEM was able to notify security analysts of possible threats that might otherwise be lost in the growing sea of event data. However, due to performance limitations of legacy systems, correlation had to be performed entirely in memory. While in-memory analysis is fast enough to detect most threats, these systems lack scalability: memory is finite, limiting correlation to relatively short time periods.
As SIEM evolves, the limitations of event correlation are also being overcome. The performance advantages of the next-generation SIEM allow stored data to be used in threat detection, supplementing the in-memory correlation that occurs during data collection. The result is a broader view of all security event data, and therefore a better overall detection capability-capable of detecting "low and slow" attacks, and even helping to identify unknown threats, or "zero day" correlation (see sidebar: the Changing face of Correlation).
There are several types of event correlation, all of which share a common goal: to find patterns indicative of larger threats from within the deluge of individual events. At it's most basic level, correlation looks for a sequence of events of time: "if event A is followed by event B, within a given time-frame, assume the possibility of threat X." Slightly more advanced correlation will abandon the condition of sequence, and will indicate that a combination of events, in any order, might indicate a threat: "if events A and B occur, in any order, within a given time-frame, assume the possibility of threat X." Both of these mechanisms rely on a finite period of observation: if the patterns do not fully appear within five minutes or ten minutes, the SIEM clears its memory and looks elsewhere.
While much more complex event correlation is possible--such as using Boolean logic, probabilistic analysis, or other more complex analytical methods--there is a more immediate area where event correlation can be improved: in the ability to correlate events collected from disparate sources, or even from disparate networks.
This is because, again, threat detection in the legacy SIEM is limited due to the underlying scale and scope limitations of the SIEM's data handling architecture. Without the underlying ability to look at all information systems--consisting of much greater volumes of events, from more sources, over more time--only a small subset of threats can be detected. The systems therefore are myopic, and can often cause more harm than good through the false promise of a "security analyst in a box" and a related false sense of security.
As SIEM evolves, correlation is required across a broader array of sources, over longer periods of time, in order to detect more complex patterns. For example: when seen together, a network flow anomaly, a SQL injection attack, and a database policy violation might indicate a successful breach of a database. However, the data available within each event is decidedly different, requiring a flexible data management system to correlate each together.
The limiting factor, again, is the core data management engine: a better answer to data collection, storage, and retrieval is required in order to allow SIEM to evolve to the next level.
Legacy SIEM solutions had limited visibility into protocols and applications-confined to what little information could be gleaned from application and server logs. Looking deeper into application activity would simply add too much strain to the already overtaxed management systems, making 'content awareness' (full application awareness based on deep packet inspection) impossible to these legacy systems.
Once the performance limitations are overcome, the SIEM is able to handle the extreme demands of content awareness. With deeper monitoring into real application and protocol use, threat detection capabilities of SIEM evolve even further, being able to detect the most sophisticated attacks, insider theft, fraudulent activity, and data leakage.
In legacy systems, reporting consists of both pre-defined and customizable report templates . Reports are run against all stored information as well as any identified threats. These reports are mapped to the requirements of relevant regulatory compliance standards, such as NERC, HIPAA, PCI, and Sarbanes-Oxley (SOX).
As SIEM evolves, the requirements of a reporting system evolve as well. Leveraging the real-time nature of the next-generation SIEM, reports become a dynamic process, where real-time dashboards provide a minute-by-minute assessment of what scheduled reports will indicate. This marriage of ongoing security operations and scheduled compliance reporting removes the possibilities of "surprises" during an audit: eliminating the added costs of secondary compliance audits, or even fines.
Notifications are a base function of SIEM, and are typically the result of an in-memory analysis at the time of collection. Simply, if a certain condition occurs, send an alert to an administrator so that immediate action can be taken. This could be the detection of a specific attack, the result of a correlation rule, or upon achieving a threshold.
In the new generation of SIEM, more events are being generated, correlated and analyzed, and as such the mechanisms used for notification needs to allow for additional parameters-including the ability to define thresholds on more sophisticated calculations, such as baselines and deviations. The next generation of SIEM must be able to produce this type of contextual analysis to support more intelligent notifications.
Legacy SIEMs perform "threat investigation" in a purely historical context, by allowing you to investigate the details of security incidents that have already occurred, in the past. As performance increases, the SIEM is able to evolve into a more active role. Users are now able to use the SIEM to quickly identify problems, diagnose them, and identify solutions to support real-time security operations. This can only be done if the SIEM is highly responsive to user input. Technically, high responsiveness boils down to how fast the SIEM can query data from its data manager while the data manager continues to support the other SIEM requirements.
In order to be an effective mission critical decision support system, a SIEM must provide a rich and flexible set of analysis capabilities. Users need to be able to start with a high- level aggregated view with analytical attributes, quickly drill down into an interesting area, and continue this process all the way down to the fine details. Anywhere along the way, users need to be able to quickly cross-correlate what they are looking at with other data views.
Additionally, users need to be able to quickly see how the data they are looking at compares to previous time periods, sometimes called "time correlated analytics". For example, if a user is looking at data between noon and 1 pm on a Monday, it is essential to be able to compare this data to say the average of the equivalent data from the five previous Mondays between noon and 1 pm, the previous correlated time periods. Doing so allows a user to determine whether or not the data they are looking at is "normal" or "abnormal". Finally, increasing the signal-to-noise-ratio of viewed data by correlating similar incoming data into a single compressed dataum is key to the effectiveness of user activities.
Developed specifically for large-scale collection and real-time analysis of data The new requirements of SIEM-Data Collection, Content-Awareness, Cross-Source Correlation, Real-Time Analytics, Long Term Storage and Analysis, and Real-Time Reporting- will quickly overwhelm any legacy SIEM that uses a standard business-oriented SQL RDBMSs for its data manager.
NitroEDB is able to support all of these requirements as the result of decades of R&D and experience in database technology, which provides a distinct-and very important-performance advantage over other database management systems and RDBMS. How? Because unlike other RDBMS systems, NitroEDB was designed for simultaneous event collection, analysis and reporting, at rates that far exceed the limitations of commercial RDBMS and even other custom database and flat-file systems used in the industry. NitroSecurity invested heavily in the research and development of NitroEDB, specifically to achieve these goals. The result is a highly optimized data management architecture, which uses patented techniques to improve performance and scalability in a variety of ways.
The NitroICE engine performs deep packet inspection, and fully decodes layer-7 information, providing analysis of how applications and protocols are used on the network. This allows for the detection of protocol anomalies, as well as for the monitoring of application contents, for purposes of fraud detection and data leakage prevention. NitroICE allows detection rules to be triggered on user, application, client & host names; IP addresses and port numbers; email addresses, subject line; website url's; filenames, types & size; protocols, date-time, printer jobs; and even document contents (e.g. PII, PHI, etc). This allows NitroICE to detect:
NitroView represents the evolution of Security Information and Event Management (SIEM) into a fully context-aware, real-time security management platform. NitroView-through the use of the NitroEDB, data management engine-is able to collect, index, correlate, and store more information, from more sources, for longer periods of time. This includes the ability to collect application content, application session detail, database transactions, and network flows in addition to logs, extending the capability of NitroView far beyond that of a legacy SIEM.
In addition, NitroView-again as a result of the NitroEDB engine-is able to retrieve stored data in real-time, providing immediate access to all information for rapid-response investigations. This makes NitroView unique: unlike legacy SIEMs, it is no longer bound to the role of a log collection and reporting tool. Instead it can be used as an integral part of ongoing, daily security operations: excelling at threat identification, investigation, mitigation, and remediation.
In addition to the expected features of a SIEM (see "An Overview of SIEM"), NitroView offers several unique features that are only possible because of the performance and scalability provided by the patented NitroEDB data management engine. These unique features include:
As threats become more complex, and as the consequences of a breach grow more severe, Security Information and Event Management Systems (SIEMs) need to evolve, becoming operational tools that support minute-by-minute decision making. Legacy SIEM solutions, crippled by the inherent limitations of SQL and flat-file data storage techniques, must first overcome a fundamental performance barrier in order to provide the real-time services that are required. Once these issues of performance and scale are overcome-as it has been with NitroView Enterprise Security Manager and the NitroEDB data management engine- the SIEM can evolve to the next level, where more information is being managed and analyzed in new and more sophisticated ways. Security information can be analyzed in more depth-all the way to the content of an application or protocol. Correlation can become broader, allowing threat detection to consider the context of users, privileges, policies, assets and applications. Incident response, of course, becomes more rapid-a direct result of the new performance requirements of the next generation SIEM.
