Key considerations for successful Log Management projects, how to build the business case, and pitfalls to avoid
Most companies use I.T. systems logs to troubleshoot problems in one system or another. NitroSecurity discusses the benefits of a strategic approach to log management, using a low-burden platform that automates log collection and consolidation across the enterprise. Significant incremental value is realized when alerting and reporting build on the automated log collection - ultimately delivering strategic value - rather than tactical.
This whitepaper describes best practices in log management, identifying methods in which internal champions can correctly build the business case for a log management project as well as identifying common pitfalls in a project that can be avoided.
The whitepaper identifies which logs should be captured and reviewed, the lifecycle of log data, common values realized, and common pitfalls to avoid.
With more then 600 customer deployments across a range of vertical markets, NitroSecurity is a leader in log consolidation and log management
Log management is often viewed as a tactical effort - the patch that addresses a set of compliance or security requirements. By taking a broader perspective, the same log management practice is the foundation of compliance while also providing significant value across a range of compliance, security, and systems management efforts. The I.T. organization has better information to handle day-to-day responsibilities and is better prepared when the unexpected occurs.
I.T. organizations should view log management as a key element of I.T. governance that meets needs of a variety of stakeholders and addresses a broad range of problems. The successful business case for log management should include multiple stakeholders, discussion of risks that can be mitigated, and indication of business process improvements that can be advanced.
NitroSecurity's NitroView ELM is ideally suited for companies seeking to take the next step in advancing log management up the value curve. LogCaster is very easy to implement - a baseline implementation takes less then two hours. One of the most scalable solutions to log consolidation, LogCaster can incrementally grow to address the largest and most complicated I.T. organizations. In addition, LogCaster integrates with NitroView Enterprise Security Manager, for enterprises where highly advanced forensics, correlation, and other SIEM functions are required.
A strategic approach to log management - one that addresses a variety of stakeholders and improves a variety of processes - is more likely to have allocated funds. Based on NitroSecurity's experiences with customers, best practice recommendations are proposed for internal champions seeking log management funding as well as some traps to avoid.
Log management is frequently seen as a key enabler of regulatory compliance, corporate security policy compliance, and audit compliance. By documenting what happens on I.T. systems, the log is the ultimate record to indicate what happened and when it happened.
Other benefits are often overlooked, even though they may be of equal or greater value depending on the circumstances. The first three hidden benefits can apply to any company seeking to improve I.T. quality and repeatability. Service provider accreditation is more narrowly applicable; only companies that seek an applicable third-party accreditation will realize this benefit. Finally, SLAs or contract obligations can apply to either vendors or customers. Since there are often penalties for performance below SLA thresholds, customers will want access to a log to identify when SLAs weren't met while the vendor will want a log to document periods when performance was within guidelines.
When viewed strategically, log management can address each identified organizational benefit. If a tactical solution is employed to address one benefit - security compliance, for instance - it might be more difficult to add events that address another (perhaps SLA compliance), but over time a log management platform should be able to address any or all relevant business values.
Regardless of the organizational reasons to adopt log management, I.T. users consistently realize productivity benefits from the ability to visualize and prioritize work that needs to be done. A quick read of the SANS Institute's Top 5 log reports illustrates some of the values that I.T. users realize (from the security perspective that is the focus of the SANS Institute). Identification of key threat vectors - both successful and failed - helps the I.T. team close down the attacker and ensure that loopholes are addressed.
Compliance obligations are a common catalyst for log management projects since they create a documentation burden. In order to demonstrate that specified procedures have been followed, reports are generated from system logs. For instance, the Sarbanes-Oxley Act makes repeated reference to "controls" - this business requirement essentially requires (among other things) an auditable process that documents I.T. systems access attempts. By consolidating system logs from key servers, it's relatively easy to generate reports on password changes, rejected log-on attempts, or other activities that are considered part of the compliance landscape.
| SOX | Sarbanes-Oxley Act of 2002 |
| PCI | The Processing Card Industry Standard |
| HIPAA | Health Care Portability and Accountability Act |
| FISMA | Federal Information Security Management Act |
| FIPS 200 | Federal Information Processing Standard: Minimum Security Requirements For Federal Information Systems and NIST 800-53 |
| GLB | Gramm-Leach-Bliley Act of 1999 |
| SB1386 | California legislation, Senate Bill 1386 |
| ISO 17799 and 27001 | International Organization for Standardization security standards |
| SAS70 | American Institute of Certified Public Accountants (AICPA) Statement on Auditing Standards No. 70, Service Organizations |
When scoping out a log consolidation project, it is important to consider what systems should be included - some critical systems are not as apparent as others. Implementation of the log management infrastructure can be done over time, but having a reasonably accurate sense of the complexity of the project and the scope will ensure that the selected log management solution will be adequate.
Obvious candidates for logging are key servers - Windows, UNIX, or Linux; transaction servers, database servers, web servers or e-mail servers. Network components like routers and firewalls are also high on the initial list. Most organizations can generate an inventory of these key elements with relative ease and prioritize them for inclusion in a log consolidation project.
Strategic log management should include key systems that are frequently overlooked by mistake. These systems frequently represent "back doors" that hackers attempt to use as a backdoor to their ultimate goals. In other cases, inclusion in log management is advised from a value perspective - they contain significant events that are helpful when you capture them.
Reviewing the logs of tape arrays, backup storage servers, and storage software are important from a systems management perspective and from a security/compliance perspective. Systematic analysis of logs can confirm that the backup process completed without error (which is not always a safe assumption to make). Additionally, logs can reveal penetration attempts that would indicate a hacker is targeting your backups.
Increased reliance on security infrastructure makes it imperative to prevent hackers from penetrating the security systems. Elements such as identity management servers, LDAP servers, two-factor authentication servers, proxy servers, and the like should be closely monitored to ensure they aren't compromised as the early stage of an attack.
Virus protection is an interesting example of valuable application-level logging. Notoriously unreliable, virus scanning applications can be monitored to confirm the security status of the underlying host. Log management can ensure that the virus database is up to date. Early warning of a new virus can even help trigger a response to malware that is too new for effective virus updates.
Complicated network topologies may introduce system access that bypasses some defenses. Log review of Citrix implementations, modem banks, terminal access servers, or the like can provide another layer of alerting.
Controlling data access is perhaps the most critical element of an I.T. security and compliance framework. It may well be the hardest to do, as well. Each company wants to safeguard its own financial data from intrusion or defacement, but many firms will also need to protect data from customers or partners - data that may include credit card information, personally identifiable information (PII) such as Social Security Numbers, and personal health information (PHI). Data of this sort is the target of most attempted penetrations and is the underlying reason for most compliance regulations. Additionally, a company's reputation can be significant damaged if it fails to safeguard this sort of key data.
It's only now that understanding of the database layer can be included in a log consolidation system. In the past, reports on database health were either limited to those within the database management console itself or were limited to reports on the server that the database used, rather then the database itself.
Historically, logs addressed the health of the server on which the database ran - including metrics like CPU utilization, disk activity, and the like. NitroSecurity's NitroView DBM (DBM) customers receive much more relevant logs - viewing information on user access, data viewed by different users, and suspicious data access patterns that fit common attack profiles. Similarly, NitroGuard IPS (IPS) customers receive relevant data about inbound security threats and network flows (who is talking to who).
Combining logs of data access with system logs provides great synergies. For one, it's possible to correlate system access with database log-in and send an alarm when a user logs onto the database with a user ID that doesn't match the Windows user name. In environments with pooled connections, the system-level information can be used to derive the named user for specific queries. Insider theft of confidential data can be identified when a DBA backs up a database outside of the scheduled routine and then correlated to a file copy process onto removable media, or an FTP outside of the firewall.
NitroSecurity is the only log management vendor with a solution to address data-layer logging. By complementing NitroView ELM with NitroView DBM - NitroSecurity's solution for data assurance - a complete logging solution consolidates data access patterns with systems layer logs.
Log consolidation serves the needs of a wide variety of users within the enterprise - gathering support (and budget) from all stakeholders increases the value of the project and the likelihood of success.
Sysadmins are interested in the health of the systems they maintain, and are accustomed to referring to sys logs. A consolidation project will likely help them use logs more efficiently and more regularly.
The functional manager is the boss of the sysadmins; functional managers are much more empowered to do their job by the implementation of a log consolidation project. They gain visibility into the work of each sysadmin, and are able to visualize patterns or trends over the entire enterprise. Each sysadmin may see scattered incidents, but in the context of the entire enterprise a manager can connect the dots to spot a trend that should be addressed.
User administration deals with provisioning accounts - creating, deleting, and modifying access privileges. Log management empowers the provisioning team by monitoring access policies and password policies as well as provides an understanding of problems the provisioning team has to fix - problems such as user lock-outs, incorrect access attempts, and the like.
The security team gains a wealth of data about access attempts and access methods. By reviewing access logs, the security team can see who is doing what - users as well as in provisioning. Users can be monitored to ensure that a user logged in to the network as one person doesn't access an application as another, that users aren't probing for security holes, etc. Security can also oversee the user admin group to ensure that only the proper number and types of accounts are created, that security access policies are followed in account creation, and so forth.
Computer Incident Response Teams are the first responders to system intrusion - in-depth logs provide them the record of what's happened that enables an efficient response. CIRT can be alerted of a potential intrusion from many different systems (firewall, identity server, etc.), but the alert is only sent after enough suspicious activity has met a threshold. In most cases, the alarm goes out after the potential intruder has repeatedly probed the infrastructure without hitting a trigger hard enough to cause an alert. Logs provide CIRT with that complete history, which can help confirm whether the event is malicious, help identify the threat vector and close off access, potentially identify the intruder, and ultimately quantify exposure which dictates the business response.
Audit has become a significant driver of log management, with virtually all technology audits requiring logs to substantiate reports. Corporate audit teams (both internal and external) can be significant supporters of log management projects when engaged correctly. Since audit teams are focused on all aspects of compliance, I.T. may need to engage in a dialogue so that the two sides understand what they need to document. Identifying what business processes need to be documented and in what level can lead to the needed realization of what computer systems need to be logged, for what period and in what level of detail.
Law enforcement is increasingly using digital evidence and logs can be critical evidence, especially of digital trespass. Logs can help companies seek redress for attacks - both in cases of disgruntled employees and from outsiders. In other situations, logs can help law enforcement clear a company from involvement in a problem, such as when a threat was created by an employee or customer, or when a third party conducts an attack from its servers (spamming or hacking).
Customers may require data that is logged - information on access to data (in a healthcare or financial service environment, for instance). Privacy laws are increasingly making vendors responsible to customers for securing access to data; those vendors need to be able to substantiate claims that data has been treated correctly and be able to document who has accessed that data and for what purpose.
Additional stakeholders may be relevant in different organizations, such as a Project Management Office, vendor management, HR, the help desk, or the like. By taking a holistic approach to the problem, it is possible to identify various elements of the organization that need to understand what has occurred in I.T. systems or that need to document what's occurred.
A short list of focus points will help ensure that a log management project is accepted by the organization, that resources are approved, and that value is realized down the road.
Some organizations will deploy a strategic log management project based on a single purpose (audit, for instance), but many more realize ongoing value when log management addresses a variety of stakeholders and values. By building consensus across the different needs across multiple stakeholders and spanning the areas of compliance, security, and systems management.
Stakeholders need tangible evidence of the value they will receive. Rather then identifying general reasons - claims like "Sarbanes-Oxley means we need to consolidate logs" - it's important to engage in sufficient dialogue to detail specific benefits. It's much more powerful to indicate that "we can reduce our Sarbanes-Oxley compliance costs by automating processes that now require three FTEs in technology and one FTE in audit."
Like any I.T. project, effective management dramatically increases the likelihood of success. By clearly identifying the scope, reasonably projecting long-term requirements, and updating the plan, it's more likely that the project be green-lighted, that it achieves operational status, and that it meet objectives down the road.
Frequently, when our customers have experienced delays in moving ahead with a project they later explain to us how political mistakes in their approval process led to misunderstandings and delays. We've compiled some of the more common missteps.
Audit requirements are often the primary driver for log management projects, but it's rare that the I.T. project owner is deeply experienced in the regulatory backdrop. The team leader doesn't need to go to law school in order to achieve success in log management; instead, focus on:
Eliminate vague regulatory references. Translate vague legalese into the technology requirements. For instance, Sarbanes-Oxley generally requires "adequate internal controls" - I.T. managers should explain how log management provides the framework that reports on specific users and passwords that accessed financial systems. Regulatory requirements have been cited for a tremendous number of projects over the past several years, leading to fatigue from purchasers and an increased amount of skepticism - providing specific examples can make the business case more real and increase the likelihood of receiving budget approval.
Strengthen alignment with Audit. When log management is primarily compliance solution, it's critical to work with the business owners of the problem - whether that owner is considered the audit team, a group with the legal department, etc. Don't make the technologists' mistake of working a solution without a clear understanding of the problem.
Consider evidentiary requirements before setting foot in a legal setting. No business person desires to get into court, but I.T. logs can be a critical piece of evidence if you are forced into a legal situation. With litigation so prevalent, having I.T. logs to prove your claims of what occurred can be invaluable. Make sure that logs are preserved in such a manner that they can used in court and aren't excluded on a technicality.
Reaching agreement is harder the more people are involved - endangering a project like log management. The security manager, CIRT manager, audit manager, and functional managers each have their own priorities and their own agendas - it's critical to understand the resource needs and capabilities of each one and approach them constructively. If the project is positioned in such a way that it answers a need the manager feels and seems to be more resource-efficient then a different approach the manager is considering, it's more likely that the manger will buy in and be willing to contribute resources.
The business case for log consolidation should consider long-term requirements for implementation, management, and support. Like any project, a budget proposal should include the implementation cost (in equipment, software, and personnel) as well as ongoing requirements for dedicated personnel, data archiving, and time from other personnel. This should reduce the pushback from the finance team since the proposal is more realistic then a simplistic implementation-only cost estimate.
Log consolidation should be viewed as a new application - consider the requirements for CPU, networking traffic, and storage. Plan and architect the solution to live within existing requirements, constraints, and the unique aspects of the environment. This reduces the likelihood of hitting a snag when a plan is developed that's incompatible with the environment.
When reviewing solution options for log consolidation, there are several key options to consider. You'll want to ensure that you can quickly start realizing benefits, address all immediate needs, enable room for growth, and minimize the risk of project failure as much as possible.
A log management program will require more attention to people than to software. Software implementations should be measured in days, rather than weeks or months, for even the largest infrastructures. Getting policies determined, hammering out responsibilities, and agreeing to what standards to measure are the difficult tasks. Policies can evolve over time, and so long as some critical determinations are made up front it's possible to leave more tactical decisions for a later date.
NitroSecurity's LogCaster can be installed in two hours as a free trial - starting the process of realizing value. Demonstrating value from an in-production system can be a catalyst that demonstrates value quickly and accelerates the bureaucratic process of buy-in and policy determination.
View data retention as a different problem from reporting. While reporting and alerting are the immediate values conferred from log management, retaining the data is a different problem with its own needs. A system may store data that's more summarized then the detail that's needed for regulatory or business needs.
Data retention creates a storage problem - consider how long data needs to be retained, with what service level, and how compression can increase efficiency of the storage process. Storage costs are constantly deceasing, but log consolidation generates such significant volumes of data that it's a nontrivial problem to plan a long-term tiered storage process.
NitroSecurity's LogCaster helps reduce the data retention problem by compacting archival logs by as much as 20x when compared to the uncompacted original. All of the data is retained in its untouched state and can be quickly recovered for review.
Consider requirements for alerting, on-screen visualization, and generated reports. Different stakeholders will need access to information in different ways - to meet the requirements of the different stakeholders it will be important to meet their access needs. Security may have a focus on real-time alerts, CIRT may want detailed logs, and so forth.
NitroSecurity's LogCaster maintains a reporting database of significant events that is separate from the archived (untouched) log. The database of important events drives the alerts and reports - with significant flexibility enabling customization of how the information is presented.
Companies can realize strategic value from system logs, with significant benefits in compliance and security. Planning and consensus building are critical elements for a successful business case for strategic log management - applying a structure to the ad hoc review of system logs that delivers only tactical benefits. A scalable, low-burden framework can consolidate a diverse set of logs, monitor events for material items, alert I.T. when critical items need immediate attention, and visualize historical data. Compliance reports, an increasing burden on I.T. groups, can be automated to reduce the ongoing drain on personnel.
|
Log in
