Log Management (CloudMonk.io)

Log management

Return to Logging topics

Log management (LM) comprises an approach to dealing with large volumes of computer-generated data logging|log messages (also known as audit records, audit trails, event-logs, etc.).

Log management generally covers:Error: File not found: cite web|url=http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf|title=SP 800-92, Guide to Computer Security Log Management|first1=Author: Karen Kent|last1=(NIST)|first2=Author: Murugiah Souppaya|last2=(NIST)|website=csrc.nist.gov

* Log collection
* Centralized log aggregation
* Long-term log storage and Log retention
* Log rotation
* Log analysis (in real-time and in bulk after storage)
* Log search and reporting.

Overview

The primary drivers for log management implementations are concerns about computer security|security,Error: File not found: cite web|url=http://www.prismmicrosys.com/newsletters_august2007.php|title=Leveraging Log Data for Better Security|work=EventTracker SIEM, IT Security, Compliance, Log Management|accessdate=12 August 2015|archive-url=https://web.archive.org/web/20141228182418/http://www.prismmicrosys.com/newsletters_august2007.php|archive-date=28 December 2014|url-status=dead system and network operations (such as System administrator|system or Network administrator|network administration) and regulatory compliance. Logs are generated by nearly every computing device, and can often be directed to different locations both on a local file system or remote system.

Effectively analyzing large volumes of diverse logs can pose many challenges, such as:

* Volume: log data can reach hundreds of gigabytes of data per day for a large organization. Simply collecting, centralizing and storing data at this volume can be challenging.
* Normalization: logs are produced in multiple formats. The process of Normalization (statistics)|normalization is designed to provide a common output for analysis from diverse sources.
* Velocity: The speed at which logs are produced from devices can make collection and aggregation difficult
* Veracity: Log events may not be accurate. This is especially problematic from systems that perform detection, such as Intrusion detection system|intrusion detection systems.

Users and potential users of log management may purchase complete commercial tools or build their own log-management and intelligence tools, assembling the functionality from various open-source model|open-source components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.Error: File not found: cite web|url=http://www.docstoc.com/docs/19680768/Top-5-Log-Mistakes---Second-Edition|title=Top 5 Log Mistakes - Second Edition|work=Docstoc.com|accessdate=12 August 2015

Logging can produce technical information usable for the maintenance of applications or websites. It can serve:

* to define whether a reported bug is actually a bug
* to help analyze, reproduce and solve bugs
* to help test new features in a development stage

Terminology

Suggestions were madeError: File not found: by whom|date=October 2014 to change the definition of logging. This change would keep matters both more pure and more easily maintainable:

* Logging would then be defined as all instantly discardable data on the technical process of an application or website, as it represents and processes data and user input.
* Auditing, then, would involve data that is not immediately discardable. In other words: data that is assembled in the auditing process, is stored persistently, is protected by authorization schemes and is, always, connected to some end-user functional requirement.

Deployment life-cycle

One viewError: File not found: Citation needed|date=October 2007 of assessing the maturity of an organization in terms of the deployment of log-management tools might useError: File not found: Or|date=October 2007 successive levels such as:

# in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
# with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.
# at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the Business|enterprise — especially of those information-assets whose availability organizations regard as vital.
# organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.
# organizations merge the physical-access monitoring and the logical-access monitoring into a single view.

References

* Chris MacKinnon: "LMI In The Enterprise". Processor November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10
* MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org, retrieved 2010-03-03
* NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf, retrieved 2010-03-03

External links

* [http://www.infoworld.com/d/data-explosion/infoworld-review-meeting-the-network-security-and-compliance-challenge-658 InfoWorld review and comparison of commercial Log Management products]