IT departments typically generate all sorts of log files, which exhaustively note all the activity on a device, within an application or on a server--or who enters the building, for that matter. As you can imagine, the amount of data is exploding at many companies. This is very much a global phenomena, as more countries impose more stringent data retention rules. 

Hewlett-Packard, one of many companies that sells systems to handle so-called event data warehousing issues, sees customers dealing with anywhere from 10 GB of data per day to 1 TB of data daily, notes ComputerWorld. 

"The typical enterprise generates more than one terabyte of log data per day and is required by law to archive the information for seven years,"  Dominique Levin, EVP of Marketing and Strategy for LogLogic, tells FierceFinanceIT. "Printed out, these logs would stretch to the moon and back more than 6000 times. In addition, logs come in different "languages" or formats, and they have to be translated before someone can understand what the string of numbers and letters actually means." 

All this can result in some serious problems. In the case of Heartland Payment Systems, site of a massive breach recently, "the logs indicating a security breach were there all along.  However, these telling logs were 'out of control' and were not captured and analyzed." So it may be time to revisit what you do with all your log data. It makes sense to analyze it somehow. LogLogic has launched its LogLogic Compliance Manager, which gets at this issue. We may see other vendors enter the market. 

The long-term goal for many forward thinkers is to use this data for purposes beyond strict compliance. Like the Sarbanes-Oxley-driven data, log data might be utilized for strategic purposes and to develop even more detailed security and compliance measures. But at a minimum, you need to impose order on your log data. Where does your company stand? - Jim