Most of us are guilty of “data hoarding”. Without a thought, we save every digital photo, email, document, presentation and spreadsheet, losing track of what we have saved along the way. Across the enterprise, employees are blindly building a bottomless lake of data, and, in many cases, a corporate mantra of “save everything, just in case” is encouraging the behaviour.

Email, instant messages, documents, ZIP files, log files, archived web content, partially developed and then abandoned applications, code snippets … all of this is now termed “dark data”.

Gartner defines dark data as “the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes.” It includes all data objects and types that have yet to be analysed for any business or competitive intelligence or aid in business decision making.

“Increased data growth over the past decade has created an unstructured data nightmare,” says Alan Dayley, research director at Gartner. “It’s not just the cost to store it. Huge volumes of dark data make it harder to find what is useful and may mean we miss business opportunities.”

Gartner predicts that through 2021, more than 80% of organisations will fail to develop a consolidated data security policy across silos, leading to potential noncompliance, security breaches and financial liabilities.

To effectively manage data growth and security, information managers will need to deploy the right tools, and educate employees on how to overcome instinctual data hoarding.

The dark data opportunity

Operational data that is left unanalysed can now be used as an economic opportunity for companies. They can look at using this data to drive new revenues or reduce internal costs.

Some examples of data that is often left dark include server log files that can give clues to website visitor behaviour, customer call detail records that can indicate consumer sentiment and mobile geolocation data that can reveal traffic patterns to aid in business planning.

“No matter which types of dark data your organisation collects, or how it is stored, the key to keeping data out of the dark is to ensure that you have a means of translating it from one form to another and ingesting it easily into whichever analytics platform you use,” says Dayley.

Generating large sums of data that serve nothing is useless knowledge. Whoever unlocks the reams of data and uses it strategically will win.

Dayley’s recommendations for organisations to manage dark data are:

• Start today. This is only going to get worse — don’t wait for that unsavoury catalyst.
• Reach out to all stakeholders and then trim involvement of unnecessary but interested parties.
• Take action — Move the data, secure the data, create accessibility to the data or delete the data, depending on the desired business outcome.  

Upcoming dates and locations for Gartner Symposium/ITxpo are:
October 30-November 2, Gold Coast, Australia