Archive to survive
Archive to survive
Industry expert warnings since the dawn of email are now starting to have very tangible and visible outcomes. By Keith Power.
Email data management and retention has become a high-profile issue in recent times, with companies fined for failing to maintain complete email archives, legal actions being settled because it's cheaper than complying with discovery requests for old emails, and court cases being lost because of missing email records or legally inadequate email archiving processes.
A 2003 Osterman Research survey, Current Email and Archiving Practices in the Enterprise, sponsored by Legato Systems, found that nearly one quarter of organisations have been involved in some sort of dispute with a customer or supplier over an email-based issue. But according to EMC, while most firms realise that they need to do something, their exact responsibilities are hazy and the right course of action is unclear: "Email storage and archival polices are controversial and not well-understood," META Group vice president John Brand concurs. "People didn't see the problem coming, and they don't like the thought of the problem being there."
However, not only has the volume of email increased substantially in the past two years, but according to Brand, the percentage of important emails is also rising rapidly and the aggressive purging of email is no longer appropriate. Furthermore, the importance of an email is not always known or its relevance recognised at a particular point in time.
The Osterman survey found that reasonably consistent with the growth in email volume there has been a substantial increase in email storage requirements. Yet while the cost of storage has fallen dramatically in recent years, the total lifecycle cost of managing storage is not an inconsequential part of the overall cost of managing a message system, the paper states.
Secondly, in order to maintain email system performance, it is important to keep online storage to manageable levels. Consequently, a growing proportion of email content must be moved offline to some storage medium in order to avoid being lost and according to the Osterman report, there are three principal reasons to archive emails:
1. Legal requirements. Many organisations, such as those in financial services, are required by statute or other regulations to maintain an archive of all records-based information. This often includes email, and even in the absence of industry-specific regulations related to email, all organisations are required to comply with records retention requirements.
2. Email content represents corporate knowledge. Much of the value that information workers contribute to an organisation is kept in the messaging system. Not to archive this data and make it available is to lose much of the value that such employees generate on a daily basis.
3. Faster restoration and protection from data loss. Restoring messaging system content from an archive is easier and faster than from a tape backup. In addition, most organisations that use tape backups do not preserve the data on them on a long-term basis. So not only does the information soon become lost, they expose themselves to severe penalties for non-compliance with email and records retention requirements.
In fact, Mark Heers, EMC's product marketing manager, Australia/New Zealand, considers the essential difference between archive and backup to be that the former is a primary copy of data, randomly written and read and usually governed with requirements for retention, authenticity and integrity; and backup is an insurance or second copy, rarely used and sequentially written and read: "Trying to obtain specific emails, such as all the emails that Mary sent on Project Delta between March 2001 and September 2002, from backup is extremely difficult. Email backup is designed to restore a database of emails should the original database become corrupted or lost. It is not designed to pick out individual emails," Heers explains.
However, the Osterman research found that most organisations do not have adequate archival practices for their email systems. Although nearly all backup their email systems regularly, 56 per cent of respondents do not keep all of the data on a long-term basis, while 13 per cent keep only critical data long-term. Most email users also tend to focus on their primary job responsibilities rather than the larger needs of the organisation and so may inadvertently discard email information that should actually be maintained for regulatory or other reasons.
Brand claims that the warnings organisations such as META have been issuing since the dawn of email are now starting to have very tangible and visible outcomes. On the legal front, he points specifically to the Financial Transactions Act, the Corporations Act, the Income Tax Assessment Act, the Tax Administration Act and the Criminal Proceeds Confiscation Act in Australia, which he says are being very much policed and enforced, especially in relation to email. In fact, he says the Australian Prudential Regulatory Authority (APRA) has quadrupled its audit teams since February 2003 and is extremely "gung ho" as a result of the HIH collapse.
"The regulatory influence we're seeing is punitive rather than prescriptive - they will tell you what the punishment is but they won't necessarily tell you exactly what you have to do. The risks and impacts are definitely real, though.
"One of the differences with email is that the record is established as soon as it is created. It doesn't matter where it exists. So you may as well find better ways of managing email than just watching the inbox size grow as people create their own personal filing cabinets. But at the moment we've taken a very informal communication system, we're using it like a formal work management system, and yet we don't have the same rigor and principle around managing email as we do other applications.
"We've seen organisations have discovery orders placed on them and they are not able to recover the [relevant] material. They know those emails exists in backup tapes that they have from three years ago, but the effort involved in retrieving them is proving to take too long," Brand says.
At the same time, he concedes that the infrastructure to archive emails is expensive to purchase and maintain. The cost of non-compliance or inadequate compliance, though, can easily exceed the cost of infrastructure, he maintains: "The cost implications [of email archiving] are not the acquisition costs, but the costs of litigation and the penalties involved. In cases where these solutions would have had some value, they appear extremely cheap by comparison. The average litigation action we've seen from our research is around $30-50 million," Brand says.