Are you struggling to manage your organisation's digital records? Don't feel you are on your own, as Kate Cumming writes, it is widely acknowledged that records are growing out of control.

There is too much data, its growing too fast and in too many different formats. Data proliferates wherever it lives – networks, servers, backup tapes, digital cameras, memory sticks, desktops, databases, web sites – you name it, a multitude of digital records lurk there. In addition, digital records are incredibly vulnerable - they are built out of fragile relationships between constantly evolving software and hardware and there is no easy or standard answer for their effective long term management.

So how do you take control? How do you turn your digital liabilities into digital assets that provide both security and tangible benefit to your organisation? The first step is to acknowledge that you have a problem. It is important to realise that managing your digital records so that they can be used and secured for as long as your organisation needs them it is a really challenging proposition.

Digital records management is hard. Don’t be complacent. And don’t be tempted to put it aside as a problem that can be dealt with tomorrow. You really need to put in place strategies to manage your digital records effectively now.

The next critical step is to minimise the extent of the problem you are facing. You can do this in a number of ways. One really powerful strategy is to limit the number of file formats you are dealing with.

File formats are the mechanisms by which different types of digital records are encoded and stored. It is estimated that average Australian office environments use between 20 and 50 different file formats and the majority of these are proprietary, not open formats. This is simply too many, adding up to 20 to 50 problems each requiring a unique solution. The solutions are not static either, but will need to be amended through time to take account of continual software and hardware evolution. This is a scenario that is just too hard and costly to sustain. It makes sense to restrict the number of file formats your organisation uses, and focus on the use of open formats wherever possible. Open formats are formats that have freely available specifications. They are not tied to a vendor, they are less subject to change than proprietary formats and they work across a variety of operating systems.

An example of an open format in widespread daily use is HTML. On the Web, you can still access HTML pages created 10-15 years ago at the advent of the Internet. They may be primitive and a little out of date but, provided their online domain has remained stable, you can still access these pages.

There would be few business records created in proprietary formats that could still be easily accessed and used 15 years after their creation across a variety of different business applications.

This is an example of the stability of open formats and how their use can contribute to greater usability and durability of your digital records. It makes sense to capitalise on this stability and durability. Many organisations are realising this and are restricting their file format choices to just a few key formats that best meet business needs and that are open format wherever possible.

Some large organisations are choosing to migrate the majority of business records from their proprietary creation environments to the open format PDF/A. By doing this they are drastically minimising the digital preservation related risks, costs and heartaches they will have to deal with in the future. These organisations are being proactive and are minimising the digital record-keeping issues they face. Another really important way to minimise the extent of your digital records management problem is to destroy digital records when their authorised retention periods have expired. By keeping on top of your destruction list you will simply minimise the number of problems you have to deal with.

You will also save on storage which, despite the popular myth, is becoming an increasingly significant cost. Even the CIO of Intel recently predicted that her company will double its storage spending by 2012 and said that this is unsustainable. By regularly destroying data you will also be able to respond to discovery orders more quickly, ease network congestion for staff and greatly reduce the costs of your digital preservation. Don’t underestimate the power and long term cost benefits of an effective destruction strategy. Creating your records well in the first place is also really important. Simple steps at record creation can make such a difference to your long term digital preservation needs.

One simple but highly effective strategy is to use templates for records creation wherever you can. Adding a bit of structure to a document by specifying what part of the document is the title, what is column A, what is column B, what font and font size is used in each section – all of this structure and definition combines to give your records a solid and identifiable form.

Solid and identifiable form is a critical factor in facilitating digital preservation strategies such as migration.

Detailed research in the Netherlands has shown that Word documents and email records created with a defined structure were much more successfully migrated and maintained than records without this defined structure. (See these and other very interesting recommendations of the Dutch Digital Testbed Project at www.digitaleduurzaamheid.nl). In defining your templates you can also add a few metadata components that can again facilitate both immediate record use and long term management.

Metadata is really a tool that you should implement as extensively as possible to support your digital record-keeping. It should be the next step in your digital preservation strategy. Metadata is essentially data that helps to manage and increase the usability of your records. Due to the proliferation of digital records, any degree of control that you can exercise will pay significant dividends. All organisations should be using metadata effectively now to control, identify and manage their digital records. However few organisations are using metadata as a targeted and considered tool. It is often used in an ad hoc manner, developed on the fly or implemented out of the box without due consideration of business or management needs.

Used in this way metadata becomes a liability rather than an asset. Used well, however, metadata can be a very powerful tool. One of its key benefits in terms of digital preservation comes in the value it can bring to the management and preservation of record authenticity. In a legal and business sense, records are of little value if you cannot demonstrate that they are what they purport to be. If you can’t account for their management, if you can’t demonstrate the role they played in business process, if you can’t show who had access to them and when, if you can’t provide the business rules that governed their use, many records lose all value.

It is critical to build metadata schema, and possibly a variety of schema for different areas of your business, to provide you with all the data you need to effectively use and rely upon your records. Design your business and records systems so that they can really utilise the power of metadata and automate the management of your business and your records wherever possible.

One key threat to record authenticity and a big digital preservation issue is migration. Considering your migration needs should be the next step in your new digital records management regime.

If your records need to be preserved for periods longer than 5 – 7 years then they will need to be migrated when their hardware and software dependencies are approaching obsolescence. A large proportion of your organisational records will require migration but migration is a highly risky process.

Migration necessarily changes data and migration necessarily moves data. It may also require the measured loss of some data in order to drive the process. All of these decisions need to be fully considered and documented as they can critically affect record authenticity and usability.

It is also important to realise that poorly designed and performed migration can result in the unintentional loss of data.

A recent survey of US news agency archives showed that 82% of the news agencies surveyed suffered unintentional data loss following migration, ranging from minor loss to ‘catastrophic’. (See InterPARES 2, Description Cross-domain Task Force Report in the InterPARES 2 book at www.interpares.org).Thus any migrations need to be designed with due consideration of your business and record-keeping requirements. Poorly designed and performed migrations can have a significant impact on the authenticity and integrity of your records.

In the worst case scenario, poor migrations that result in catastrophic data loss can mean that you are unable to meet retention and/or e-discovery requirements which could result in legal and financial penalties for your organisation.

All of these issues and more are discussed in detail in State Records NSW’s new guideline publication, Managing digital records. Available free via the State Records website at www.records.nsw.gov.au these guidelines provide detailed advice on making digital record-keeping achievable, on implementing record-keeping systems, on developing and applying metadata, on planning for and performing migrations and on managing difficult formats such as emails and digital photographs.

If any of these issues are a concern for your organisation, take a look at these guidelines. And remember, your effective digital records management needs to start now. To leave it any longer is to place both your organisation and your digital records at risk.

Kate Cumming is a Project Officer in the Government Record keeping program of State Records NSW, where her main areas of responsibility concern digital record keeping, principally system design, record migration and metadata. Kate holds a PhD from Monash University.