Staying afloat in the data deluge

Torrents of data flow in every day to the Murray-Darling Basin Authority (MDBA), as it tracks the ebb and flow of southeast Australia’s major inland rivers. Brian Catto, CIO of the MDBA, is tackling ever-growing volumes of structured and unstructured data.

The Murray-Darling Basin Authority was established in 2008 to develop a plan for management of the water resources of Australia’s largest inland rivers and the home for more than two million people resident in the Murray  Darling Basin.  The federal government agency faces the standard challenges of implementing a solid records management platform for the digital era, digitising paper based workflows and managing an explosion of data.

Catto is engaged in a push to evolve the MDBA to be the authoritative information service for the Murray-Darling Basin.   This includes development of a modelling environment to process the increasing volumes of data, and investigating the benefits of a SharePoint 2010 implementation to improve collaboration, records keeping and workflows. 

“We use a lot of geospatial data which also presents its own problems, in terms of large storage demands and managing metadata,” said Catto.

Modelling river data is key to the work undertaken by the MDBA.   More data leads to better modelling, but requires greater processing power and greater storage.   This requirement has led to the creation of a high-speed, high-capacity processing environment, called the CoRE, or Computational Resource Environment.

The CoRE was built in-house in 2010, greatly enhancing the ability of the MDBA to analyse the data and consider various water management scenarios.

The CoRE is comprised of high-performance computing blade servers capable of hosting the Authority’s modelling environment to provide better performance and capacity than previously available to the Authority.   The CoRE is robust enough to host the Basin Plan Modelling work and flexible enough to cater to its projected growth over the next four years.

The processing power of the modelling system includes the provisioning of five high-performance blade servers with four CPUs, eight processors and up to 256 GB RAM each, running Windows Server 2008 r2 (64b), to host all components of the CoRE.

Another major project being undertaken by the MDBA ICT team is the investigation of the benefits that SharePoint 2010 might bring to the MDBA.

Upgrading the TRIM EDRMS from v6.2 to v7.1 will enable the implementation of a user friendly SharePoint 2010 interface to TRIM.   This will encourage greater use of the TRIM system as the electronic repository of MDBA records.   A pre-requisite for this upgrade is a recent migration from Windows XP to Windows 7. In addition to Office and TRIM, enterprise apps used at the MDBA include Finance One and Chris21 HR.

“We are investigating a few things with SharePoint,” said Catto.

“The first is to provide a user-friendly front end to TRIM as user feedback suggests that the native interface is not particularly intuitive for non-records managers. We want something that is a bit more logical for end users to encourage better use of TRIM.  To implement this with TRIM 6 required a 3rd party SharePoint integration tool.   However TRIM 7 comes with built-in integration with SharePoint 2010.

“We also want to put a standard platform across our intranet and Internet with a single content management system. At the moment we have a range of different PHP platforms including Drupal, Xend and MySource Matrix.

“The ability to customise a Web site with SharePoint 2010 would be a big improvement.”

“Automating manual workflows with SharePoint is important area where SharePoint may reap rewards for the MDBA.

“Collaboration is another key function of SharePoint 2010, so we are looking at how this could be used in MDBA – allowing multiple people to edit a single instance of a document at the same time can be much more effective than emailing the document to all reviewers and then attempting to reconcile the individual responses”

Open government is also a big challenge, felt especially keenly by the MDBA as the development of the Murray-Darling Basin Plan is attracting keen interest from the community, industry and the media.
“As part of our goal to provide the authoritative information service about the Basin we are increasingly making our data and research available to the public. But once it is available you must maintain the level of availability and it generates further work as people express even more interest in it.

“So far we have released over 1,500 research reports and datasets via the website, and all of it can be taken and massaged in different ways, often in ways that were never thought of before.
“A good example is FlowTracker which draws on the information we release, mashes it up with data from others, and generates a map of how the recent floods moved through the Basin (http://riversmart.net.au/Flowtracker.htm).

“These are the type of challenges that arise from open government.  Again a SharePoint 2010 implementation may help us manage these issues.”

With the development of the CoRE, and potentially the benefits of a SharePoint implementation, the MDBA will be able to meet the challenges of today, and be well positioned to meet the challenges of tomorrow.