It began as a trickle and eventually become a veritable flood of disasters to strike Australia's northeast in early 2011. The devastating Queensland floods were followed in quick order by Cyclone Yasi, putting urban and regional councils under enormous pressure to support emergency and crisis management.

One of the regions to be hit by rapidly rising floodwaters was Toowoomba, where images of a destructive inland Tsunami carrying cars in its chaotic grip flashed across the globe courtesy of YouTube.

The huge influx of water into the valley’s river systems resulted in a tragic loss of life and massive damage to council infrastructure such as roads, bridges and sewers.

The immediate impact for Toowoomba Regional Council’s IT infrastructure was not dire, as the council’s twin data centres were located at sufficient height to remain out of danger.

The main challenge arose during the tumultuous period after the floods struck without warning, as the council rushed to put together a call centre to handle interactions with a beleaguered public. Meanwhile the council web site and internal communications were hampered by an Optus Internet outage that struck during a peak period.

Paul Fendley, Branch Manager of Information Management at Toowoomba Regional Council, watched as the floodwaters rose.

Toowoomba Regional Council uses Open Text eDocs for document and records management, with a secondary data centre located across town which duplicates all systems and data. Neither were threatened.

While Council records and data were safe, an outage for the Optus link meant many staff on Blackberries could not receive email.

“The Toowoomba flood was very severe and without warning but mostly in a fairly narrow corridor around our creeks. It was a different type of event to Lockyer Valley and then the slow flood with a few days warning in Ipswich and Brisbane,” said Fendley.

As he struggled to email staff who were no longer responding, Fendley pushed on to establish a disaster coordination centre as quickly as possible. Ironically, plans were in train to construct a new customer contact centre in Toowoomba which will house a disaster coordination centre (DCC) , and is due to come online in March.

However without that in place, staff had to make do with an empty training room with phone and network connections that was prepared to house external emergency services.

“Calls to our Disaster and SES phone numbers were handled by a temporary 8-seat call centre we set up from scratch. We seconded dozens of staff from across the organisation to man it 24 hours per day for first five days then gradually reducing over the next fortnight.

“Fortunately we are able to utilise some new technology we had recently implemented with a view to supporting a disaster scenario.”

As council staff dealt with the torrent of calls from concerned citizens, they were able to exploit a recent effort to provide integration between Toowoomba’s customer system and GIS, that meant staff were able use the same customer and property system they were familiar with.

“Data came up in realtime on a large screen in our GIS application with cartography indicating the type of request and the status of that request in terms of being actioned,” said Fendley.

“From an ICT perspective many things went really well. From a GIS point of view we were able to spatially represent most of our core organisational information. We were also able to fly some oblique aerial photography in the first days after the event and make that available to staff across our very large council region to undertake some cursory investigation into the extent of damage.”

“Things that didn’t go so well were the very small and the very large, the things you don’t tend to think about. For instance, not having enough 12V mobile phone chargers around for people who were working 24 hours a day and perhaps had left their charger at home. So we had to quickly procure them.

“We couldn’t locate an old school fax machine when our external Internet access went down. The Optus trunk down to Brisbane went down for about four hours so we lost all Internet. This meant we lost our disaster management Web site that we were communicating to the public, and all ingoing and outbound email, which meant we lost ability to receive alerts from the bureau of meteorology and Main Roads, hence the need for a fax machine. While our internal email was functioning fine, almost 100 staff on Blackberrys weren't able to receive messages, because when your external access to the RIM service goes you lose them.

“It teaches you how dependent you are on Internet access and makes you look at steps you can take to mitigate it. If you do lose the Internet what impact will it have on you and what’s the contingency. Within a few hours we’d been able to get our disaster management web site back up through a hosting agency, but email and blackberry email was a real pain.”

Dean Wright, IT Services Manager, Charters Towers City Council, looked at the path of oncoming Cyclone Yasi on February 2 and made ready for what was flagged as potentially the worst to ever hit the region.

“Just after midnight on the Wednesday night we shut our servers down and left them off,” said Wright.
As it turned out, there was not a lot of damage, apart from that caused by uprooted trees being flung about by the wind.

“The power did go off at 5am and we got it back on Wednesday afternoon about 6.00pm and restarted servers and everything is back to normal,” said Wright.

All of Charters Towers’ local government systems are backed up nightly and tapes are taken off site.
“We are in the process of having a SAN connected by fibre optic cable to another council building to replicate our servers.

“The tape magazines will be moved to new site and we will run our backups to tape across the fibre connection.”

The Dataworks optical filing system is used and any important documents are scanned in and housed in there. All mail is scanned as it arrives at council and hard copy records were safely stored.

Hinchinbrook Shire Council was another to decide to shut down all of IT services on Wednesday afternoon before Yasi was due.

A new $800,000 server room with 12” thick walls meant Colin Valinoti, Information Technology Manager slept easy in his bed.

“At 7.00am the Thursday morning after the cyclone the CEO called me and within three hours we all our sites with power up and running ready for the call centre to receive calls from the general public,” said Valinoti.

“We backup every night, and all of our remote users use Citrix. We have off-site storage and our disaster recovery (DR) site replicates through the cluster and is located 3km away with fibre connection.

“We are looking to go outside our power grid with a DR and backup site located 100km away in the future. With the virtualisation on our server farm it makes it very easy to implement that type of scenario.
“Hinchinbrook Shire Council still has a lot of physical records situated in a dedicated records room at a disaster recovery site, while all new documents are scanned into the RecFind EDRMS platform from Knowledge One, which is currently being upgraded to RecFind 6.

“We scan everything in and it’s all kept electronically. RecFind has plug in tools for Outlook Word and Excel. When an officer receives business correspondence via email they can “button” it into the relevant folders in RecFind,” said Valinoti

Unlike other areas like Tully and Cardwell and Port Hinchinbrook, Ingham did not suffer such extreme damage, although some roofs did get removed and trees took down power lines, while water was down for a while but came back very quickly.

“A lot of people look to the council in times of flood and emergency,” said Valinoti.

“There will be a debrief with CEO and other management, they were pleased with IT delivery, we are used to flooding, as we had a similar experience in 200. We have protocols and procedures in place and we can setup call centre in less than two hours, as we now have dedicated phones and headsets.”

At Western Downs Regional Council, the damage to infrastructure will take years to rectify, with 7023kms out of our total road network of 9000kms flood affected and losses to the agricultural sector in the vicinity of $400 million.

During the floods there were temporary, but lengthy, outages to essential services such as water treatment plants and sewerage systems.

Western Downs Regional Council has real-time replication of data between its main site in Dalby and the office in Chinchilla for disaster and redundancy purposes. All data is kept on Netapp SANs (one in Dalby and one in Chinchilla). Intersite connectivity is maintained by a council-owned high bandwidth, low latency microwave radio link between all of Council's major sites.

Information Technology and Communications Manager, Peter Greet said while many houses and businesses in Dalby were flood affected, council data and operations remained out of harm’s way.

“Our website was a critical communication method with media alerts posted several times a day during the event. However, this medium only reaches a certain percentage of the community and obviously is ineffective during power outages. Our Web site is hosted in Brisbane and consequently went offline for a number of hours when the CBD was shutdown,” said Greet.

“3G services were interrupted but this is thought to have been from network congestion.”