The Australian Bureau of Statistics (ABS) says that the 2016 online census form was subject to “four Denial of Service attacks,” which prompted the ABS to shut down its Census website as a security precaution on Tuesday night.

While the ABS maintains that 2 million forms were successfully submitted and safely stored, thousands of Australians were prevented from taking part in the Census due to the website crash.

The Prime Minister has announced an inquiry amid extensive criticism of the ABS and its $A9.6 million outsourcing contract to ex IBM.

A widely distributed list of contracts issued by the ABS for the 2016 Census shows that more was spent on office plants than load testing (see image below).

Reaction from the Web development and IT community on LinkedIn has been scathing

Jonathon Thorpe, a Systems Architect at Zettagrid, writes that "No one's ever been fired for buying IBM", an old adage from the 70s and 80s is starting to look like a $10m joke when you look at the ineptitude of the design of the hosting infrastructure for this year's Census. Some interesting observations from what can be seen of the Census site's presence on the Internet:

1. Everything is hosted out of a single site - IBM's data centre in Baulkham Hills.

2. There are only two upstream providers - Telstra and NextGen Networks advertising a single /24 with no evidence of distribution to any local peering exchanges which would massively help with maximising the capacity. (Read more HERE)

Performance Assurance CTO Paul Brebner cites reports prior to the Census night noting that Sportsbet offered odds of $1.50 that the Census website would crash on Census night, with $2.50 odds for it to remain online for the duration.

Punters were confident the site would crash, with 90% of the money backing an outage.

“Apparently the only people surprised that the ABS Census website “crashed” on census night (9 August 2016) was the ABS,” he writes.

“The census website is an available of a system with an Open Load profile. This means that there is no obvious or fixed limit to the number of potential users. There are really only 3 solutions to cope with this in practice.

“A Limit the number of concurrent users (turn the workload into a closed workload).

“In practice the ABS could have done this by changing their advertising to encourage a more uniform workload over a number of days.  Also, given that the ABS mailed out a unique access code to every household this should have made the workload more “closed” than open. However, in practice the main landing website was completely open which could easily have saturated first.

“B Have substantially more fixed network and computer resources than you expect to need well in advance.

“This can be expensive, particularly for a system with an advertised life of 1 day. You also have to ensure that the software running on it is scalable and can efficiently use all the resources. This is not trivial.

“C Elastically scale up (and down) the resources as required.

This is typically what happens in public cloud platforms.  However, due to the privacy requirements the use of an overseas public cloud would have been obviously problematic.  From reports the census website appears to have been implemented on a private IBM Softlayer cloud.  However, a limitation with private clouds is that they are often not as elastic (or have as many resources available) as public clouds. The Softlayer virtual machine spin-up time is advertised as between 5 and 15 minutes which for a rapidly increasing load may not have been fast enough to keep up with the demand.

Read more HERE

David Berkelmans, and IT Audit Executive Director at Synergy Group, believes the Census failure may impact risk appetite in the Commonwealth public service for future digital initiative projects.

“From the outset, the two clearest risks to the 2016 Census were:

• Breach of data
• Discontinuity of service

“I assume these two risks were right at the top of any risk register and management were satisfied that both risks had adequate mitigating controls. Before the census night, there was plenty of chest beating from across the political spectrum, within industry and the public service, especially around risk 1.

“From what I can gather from what I have read in the press, on 9 August the census website was subjected to denial of service and attempted data breach attacks.

“The attempted data breach attacks became so significant, that at some point, in order to ensure risk 1 did not eventuate, risk 2 was accepted and realised. Possibly one of the mitigating controls for risk 1 was to pull the plug to prevent risk 1 occurring.

“The statements made by the ABS and by the government indicate that risk 1 never occurred, but obviously risk 2 did. It would seem a very sensible decision was made that risk 1 could never occur and an acceptance that risk 2 would have to occur to prevent risk 1 from eventuating. What other controls could they have put in place to prevent risk 1? We may never know.

“What does this mean going forward? The Commonwealth Government’s agenda is to move towards more digital services in their interaction with the public. Any digital initiative will always have risk 1 and 2 at, or near the top, of any risk register.

“For me, the key question we need to think about is, “Can you ever 100% adequately mitigate risk 1 and risk 2 simultaneously when you’re asking millions of users to use one system at the same time?” I must admit, I don’t know the answer to this question and possibly no one does. Maybe the problems experienced during the 2016 Census show that you can’t. Or maybe you can and they missed a key control they could’ve put in.

“Regarding future digital initiatives, there probably won’t be many projects similar to the 2016 Census.

“The only future initiative that I can foresee falling into this category is electronic voting. In the wake of what has happened with the Census, I think it will be some time before any Commonwealth Government has the risk appetite to accept risk 1 and risk 2 simultaneously for an election. The ACT Government will be holding an election later this year and will be using electronic voting, however the number of registered voters in ACT compared to federally and other states is very small. It will however, be interesting to see if they can effectively mitigate both risks.

“Given the political impact of the 2016 Census, it begs the question of whether this will cause a rethink on all digital initiatives. Will it alter the risk appetite on existing and future projects? Will it stifle new projects? Will potential initiatives that could drive efficiencies be abandoned due to the risk of political fallout?

“I hope not, but we all need to watch this space.”

Read the ABS’s media release here.