As clear as water?
As clear as water?
South Australian Water put the top five business intelligence applications to the test.
Business intelligence developers claim their applications will make understanding the data within your business as clear and simple as a plain old tap water. Vendors in this market promise to slice and dice the data in your data warehouse and serve you with every item of information from a single source. However, as South Australian Water discovered, expecting a business intelligence application to clarify your data isn’t always as clear as you would think. The utility conducted a test of some of the top players, and the results poured cold water on the ambitions SA Water had for a business intelligence application.
South Australian Water Corporation is a government-owned company supplying water to 1.4 million people in the state. Like all utilities, the company generates tremendous volumes of data on its customers, their usage patterns, its large number of employees, as well as reams of technical data on internal systems.
”We are data rich, but information poor,” said Andrew Yates, the manager for information delivery at SA Water. For this reason, the corporation decided it needed a business intelligence (BI) application to analyse the data and allow it to make ad hoc enquiries. Prior to the adoption of a BI tool, SA Water had a number of different applications for analysing data on a departmental basis. Mr Yates and his team felt that the utility needed an application to analyse problematic areas, such as the corporate mobile phone usage, and a tool that could go enterprise wide for greater business analysis. Web Focus was in use by the human resources, finance and asset management teams, whilst Oracle and Crystal Decisions were used by retails and ministerial users.
Selecting a BI application followed a move to standardise the back end on the SQL database platform. Data from three different legacy systems was pooled onto SQL, and a browser based application was needed to view this information.
”[After] We standardised, as it was very difficult to get anything meaningful out of the legacy systems. At the time, the cubing capabilities of SQL were considered quite good, and better than the Oracle capabilities,” said Mr Yates. SA Water did not want to replace the legacy systems as it would be too costly. “We needed to extend their life and have useful information deducted,” he said.
Two options were floated: to install an out of the box BI application, or to build their own Web interface to the SQL databases. The latter idea was not pursued, as SA Water felt that it didn’t have the skill set, according to Mr Yates. Central to the adoption of the BI application was the desire to have all corporate data centrally managed.
”I am a believer that data needs to be centrally managed and controlled. You need to have one version of the truth. It wouldn’t have been uncommon to have two managers saying the same thing, but with different figures. You need centrally controlled data storage,” said Mr Yates.
As a result, the system would have to act as the funnel into which data was poured, with the data coming from a number of tools currently in use. These included: Web Focus for the human resources, asset management and finance departments; Oracle for the retail arm; Active Server pages for the financial extracts; and Crystal Reports for the Water transportation system.
SA Water set about designing a test for the top BI applications to complete. The test used SA Water’s Mobile Phone Data Mart as its test bed, and 13 days were set aside for the applications to prove their worth. As a state wide utility, SA Water’s engineers and management spend a great deal of their time on the road or away from the office. SA Water needed to analyse the information contained within the mobile phone bills.
Each vendor was sent a scenario with the details of the Web server they would have to connect to, and the tasks they would have to perform. A Windows 2000 Web server using IIS version 5.0 and a database server of the same specification using SQL Server 7 were to be connected to. The application then had to connect to SA Water’s Mobile Phone Catalogue. From this, the application would have to create and publish a report on all mobile phone bills up to $200, a report on all international calls for a month and to be sorted into descending order, a report on total call charges for each business unit within SA Water, and a report on individual call charges.
The Mobile Phone data was chosen because Telstra was providing SA Water with the bills, which were then placed into an Access database. The bills were then printed and manually delivered to the workers.
”We thought there has to be an easier way to do this,” Mr Yates said. “This was issued as the pilot to develop a mobile phone cube of the Telstra data and then the reports were developed on the Web.”
SA Water’s BI requirements demanded that the BI applications use a Microsoft data cube. Some vendors feel that this prevented their application from performing at its optimum level of ability.
With a test devised and the BI application market researched, SA Water sent out invitations and the test scenario to the top vendors. These included Business Objects, Crystal Decisions, Cognos, Brio, Pro Clarity, ComOps and Hummingbird. These vendors were chosen as they were recommended by Microsoft, or were members of the Microsoft Data Warehouse Framework Alliance. Each received a spreadsheet “listing the essential and desirable BI features.”
Some vendors failed to reply to the invitation. Amongst these were Brio, who failed to reply completely, whilst Business Objects and Hummingbird were late with replies. ”I don’t think they gave us any credence at all,” said Mr Yates of Brio’s treatment. The evaluation report stated that “Hummingbird and Business Objects almost fell into this category [non respondent] due to a lack of timely communications between their head offices and local South Australian channel partner”.
Five applications lined up for SA Water test. These were: Australian company ComOps with its Business Intelligence application; Cognos; Crystal Decisions with its Crystal Enterprise suite comprising Crystal Reports and Crystal Analysis; Hummingbird’s Analyze; and Business Objects’ Analytics. Each company was allowed a day to connect to the SA Water test database and carry out the tests.
Business Objects and its South Australian agent, SWOOD Financial Solutions, experienced difficulties in installing the software to the Aspect development environment used by SA Water. In the tests, SA Water found that the Business Objects team were unable to produce multi-dimensional reports in the demonstration time. The report stated that was due to the SWOOD staff having “limited” skills in this area. SA Water also found that the Business Objects thick client application was slow to produce data sets from a “mildly complex cube”.
”There are quite a few modules. It would add another level of complexity. You are always trying to do things fairly simply with an administrative type solution,” said Mr Yates of the Business Objects application.
Business Objects were given a chance for further tests, as the report states: “Efforts to further test the Business Objects product with guidance from local support staff proved a difficult task and could be attributed to a lack of multi-dimensional BI product knowledge with Infoview.”
”It took Business Objects a long time to process the information,” said Mr Yates. “It wasn’t particularly user savvy. There is definitely something within their architecture that makes it quite slow.”
Despite these problems, SA Water was generally impressed with the application. “The reviewer is confident that the toolset could provide SA Water with a gold plated Web query and reporting solution for management reporting providing a suitable Business Objects resource completes the installation and training,” the report stated. When SA Water followed up on Business Objects’ client references, it found that the University of Queensland felt that the product had “not lived up to expectations”.
”It was evident after several attempts to create a complex report that this was doomed to fail,” SA Water’s report said of the Hummingbird BI application, Analyser. Tests on the application were abandoned after two hours when Hummingbird representatives admitted that they connected to the SQL
Server 7 OLAP cubes, and not the SQL Server 2000 Analysis Services cubes as the test scenario requested.
”I was very disappointed with the results we got from Hummingbird,” Mr Yates said. As Hummingbird was a late entrant, SA Water said that it did not spend sufficient time with the application to truly judge its abilities against rival products and the test had to be abandoned.
The local company, Melbourne-based ComOps, very nearly won the SA Water deal. SA Water was impressed with the way that their Business Intelligence application automatically detected the table joins with the existing relational database when carrying out relational reports, as a result ComOps can detect the links between tables well. The application was quickly installed, and was producing reports within two hours.
”The ComOps staff successfully published all reports as requested, well within the allotted timeframe and they had sufficient time for lunch!" the report stated. “This was primarily attributed to their BI product manager Garry Holland.”
Mr Holland impressed SA Water, but he was also seen as the potential chink in ComOps’ armour. Mr Yates said he felt that as Mr Holland was pivotal to the ComOps application, if he was to leave the support and development of the application could be adversely affected. He added some slight criticisms of the ComOps application, saying that he felt that its presentation layer was not as good as some of the other contestants. ”ComOps was good at slicing and dicing the cubes, but the presentation layer wasn’t very good,” he said. ComOps won the first round of testing, and the SA Water report described the Melbourne company as a “clear ‘past the post’ winner” due to its knowledge of Microsoft multi-dimensional databases, ability to produce all required reports, satisfactory client references and the user-friendliness of the application.
A second round of tests was organised, as there were a number of outstanding issues with the ComOps application and the SA Water representatives felt that the others could address the requirements of SA Water. Before committing to ComOps, SA Water wanted to ensure the product was stable, had the necessary support, particularly as the company does not have an office or partner in South Australia, and that ComOps would be profitable. Also under question were ComOps’ ability to work with relational databases, and whether ComOps could service two large customers, as the company had an existing relationship with Australia Post.
Installation of the Cognos BI tool took six hours, and the reports generated from the application failed to provide users with totals, group headings, dates and mobile phone contacts. Multi-dimensional reports also failed. Mr Yates said that Cognos was unable to resolve these issues, and SA Water had to move on.
”Their ability to work with multi-dimensional cubes was lacking,” said Mr Yates. Eventual winner Crystal Decisions was successful in connecting to the Analysis server and in producing reports, although there was a problem with one part of the data. Along with Business Objects and Cognos, Crystal Decisions was invited to the second round of tests. The Crystal tools successfully reported to a Crystal portal, and the only reservation SA Water had of the product was the five-day training course required.
According to Mr Yates, SA Water were happy by the end of the second round of tests that the Crystal Decisions application would provide the multi-dimensional reports they required, and that Crystal Decisions had the resources to support SA Water.
As the manager of information delivery and one of the key people behind the BI tests, Mr Yates said he was surprised at how some of the top flight BI applications struggled to work within the SQL data cubes, especially the applications from Business Objects and Cognos.
”They are touted as the industry leaders. We were expecting far better things from them and it was a bit disappointing. It may be that we pulled them right out of their comfort zone,” he said. Mr Yates found that working within their own proprietary cubes these applications were fine. “I would feel that unless you had their product in the organisation previously, they are asking a lot to comply with their methodology. I just feel that they tend to push you down a direction they are comfortable with,” said Mr Yates. He advocated a move towards an open source frame of mind, which he described as being “advantageous”.
The tests that the SA Water representatives set were not, in their opinion, overly complicated and demanding, although Mr Yates admitted there was a stage where he wondered whether he was asking too much.
”BI vendors have always been touting how good their reports are. I found them wanting in the cube environment. I still feel that what we asked is fairly basic,” he said.
One aspect that did come out in the wash is that utilities and businesses looking for a business intelligence application in South Australia are woefully supported compared to those in New South Wales and Victoria.
”We really had to struggle to get people across. Some of them weren’t willing to make the trip from the east to do it. We ended up almost begging,” he said. SA Water is one of the largest organisations in the state, and as a utility is as large as some of the water utilities in Victoria.
Mr Yates said ComOps was the keenest, and once they had found the right person to deal with, both Crystal Decisions and Business Objects acted on their requests. SA Water contacted Hummingbird through its relationship with Solution 6, which has a presence in South Australia, but they were late to respond.
The final call was close. ComOps had impressed SA Water all through the process, but there were concerns.
”We were concerned with their size. They are heavily reliant on one individual, he was their expert, I think if something happened to him we would have been exposed,” Mr Yates said.
As a result the Crystal Decisions was selected, and although there are still some bugs to be ironed out concerning the scalability of the application, Mr Yates said his department has had some good wins, with Crystal Decisions being adopted by the finance department and the CFO found a figure that had been eluding him within 15 minutes. The Crystal Decisions application has been installed and is now trickling through SA Water.