Is Big Data bringing together the traditionally separate worlds of structured and unstructured data? Enterprise analytics leader SAS believes the growing uptake of its tools for social media analysis could be the trigger to crossing the divide.

At a Sydney seminar to introduce the company's tools for high performance analytics (HPA), Jim Davis, Senior Vice President and Chief Marketing Officer for SAS outlined how this was not just your traditional business intelligence.

“I don’t know of any organisations that aren’t thinking about incorporating unstructured data into their BI platform, whether its voice video, measuring consumer sentiment or from social media,” said Davis

This is the much hyped world of “Big Data”. Agreement on what is meant by the oft-used phrase is not universal. Davis was happy to volunteer his definition of “Big Data” as “data that exceeds the processing capacity of conventional database systems.”

Traditionally the task of gaining business intelligence has involved the job of moving transactional data off operational systems to create a data warehouse. This allowed data to be arranged in preformatted “cubes” to allow production of reports without impacting on the performance of business systems.

However the speed and power of parallel processing is now allowing companies such as SAS to offer the ability to analyse immense quantities of data in realtime using its powerful in-memory analytics solution.

“You don’t need OLAP anymore. You don’t need to wait for IT to build cubes,” said Davis.

The HPA tools that SAS provides require a blade server running regular Intel chips that can be configured from 4-blades at around $A50,000 up to a massive 48-blade server at $A500,000.

At the launch SAS gave a live demonstration of heavy duty predictive analytics in front of a mix of business and IT managers from Australia’s big banks, retailers and telcos.

Oliver Schabenberger, Ph.D, SAS Lead Architect for High Performance Analytics, cited the example of a US bank that was able to analyse a full one billion records using data from 500,000 loans it had made to its customers.

Previously the bank had only been able to run analysis on sample data sets, however with the trends realised in the full analysis it was able to identify opportunities for on-selling products that would more than cover the cost of the analytics hardware.

Schabenberger was able to show in real time the HPA solution completing in 23s a task that would previously have taken up to 10 hours.

“Making high performance analytics near realtime provides a chance to model scenarios and in the financial sector it provides analysis on the day of the trade, which will make people money.”

However the examples that SAS demonstrated were very much in the traditional mould of business intelligence, i.e. the analysis of data from transactional systems held in relational databases.

SAS offers a suite of products for Text Analytics  that is specifically focussed on finding information buried in unstructured text.

This includes products for automating classification of documents and creation of metadata, sentiment analysis and consolidating structured data analysis with unstructured text 

“If Big Data is the new oil, then High Performance Analytics is what will energise the whole economy,” - Mikael Hagstrom, SAS Executive VP, EMEA and Asia Pacific. 

James Foster,  SAS Australia and New Zealand Practice Lead for High–Performance Analytics says uptake for the Text Analytics suite has not been as rapid as traditional structured data analytics.

“However demand is now picking up as users wrestle with ever more unstructured content,” said Foster

This is being driven by a growing need for social media analytics.

“Companies want to know what consumers are saying about them. They can also relate this to lots of information they already have on their customers from call centre records and feedback forms.

“When somebody makes a comment about them on Twitter, companies want to be able to understand who is that person, what do they buy and what influence they have before they decide what action to take.

“Basic listening is already being done, but we can bring analytics to bear on the social media.”

This type of analysis has already been used successfully used by the United Nations to predict unemployment rates based on consumer sentiment expressed on social media.

Analysing half a million blogs, forums and news sites, SAS Social Media Analytics and SAS Text Miner examined two years of social media data from the US and Ireland for references to unemployment and how people were coping.

SAS compared mood scores and conversation volume with official unemployment statistics to see if upticks in those topics were indicators of spikes in unemployment. The analysis revealed that increased chatter about cutting back on groceries, increasing use of public transportation and downgrading one’s automobile could, indeed, predict an unemployment spike.

It has also been applied to analysing how many people complain of getting the flu as a means of keeping track of potential global pandemics.

SAS believes the use of high performance analytics to analyse social media this way will lead to a growing awareness of the power of its Text Miner product to exploit data that organisations already have, for instance email and call centre records.

“The concentration on social media makes enterprises realise they have been sitting on this data,” said Foster.

The 1823 Call Centre of the Hong Kong government's Efficiency Unit acts as a single point of contact for handling public inquiries and complaints on behalf of many government departments. 1823 operates round-the-clock, including during Sundays and public holidays. Each year, it answers about 2.65 million calls and 98,000 e-mails, including inquiries, suggestions and complaints.

“Having received so many calls and e-mails, we gather substantial volumes of data. The next step is to make sense of the data,” says the Efficiency Unit’s Assistant Director, W. F. Yuk. 

“Now, with SAS text mining technologies, we can obtain deep insights through uncovering the hidden relationship between words and sentences of complaints information, spot emerging trends and public concerns, and produce high-quality complaints intelligence for the departments we serve."

Other examples of unstructured data analytics that Foster was able to cite includes its use by manufacturers to analyse warranty claim forms to get a picture on what products are failing and what parts were involved.

“Most examples are around customer/citizen engagement, said Foster.

“Our focus with Big Data is on business outcomes. What are you trying to improve? What information can you leverage to do that?

“SAS is an analytics company. We are telling people to collect every bit of data that you can. Our focus is on analysing the data that is relevant to your organisation and use it to drive a business outcome.

Although Foster acknowledges an ongoing divide between those who manage “content”, i.e. unstructured data  and those who are responsible for data warehousing and BI.

“There is generally not a holistic view, as enterprise search is generally seen as an operational issue rather than part of a strategic use of information.

“SAS is known as a leader in business analytics – but we also assist customers around the world in information management. Getting access to the right data at the right time is critical in any analytical activity”

“The fact that in most organisations the data warehousing team is separate to the enterprise content management (ECM) team is an opportunity lost.”               

Beyond the firewall

One of the main differences between “Big” data and the data that organisations have traditionally analysed and managed is that it now comes from many locations outside your organisation, according to Christopher Preston, Senior Director, Integrated Technology Strategy, EMC Information Intelligence Group.

“You are actually tapping into databases and data sources that are outside your organisation, and that’s what’s different. This data is both structured and unstructured and this is where technologies like Hadoop have been applied.

“Organisations are tapping into government or other third party databases, even twitter feeds.”

Recent technology innovations now require businesses to incorporate sources such as mobile and social data, digital video, and sensory information into their case management processes.

EMC is promoting the use of its products such as Documentum xCP and Greenplum to drive action and insight from big data.

“Better business decisions can be made by responding to events, analytics-driven triggers and learning from previous cases,” said Preston. “The issues are many. How do I process that at scale? Traditional databases and traditional BI tools simply can’t do it. A limitation with BI today is you are only looking at a fraction of the points you could potentially be looking at.

“The mantra in big data is different to the typical BI approach. No matter how remote you think the relationship might be, bring the data in because you will be surprised at what insights, correlations or patterns that you may derive.”

Preston notes that traditional BI has always had the aspiration to deliver in realtime but has never been able to deliver on that promise due to native limitations.

“Traditional business intelligence (BI) is like an orchestra, it’s very structured and there is a  conductor, everyone knows their place. Big data is more like a jazz ensemble, where people are riffing off each other, it’s very iterative.”

Business processes have traditionally been tied exclusively to particular functions within an organisation, e.g. accounts payable, accounts receivable, HR.

“BI has been used in the context of process for decades, at any step of the business process I could always bring up a dashboard or have that as an event trigger based on tracking certain heuristics, i.e. a certain item of work has been sitting too long at a given step.

“Big Data is coming in at the next level where I might be able to proactively deliver  information to you as a knowledge worker based on an event or transaction, when you actually need it and whether you know you need it or not.

“I might be able to use weather patterns or other events to detect something that may affect my supply chain, for instance the 2011 flooding in Thailand that had a huge impact by damaging production facilities for car parts and computer components. If I see a similar pattern emerging I may be able to divert my supply chain to another region or engage different partners and do that dynamically

“You could always do that with process management but the issue is what are the triggering mechanisms and this is where big data actually enhances that.

“What it really gets down to is how do you get to this greater precision of decisioning.”

Preston also sees there is a huge treasure trove of information sitting in documents held by organisations in a range of repositories which people haven’t fully tapped into yet.

“Traditionally ECM and BI have been split, what you now see is the development of a new role, the data scientist, entrusted with job of helping a knowledge worker within an organisation be more effective. 

“We all manage and apply structured and unstructured information every day as we work. With Big Data we can enhance that with analytics to proactively route information to the right individual at the right time. That creates a totally different dimension for process improvement.”

Big data blogger Leon Katsnelson, who works in database deelopment with IBM, writes, “There is no magical point of terabytes, petabytes or any other arbitrary marker on the volume scale that serves as a signal for people to get interested in big data. 

“It is more of a feeling that you have ventured out in some big waves and you are in the environment that far exceeds the design parameters of your ship. It does not mean that you need to batten down the hatches and hope to weather the storm. 

“Big data is not a fad or a hurricane that will pass returning us to the calmer seas. It is quickly becoming the fact of life for the modern enterprise and IT will have to learn to deal with it. 

“There are people out there who travel the world over seeking out giant waves to ride. Take a look at some of the startups and established companies who have embraced the big data challenge and are enjoying the ride.”