There is a strategic value in moving beyond federated search, writes Scott Coles.

Poor and limited access to critical intelligence has been at the heart of high-profile information access failures worldwide.  In the United States, federal intelligence bodies failed to “connect the dots” they had been compiling when Al Qaeda terrorist Umar Farouk Abdul Mutallab attempted to blow up an airliner in late 2009.

In the United Kingdom, the cases of Khyra Ishaq and Baby P highlighted the all-too-common lack of early warning systems that could have saved the lives of young victims.  Child protection services agencies possessed the information that could have protected Ishaq and Baby P but not the infrastructure necessary to alert them to potential problems.

These, of course, are just the most recent examples of a persistent knowledge management issue.  But our biggest failure to date in trying to tackle these challenges is to insist on a complex data migration engagement, where we attempt to merge massive amounts of information from disparate data sources.  Not only are these projects costly and time consuming, but they almost always fail.  Federated search has emerged as the palatable alternative, as it eliminates the need for any data preparation or migration. 

A simple definition of federated search would be “the searching of multiple data sources simultaneously”. Search usage in business environments has evolved significantly even over the last twelve months. It was only a little while ago that the argument for search was user productivity.

The focus then shifted to ‘Enterprise Search’ – providing users access to ‘whole of company’ information. Enterprise search then evolved to federated search – to the point where the two terms are now used interchangeably.

With today’s heightened focus on risk, many CIOs are now recognising the outcomes that can be generated through federated search. The key premise being to avoid risky and costly data migration or physical aggregation exercises, and leave data in place. In today’s enterprise, data needs to live and breathe in different places. 

The various information systems in a typical enterprise are purpose-built for the type of information that they contain.  The term “federated search” does not really do the concept justice.

More accurately, it is the “virtual aggregation of data” that is increasingly inspiring government organisations and commercial enterprises to take this strategic approach to knowledge management.

No longer compelling is the move of vast amounts of legacy data into a new document management system just to provide access. If the definition of legacy data is “data that was created in the past and is not going to change”, then why not provide user access to this data through federated search and implement the new document management system for new documents, thus avoiding the migration?

Of critical importance today is an organisation’s ability to virtually aggregate key information from various repositories without the cost or risk of the physical alternatives.  What business intelligence technology has done for structured information, virtual data aggregation is doing for unstructured content.

To this end, we’ve seen customers such as Volkswagen leverage virtual aggregation to identify potential legal pitfalls before they become a material issue.  Virtual aggregation of data is also a very real requirement with Victoria’s Department of Treasury and Finance, where data from multiple HP TRIM repositories across multiple internal departments is being virtually aggregated, thus avoiding creating a single monolithic alternative.

Federated organisations in particular can benefit enormously from this approach, which does away with the requirement for a large central IT exercise.

If today’s executive dashboard highlights an organisation’s financial KPIs, tomorrow’s interface will offer key decision makers a window into everything from customer sentiment to competitive threats. 

Virtual aggregation is at the heart of this solution, providing executives with the confidence that their information access capabilities are comprehensive, reliable and relevant.