The Future of Search

Search is big business these days, but are the gatekeepers to the information superhighway really giving you the results you want? Greg McNevin speaks with Wikia CEO Gil Penchina about the future of search…

With so much information out there, search is a fundamental tool that keeps the internet as we know it running. Bringing the world to everyone’s fingertips in a simple, straightforward way is what has rocketed Google into its lofty position. However, as Spider-man’s clichéd motto tells us, with great power comes great responsibility. Could Google’s search muscle and colossal advertising capabilities have corrupted its ability to deliver information?

The guys at Wikia certainly think something is rotten outside the state of Denmark. As Wikipedia founder Jim Wales writes on search.wikia.org, “search as we know it is broken. It is broken for the same reason that proprietary software is always broken: lack of freedom, lack of community, lack of accountability, lack of transparency. Here, we will change all that.”

Wales has been nurturing a new vision for search, one that like other wiki projects, is rooted in transparency, open standards and user interaction. The project, informally dubbed Wikiasari, aims to bring people-power back into the equation and improve search results through human judgement rather than algorithms and keywords.

“Jimmy had the idea for a user-powered search engine back in 2004, and named the project Wikiasari,” said Penchina. “Recently, we started thinking about how to turn that into a reality.

“Wikia is focused on enabling people to collaborate on any project that has a social good. As a result we are leading the collaborative search project (Wikiasari).”

The project is based on Nutch, an open source search engine built on the Lucene open source search and index API supported by the Apache Software Foundation. While Search Wiki is still in the idea stage and details are scant, the idea on its own has generated an immense amount of interest regarding how it will tackle the problems inherent in current search technology.

Policing the Police

Traditional search engines such as Google and Yahoo! use various complex algorithms to determine search results, and few outside the company know exactly how they work. Google technology and its page ranking system in particular are shrouded in mystery, leaving questions surrounding objectivity and accuracy of its rankings up in the air.

With search engine ranking so important to modern business, lack of transparency can raise all sorts of questions as to how one page is more applicable to a given search than another. And if Google can blacklist a site (aptly known as the Google “Death Sentence”) then one could ask where the accountability for these decisions lies?

When almost 50% of searches worldwide are conducted via Google, does this affect its objectivity? And what about the data it collects? What are its obligations to personal privacy?

Another concern confronting traditional search techniques is that they can be twisted to yield more desirable results for affiliates and advertisers, not to mention to appease less than liberal governments if push comes to shove. Search for a product on Google for example, and chances are you’ll have to wade through a host of affiliate pages, link farms or sent through endless circles of price grabber sites.

The Search Wikia project will of course have to rely on algorithms too in order to deliver results. The maths behind it, however, will not be left to its own devices. This is where the project’s user-power comes in. By empowering users to give a search result a rating for effectiveness and then ranking accordingly, the Search Wikia team believes that bias, no matter where it comes from, can be largely eliminated.

“We believe that you need a combination of machine labour and filtering, combined with human judgement to get the best search results - and that in the end, human judgement should trump machine algorithms,” said Penchina.

For example, BMW was famously blacklisted from Google for trying to artificially boost its ranking by creating fake websites to link back to its main web presence, and Wikipedia itself has popped up in the news a few times when malicious users have tried to use it to manufacture political scandals such as the recent Del Monte share scam.

While complete openness and the ability to be edited by anyone would be the engine’s biggest advantage, there is an argument that it could also prove to be its biggest vulnerability. Much more so than is usual perhaps with open source software.

Penchina and Wikia in general has faith in people, however, saying that it is precisely its openness that will let the engine avoid being “gamed” by malicious users, avoid ranking and advertiser bias and above all, avoid being directed by governments.

“We believe people are basically good,” said Penchina. “I worked at eBay, where the same comments were often made. Our job is to empower good people, encourage them to make the world a better place, and protect them from the few bad eggs.

“All ranking decisions will be transparent and auditable by anyone. We believe that transparency reduces the risk of commercial interests trumping the ‘right answer’.”

He adds that while some may choose to act dishonestly, “if enough people see a bad result, someone fixes it (the same reason web-mail providers have a spam button).”

On top of this, because the project is open source it is likely that the vast amount of information and user contributions it collects, collates and categorises will be available for anyone to access, providing complete transparency.

Forever Polishing

With such a vast array of information out there and billions of search possibilities, the project has a lot of work ahead not only in its technical development but also to crawl the web and sharpen result accuracy. However, Penchina believes that over time the project could become a force to be reckoned with. “Like Wikia and other user controlled projects, in the early days, you have a rough rock, but with enough polishing a diamond emerges,” he said.

“Our goal is to enable people to control and contribute to the media they consume, whether it is books, magazines, search engines or other types of media. Over the next 5-10 years we hope to bring the power of collaboration to many other aspects of the internet.”

Like open source, timing has been extremely important. 10 years ago the thought of giving intellectual property away for free was a concept that existed mainly for small, hobbyist built applications that had little commercial potential. Now, due mainly to the open source software movement and the communities driving it, technological altruism and the idea of working for the greater good of society is spreading, and with it free software and free information for all.

As computer use and the number of savvy net users increases so too does the collective knowledge of the community, making a user-powered search engine viable for the first time.

Penchina certainly agrees on this front; “I do believe that this level of collaboration might not have been possible 10 years ago, due to a lack of critical mass of experienced Internet users.”

Support for open source and community-based projects is also increasingly attracting the attention of big-name investors. While Search Wikia is a volunteer project, the Wikimedia Foundation takes donations to fund its projects and Wikia Inc. itself has recently scored several million in funding from Amazon.com. Each Wiki branch is its own entity, however, with support snowballing on so many fronts the internet landscape could be in for some serious wiki-renovation.

While it would be hasty to give the project a “Google Killer” tag, Penchina is excited about the future of user-powered search, and says that judging by the response to the idea in the media and by the general public, the project’s future is bright.

“I am astonished at the worldwide interest this simple idea has generated, the 100's of volunteers who have already shown up and the passion for change that we have seen,” said Penchina. “Frankly, we can't do this by ourselves and it has been tremendously exciting to see how much passion this idea has created amongst the technology community and beyond.”