Innovating the Search Engine

Listen up, Google! I have had a vision and I know what the next step in search engine technology is. It came to me as a spark of sudden hope during a frustrating journey down search engine back roads.

You see, the folks at Read/Write Web are right. The Search is “game-over.” Google has won. But there is still a search for the Google-killer.  Problem is, everybody is going after the wrong features. from Read/Write Web: From Search to (Re)Search…

• Cool new features – user interface, alerts, visualization or whatever. The problem is that no single feature is enough for users to switch from Google and most people don’t have the time or motivation to use multiple search engines.
• Natural Language. There is big money riding on this one. It feels wrong to me. This is too much heavy science to crack problems that are totally simple for humans; and Web 2.0 is getting pretty good at aggregating the expanding global pool of knowledge workers.
• Vertical Search and Human Search. I put the two together. Human Search works best in well-defined domains. There are lots of Vertical Search engines that already work well and plenty more will come.

The article cites some good ideas for what this mythical Web 2.0 app should be, but that’s where my idea diverges.

I’ve been mulling over the idea of “concept searching” in my mind as of late. The problem with today’s search engines is that they’re still related strictly to my search terms. The problem is exposed when I need to do a search for terms that may not appear exactly as I type them, but together form more of a characteristic or conceptual pattern of something else. For instance, if I search for red, orange, and yellow, I will get a wide diversity of sites which have these words prominently displayed in HTML content. But maybe what I’m really after is artistic impressions of warm color use. Or perhaps I’m looking for images of fire…

The point is, the terms I specified are related in ways beyond just their placement on a web site. Flickr does a very nice job of demonstrating the power of tagging, categorizing, and respectively searching. By combining definitions of the words we’re searching for, they’re able to build what are called “clusters.” Clusters are groups of tags that seem related and Flickr allows you to pick a cluster to further identify the concept you’re searching on. It’s not perfect yet, but it demonstrates my point very well.

http://www.flickr.com/photos/tags/red/clusters/:

This is the next revolution in search engines. It’s a combination of Natural Language and research search types. By analyzing the words’ meanings and allowing the user to refine their particular definition, Conceptual Searching can be made possible.

*UPDATE:
I haven’t read into this much yet, but it looks like Microsoft is already entering into discussions based upon what I’ve said. I like to think they’re reading my blog and getting their best innovations here!

Microsoft Research