Musings on visual search

The story so far: Search engines throw textual queries against textual databases. You want to search for images? Tag them with textual metadata that fits into the search engine paradigm.

The problem: Some things are hard to put in words. You want to hand the search engine an image (or an audio clip or a video clip or the tingling feeling you get in your toes when you get a text from your crush) and have it return things that are similar in one or more ways, maybe clumped by the type(s) of similarity. For example you might feed a search engine a backlit picture of your cat and get one cluster of results for images that have similar lighting, a cluster for images with calico cats, and a cluster of domestic scenes (this list limited only by my narrow imagination).

Various approaches to designing this kind of search have been tried. Graduate students in artificial intelligence research methods of extracting semantics data from unstructured (vis-a-vis semantics, at any rate) files. Google crowdsources extensive image tagging through a two-player game of description. Etsy has a great widget that finds product pictures based on your selection of a color from a trippy color bar.

Meanwhile, there are lots of excellent ideas for visual representations of search results and some of these will doubtless start replacing the List o' Links soon.

Those are great projects.

On the other hand, I think people sometimes loose the thread when attacking the problem. For example, you feed textual search terms to oSkope and it gives you back a List o' Pics instead of a List o' Links. The pics don't seem to be organized by any relationship between them and the results are flat. It's more difficult to use than the List o' Links with little annotations that are so familiar from Google's search research and it gets us nothing. If we want to see the front cover for a particular book in the absence of any other data, we can do a Google Image search and be done in seconds. oSkope doesn't help us decide which of the zillion genre fantasy books sold on Amazon we'd like to take a closer look at.

Visual search and visual search results need to help us quickly hone in on content we want to spend time with by making relationships between results clear, providing quick options for different relevant views, and giving us enough information to make decisions. The future of search is search/browse on mash-up content pages that dynamically suck information from all over the place. Visualization tools will be excellent helpers on these pages, but only if they add real value by helping us find exactly what we need as quickly as possible. No matter how pretty a widget like oSkope is, nobody is going to use it unless it can do more than look slick at a marketing demo.