Microsoft Research Researcher Xie is working on technology called Photo2Search, which is designed to provide information on the go for users of camera phones. Photo2Search gives users a way to search a Web-based database by using nothing more than an image captured by a cellphone equipped with a digital camera.

“This technology,” Xie says, “aims to solve the problem of mapping a physical-world object to a digital-world object. You see an object in the physical world, and you want to know the corresponding information in the digital world—for example, its price on the Web, user comments, or Web sites. There are many different solutions. You can use a bar code or radio-frequency identification. But using a picture of the object is very convenient and very easy to deploy.”

Photo2Search works like this: Seeking information about something seen, a user takes a photo of the object and sends the photo, via e-mail or Multimedia Messaging Service, to a Web-based server, which searches an image database for matches. The server then delivers database information - whether it be a Web page featuring the object in the photo or information associated with the object - to the user, who can act on the information received: read a menu, enter a gallery, book a hotel room, make a purchase.

When Xie, who collaborated on the Photo2Search technology with some Microsoft Research Asia colleagues and a handful of visiting students, first developed his concept of image-based Web searching using photos captured using camera phones, most prior work in this area had been based on Content Based Image Retrieval (CBIR) approaches, which index the content of images by features such as color, texture, shape, object layout, and edge direction. But while such approaches, which take considerable computational resources, are able to identify photos with some similar visual features, they don’t necessarily excel at locating ones with the same prominent object or scene pictured in a query image.

“We found speed is a very big concern,” he says. “Most computer-vision algorithms are slow.”

In the second half of 2005, Xie and colleagues rebuilt the system, with image matching based on some well-known computer-vision algorithms that extract features from images. That choice proved productive, resulting in an efficient, high-dimensional index that can search through a large image database and return results quickly - combing through a collection of 6,000 images and delivering matches in a mere three seconds using a common laptop. At that point, the process begins to enter the realm of the practical.

“The coolest thing,” Xie says, “is that you can use a pure image as a query, with no text. That is a totally new search experience.”

It remains to be seen how Photo2Search might be incorporated into an upcoming product, but the possibilities are intriguing.

“There is still a lot to do to make this technology into products,” Xie cautions, adding, “If we can make it practical, then this is a big contribution for both local-search and mobile-search products.”

Source: http://research.microsoft.com/displayArticle.aspx?id=1434