Search engines have focused largely on crawling text on web pages, but Google is knee-deep in research about how to analyse and organise structured data, according to company scientist.
"There's a lot of structured data out on the web and we're not doing a good job of presenting it to our users," said Alon Halevy during a talk at the New England Database Day conference at the Massachusetts Institute of Technology,
Halevy was referring in part to so-called 'deep web' sources, such as the databases that sit behind form-driven websites like Cars.com or Realtor.com. Google has been submitting queries to various forms for some time, retrieving the resulting web pages and including them in its search index if the information looks useful.
But the company also wants to analyse the data found in structured tables on many websites, Halevy said, offering as an example a table on a web page that lists the US presidents.
And there are reams of those tables - Google's index turned up 14 billion of them, according to Halevy. He "realised very quickly that over 98 percent of these are not that interesting," but even after significant filtering there remain about 154 million tables worth indexing, he said.
One of Google's ultimate goals is to provide results that organise "aspects" of a search query, particularly an exploratory one such as 'Vietnam travel', as opposed to a query for a specific fact such as 'Vietnam population', Halevy said. The former query would produce information about visa requirements, weather and tour packages, for example.
The idea has echoes of the search service offered by Kosmix, but Google wants to go further, according to Halevy. "Kosmix will give you an 'aspect,' but it's attached to an information source," he said.
Searching for 'Vietnam travel' on Kosmix gives an organised set of results including restaurant reviews from the New York Times, images from Yahoo and Flickr, shopping information from Shopping.com and general web results from Google.
"In our case, all the aspects might be just web search results, but we'd organize them differently," Halevy said.