The Deep Web

Open Source Big Data Initiative.

The Internet contains a vast trove of information, sometimes called the “Deep Web”, that isn’t indexed by search engines.

The Defence Advanced Research Projects Agency (DARPA) has been developing tools as part of its Memex program that access and catalogue this mysterious online world. Researchers at NASA’s Jet Propulsion Laboratory in Pasadena, California, have joined the Memex effort to harness the benefits of deep Web searching for science. Memex could, for example, help catalogue the vast amounts of data NASA spacecraft deliver on a daily basis.

“We’re developing next-generation search technologies that understand people, places, things and the connections between them,” said Chris Mattmann, principal investigator for JPL’s work on Memex.

Memex checks not just standard text-based content online but also images, videos, pop-up ads, forms, scripts and other ways information is stored to look at how they are interrelated.

“We’re augmenting Web crawlers to behave like browsers, in other words, executing scripts and reading ads in ways that you would when you usually go online. This information is normally not catalogued by search engines,” Mattmann said.

Additionally, a standard Web search doesn’t get much information from images and videos, but Memex can recognize what’s in this content and pair it with searches on the same subjects. The search tool could identify the same object across many frames of a video or even different videos.

The video and image search capabilities of Memex could one day benefit space missions that take photos, videos and other kinds of imaging data with instruments such as spectrometers. Searching visual information about a particular planetary body could greatly facilitate the work of scientists in analysing geological features. Scientists analysing imaging data from Earth-based missions that monitor phenomena such as snowfall and soil moisture could similarly benefit.

Memex would also enhance the search for published scientific data, so that scientists can be better aware of what has been released and analysed on their topics. The technology could be applied to large NASA data centres such as the Physical Oceanography Distributed Active Archive Center, which makes NASA’s ocean and climate data accessible and meaningful. Memex would make PDF documents more easily searchable and allow users to more easily arrive at the information they seek. Awareness of existing publications also helps program managers to assess the impact of spacecraft data.

All of the code written for Memex is open-source. JPL is one of 17 teams working on it as part of the DARPA initiative.

Memex is related to DARPA’s previous Big Data initiative called XDATA, managed by DARPA Program Manager Wade Shen. That research effort is also aimed at processing and analysing large amounts of data, with defence, government and civilian applications. JPL was one of 24 groups involved.

“We are developing open source, free, mature products and then enhancing them using DARPA investment and easily transitioning them via our roles to the scientific community,” Mattmann said.

Continuum Analytics Inc. of Austin, Texas, and Kitware Inc. of Clifton Park, New York, are partners on the JPL collaboration with Memex. JPL is a division of the California Institute of Technology.


Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s