Searchlight puts data in a whole new light
December 2009
NSIDC is reducing the distance between our users and our cryospheric data holdings, thanks to an evolution in NSIDC data architecture called Searchlight.
A new search tool, and more
NSIDC is changing our user access model from "Search and Order (and Wait and Download)" to "Discovery and Access." We are reducing the distance between our users and our data holdings, with our new infrastructure that supports users finding and browsing actual data holdings online. No more waiting for FTP staging followed by lengthy research to open and view data, only to find that it is not what you wanted or expected.
Powered by the NSIDC Searchlight engine, our Beta release Advanced Data Search online tool delivers data downloads immediately, with the user in control of on-the-fly output reformatting, reprojection and subsetting. In this way, we are delivering only what the user wants, and how they want it. The sooner our users can obtain what they want, in a format they can quickly understand and use, the more time they have to devote to their own scientific investigations.
While Searchlight provides extensive functionality for searching, browsing, retrieving and manipulating NSIDC data holdings, it was built with more than just data search in mind. Searchlight represents a suite of fundamental architectural changes that allow us to develop even more capabilities to help researchers work with cryospheric data.
Flexible, efficient, reusable
NSIDC Searchlight has been efficiently developed by focusing on reusing open source software, following best practices, and implementing proven design patterns. Future reuse of Searchlight components are supported at varying levels, ranging from the database and model components to Web services.
Built on open source systems and designed to provide reusable components, Searchlight uses GDAL and Proj.4 for manipulating data and format conversions, the netCDF Java library for creating netCDF output, MapServer and OpenLayers for defining spatial criteria and the JTS Topology Suite (JTS) in conjunction with Hibernate Spatial for database interaction and rich OGC-compliant spatial objects. The application utilizes popular Java and JavaScript libraries including Struts 2, Spring, JPA (Hibernate), Sitemesh, JFreeChart, JQuery and a PostGIS PostgreSQL database.
Going beyond search
Services for the Analysis of the Greenland Environment (SAGE) is the first application developed at NSIDC that leverages NSIDC Searchlight. SAGE was developed alongside Searchlight with a similar architecture, and reuses Searchlight components wherever possible. Searchlight handles all of the searching and preparing of data for SAGE. SAGE adds analysis capability and a unique interface for specific Greenland data sets. Using limited resources, the SAGE team has created an impressive application by reusing Searchlight. SAGE has demonstrated that Searchlight can be reused by other applications to speed up development of NSIDC applications providing new and/or targeted features.
Future developments
The Searchlight architecture is extensible and flexible for future enhancements, maintainable for a long expected life span, and capable of scaling to meet future growth and additional services to meet the changing needs of our user community.
In January 2010, the Advanced Data Search Beta, powered by Searchlight, will be available for use, including fourteen of NSIDC's most popular data sets. Contact NSIDC User Services for details. We plan to have a full release of the new search tool later in 2010, including all of our data holdings.
