Lustre to sharpen storage agility
Lustre to sharpen storage agility
A new research project has been launched to write software that will shift the computation and transformation of data from client computers to storage devices, so that calculations will be made quickly inside the system of data intensive environments.
This project, called "active storage", is being conducted by Silicon Graphics and the Pacific Northwest National Laboratory, and is aimed at addressing priorities in the chemical, physical and biological sciences.
This software, which is based on the open source Lustre file system, is expected to improve the processing capabilities of large scale computing clusters.
Scientists will be able to take raw data sets from the file server and make calculations to identify data signatures and patterns before this data is transferred to client systems.
Scott Studham, the PNNL associate director, said. "By developing methods to perform computing inside the file system, we will be able to reduce the amount of redundant data transfers, which routinely undermines productivity and lengthens the time to solution.
"This vastly more efficient approach to data-intensive storage promises to significantly speed scientific discoveries in life sciences, national security, and even film and video production."
Active storage is also expected to be useful in areas that involve data mining, such as law enforcement, marketing databases, and the insurance industry.
Current data mining applications will have to adapt in order to be compatible with active storage, according to PNNL, but they do not think this will be too difficult to achieve.
Dr Eng Lim Goh, the senior vice president and chief technology officer at SGI plans to evaluate how the research could evolve the company's existing SGI InfiniteStorage CXFS shared file systems. "In this alliance with PNNL, we are committed to developing and delivering innovative storage technologies that solve problems unique to data-intensive environments.
"We've built systems with large, monolithic, globally addressable memories to contain these datasets in their entirety, which is one approach of solving the problem. The alliance with PNNL will work on another approach: in-storage analysis. We look forward to the possibility of incorporating results from this research into future version of CXFS."
Related Article: