Since 1996, NCSA's HDF group has collaborated with members of the tri-lab Data Models and Formats (DMF) group, and others, to develop and support a common standard format and I/O library that can handle the enormous I/O demands of ASC applications on the labs' major systems. To meet these requirements, the three labs and the NCSA HDF group together developed the HDF5 file format and I/O library. HDF5 has subsequently become the format of choice for a wide and growing range of applications, not only within the labs, but nationwide. Indeed, the development and support of HDF5 within the Labs has spawned more than 200 HDF5 applications worldwide.
The early years of the project focused on developing the basic format and library. Thereafter, the focus expanded to adding important new features to the format and library, maintaining the software and documentation, developing tools to make it easy to work with the format, working with vendors to support the format, and working closely with our users to identify and address the needs of specific applications. The result of this work was to transform HDF5 into software of production quality that offers competitive I/O performance on the primary ASC machines.
As applications matured, and as newer systems came on line, the performance and functionality of the parallel version of HDF5 have emerged as particularly important, so the project embraced this area in addition to the others.
HDF5 benefits ASC most if it is available and used beyond the ASC project, by the scientific community generally. An open HDF5 standard expands the number of colleagues with whom we can share data and software, and encourages the development of applications and tools for working with HDF5 data, including commercial applications and tools.
The Tri-Lab investment in HDF5 has paid dividends well beyond those in the Labs themselves, as HDF5 has become a technology of choice for hundreds of projects and thousands of users. HDF5 has received significant additional support from NASA's Earth Science Enterprise, which uses HDF5 as the standard format for its most recent major mission of the Earth Observing System (EOS). As well, HDF5 has been chosen as the distribution and archival format for the successor to EOS, the National Polar Orbiting Environmental Satellite System (NPOESS). These two missions must distribute one terabyte and six terabytes of data. respectively, in the HDF5 format.
Other organizations and agencies that have supported HDF5 research and development in recent years include the National Science Foundation, the Department of Defense, Boeing, the National Institutes of Health, and the National Archives and Records Administration.
The value of an open HDF5 is also realized in the growing number of commercial products that support HDF5, including scientific software such as Matlab and IDL. In addition, several communities have chosen to standardize on HDF5, such as the worldwide Nexus project supporting Neutron, X-ray and muon science, and the CFD General Notation System (CGNS) data standards project.- - Last modified: 13 February 2014