hdf images hdf images


Peter Cao, The HDF Group

Mike Folk, The HDF Group

Nahil Sobh, NCSA

Paul Ricker, NCSA

- - August 2006


It is common to apply simple data reduction operations and filters to large data, in which the resulting data is small relative to the original data. When the data is stored remotely, the cost of moving data to a local platform for such analysis can be very high. A great deal of time can often be saved by performing these operations on the data in place.

An application, "HDF5-on-SRB 1," addresses this problem by enabling a client to access objects in-place in HDF5 files stored remotely in an SRB. The success of the HDF5-SRB project has prompted the NCSA and SDSC to take the project to the second phase: supporting selected applications in the Strategic Applications Program (SAP). FLASH 2 is one of the SAP projects selected for initial implementation.

FLASH is a modular, portable, highly scalable, adaptive-mesh simulation code for astrophysical hydrodynamics problems. FLASH Cosmology 3 is a set of FLASH modules for cosmological applications, particularly the simulation of a large-scale structure.

There are two challenges in the FLASH cosmology simulations.

For example, a command line tool, FLASH slice extractor developed by Paul Ricker at NCSA, retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement) data file from FLASH simulations.

Figure 1 shows an example of such a slice of AMR data. This tool requires the data file to be at local machine. In the following proposed work, we would like to extend this tool to use the HDF5-SRB model to retrieve part of a data file, such as a slice, directly from where the data is generated.

Also, everyone will have access to the updated data after a new simulation run.

* The project is sponsored by the NCSA/SDSC-led Cyber-Infrastructure Partnership (CIP) and the National Laboratory for Advanced Data Research (NLADR), NFS PACI project in support of NCSA-SDSC collaboration

Slice (gas temperature) and isosurface (gas density) in a 5123 particle/mesh simulation

Figure 1 -- Slice (gas temperature) and isosurface (gas density) in a 5123 particle/mesh simulation


The goal of this project is to use the HDF5-on-SRB for performing simple operations from a remote client on large Flash files stored in a remote SRB, and in so doing develop a framework to provide similar capabilities to other applications.

If successful, this will enable users to query and analyze Flash files without having to incur the latency and storage costs associated with fetching and storing the files locally.



This project will be divided into three phases: Phase I, Phase II and Phase III. Phase I and part of phase II are required. Phase III and part of phase II are optional, depending on the duration and amount of the funding. The time estimates are based on an half (50%) FTE.

Phase I: a command line client utility and server function to support selection of FLASH AMR mesh data.

Tasks Start Finish
Collaboration between NCSA and SDSC   Duration of the project
Collecting requirements 10/2/06 10/13/06
Exploring the current tool 10/16/06 10/27/06
Implementing server function 10/30/06 12/22/06
Implementing client utility 12/25/06 2/16/07
Testing with FLASH AMR mesh data 2/19/07 3/16/07
Deploying to evaluation server 3/19/07 4/13/07
Documentation and release 4/19/07 5/11/07

Phase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation

Tasks Start Finish
Collaboration between NCSA and SDSC  Duration of the project
Query and return simple statistics 5/14/07 8/17/07
Extract and return particle information 6/25/07 8/17/07
Analyze and interpolate line segment 8/20/07 10/12/07
Add support for png and ASCII text 10/15/07 11/23/07

Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.



- - Last modified: 13 February 2014