This document compares HDF5 with Hadoop and discusses how they can be used to complement one another.
This document defines a new HDF5 interface based on an architectural style for network-based architectures called REpresentational State Transfer (REST). It describes HDF5 resources, URIs, and resource representations, and shows a simple example of how to use this interface to populate an HDF5 store.
Please note that this is version 0.1 of this specification. Resource granularity and representations are based on our own experiments and a limited understanding of "real" applications.
This paper was co-written with Mark Howison at Lawrence Berkeley National Laboratory, and describes useful performance improvements that are possible by tweaking HDF5.
This document describes experiments that were done to determine the usefulness of accessing HDF5 files with SSHFS.
Delivered at the "Digital Preservation Interoperability Framework (DPIF) Workshop", National Institute of Standards and Technology (NIST), Gaithersburg, Maryland, March 29-31, 2010, http://ddp.nist.gov/workshop.
Big HDF FAQs
Everything that HDF Users have Always Wanted to Know about Hadoop . . . But Were Ashamed to Ask
RESTful HDF5 Interface Specification 0.1
Tuning HDF5 for Lustre File Systems
Accessing HDF5 Files via SSHFS (FUSE)
HDF and Data Preservation
Balancing Performance and Preservation -- Lessons Learned with HDF5 (presentation), Folk
This document describes The HDF Group's experience applying virtualization to the HDF Windows support effort.
HDF Virtualization Review
NCL, GrADS, PyHDF, GDL, and GDAL are five open source software packages that can manipulate either HDF4 or HDF5 data. This document provides a limited evaluation of HDF support for these packages from the user's point of view.
This document describes how to use NCL, GrADS, PyHDF, GDL, and GDAL to access HDF files.
This brief describes how the HDF5-OPeNDAP project facilitated the use of HDF-EOS data to track Beijing's air quality during the August 2008 Olympics.
NetCDF-4 is an I/O software package that retains the original netCDF4 APIs while using HDF5 to store the data. Sponsored by NASA ESTO, netCDF-4 is the result of a collaboration between Unidata and The HDF Group.
This report on the performance of netCDF-4 uses benchmarks and examples to:
- 1) Compare the performance of netCDF-3 and netCDF-4
- 2) Discuss performance tuning with netCDF-4
- 3) Discuss performance pitfalls with netCDF-4
Some of the performance tuning and pitfalls discussions may also be of interest to users of HDF5 who do not use netCDF-4.
Evaluation of HDF Support for NCL, GrADS, PyHDF, GDL and GDAL
Usage of NCL, GrADS, PyHDF, GDL and GDAL to Access HDF Files
Brief: HDF5-OPeNDAP Project used in tracking air quality for 2008 Olympics
NetCDF-4 Performance Report: June, 2008
Older papers are available from https://gamma.hdfgroup.org/papers/papers/.- - Last modified: 26 January 2017