Please, help us to better serve our user community by answering the following short survey: https://www.hdfgroup.org/website-survey/
HDF5 1.14.5
API Reference
Loading...
Searching...
No Matches
H5FDsubfiling.h File Reference

Data Structures

struct  H5FD_subfiling_params_t
 Subfiling parameter structure that is shared between the H5FD_SUBFILING and H5FD_IOC drivers. More...
 
struct  H5FD_subfiling_config_t
 Configuration structure for H5Pset_fapl_subfiling() / H5Pget_fapl_subfiling() More...
 

Macros

#define H5FD_SUBFILING   (H5FDperform_init(H5FD_subfiling_init))
 
#define H5FD_SUBFILING_NAME   "subfiling"
 
#define H5FD_SUBFILING_CURR_FAPL_VERSION   1
 
#define H5FD_SUBFILING_FAPL_MAGIC   0xFED01331
 
#define H5FD_SUBFILING_DEFAULT_STRIPE_SIZE   (32 * 1024 * 1024)
 
#define H5FD_SUBFILING_DEFAULT_STRIPE_COUNT   -1
 
#define H5FD_SUBFILING_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 "_%0*d_of_%d"
 
#define H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 ".config"
 
#define H5FD_SUBFILING_STRIPE_SIZE   "H5FD_SUBFILING_STRIPE_SIZE"
 
#define H5FD_SUBFILING_IOC_PER_NODE   "H5FD_SUBFILING_IOC_PER_NODE"
 
#define H5FD_SUBFILING_IOC_SELECTION_CRITERIA   "H5FD_SUBFILING_IOC_SELECTION_CRITERIA"
 
#define H5FD_SUBFILING_SUBFILE_PREFIX   "H5FD_SUBFILING_SUBFILE_PREFIX"
 
#define H5FD_SUBFILING_CONFIG_FILE_PREFIX   "H5FD_SUBFILING_CONFIG_FILE_PREFIX"
 

Enumerations

enum  H5FD_subfiling_ioc_select_t {
  SELECT_IOC_ONE_PER_NODE = 0 , SELECT_IOC_EVERY_NTH_RANK , SELECT_IOC_WITH_CONFIG , SELECT_IOC_TOTAL ,
  ioc_selection_options
}
 

Functions

H5_DLL hid_t H5FD_subfiling_init (void)
 Internal routine to initialize H5FD_SUBFILING driver. Not meant to be called directly by an HDF5 application.
 
H5_DLL herr_t H5Pset_fapl_subfiling (hid_t fapl_id, const H5FD_subfiling_config_t *vfd_config)
 Modifies the specified File Access Property List to use the H5FD_SUBFILING driver.
 
H5_DLL herr_t H5Pget_fapl_subfiling (hid_t fapl_id, H5FD_subfiling_config_t *config_out)
 Queries a File Access Property List for H5FD_SUBFILING file driver properties.
 

Macro Definition Documentation

◆ H5FD_SUBFILING

#define H5FD_SUBFILING   (H5FDperform_init(H5FD_subfiling_init))

Macro that returns the identifier for the H5FD_SUBFILING driver. Returns a file driver identifier if successful; otherwise returns H5I_INVALID_HID.

◆ H5FD_SUBFILING_CONFIG_FILE_PREFIX

#define H5FD_SUBFILING_CONFIG_FILE_PREFIX   "H5FD_SUBFILING_CONFIG_FILE_PREFIX"

Macro for name of the environment variable that specifies a prefix to apply to the subfiling configuration filename. Useful for cases where the application wants to place the configuration file in a different directory than the default of putting it alongside the generated subfiles. For example, when writing to node-local storage one may wish to place the configuration file on a scratch file system readable by all nodes, while the subfiles are initially written to the node-local storage.

The value set for this environment variable is interpreted as a pathname that must already exist.

NOTE: As this prefix string will be encoded in the driver info message that gets written to the file, there is an upper limit of about ~900 single-byte characters for this string, though possibly less due to other information the driver may encode. Avoid long prefix names where possible.

◆ H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE

#define H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 ".config"

The basic printf-style template for a H5FD_SUBFILING driver configuration filename. The format specifiers correspond to:

%s
base filename, e.g. "file.h5"
%PRIu64
file inode, e.g. 11273556

yielding a filename such as:

file.h5.subfile_11273556.config

◆ H5FD_SUBFILING_CURR_FAPL_VERSION

#define H5FD_SUBFILING_CURR_FAPL_VERSION   1

The version number of the H5FD_subfiling_config_t configuration structure for the H5FD_SUBFILING driver

◆ H5FD_SUBFILING_DEFAULT_STRIPE_COUNT

#define H5FD_SUBFILING_DEFAULT_STRIPE_COUNT   -1

Macro for the default Subfiling stripe count value. The default is currently to use one subfile per node.

◆ H5FD_SUBFILING_DEFAULT_STRIPE_SIZE

#define H5FD_SUBFILING_DEFAULT_STRIPE_SIZE   (32 * 1024 * 1024)

The default stripe size (in bytes) for data stripes in subfiles

◆ H5FD_SUBFILING_FAPL_MAGIC

#define H5FD_SUBFILING_FAPL_MAGIC   0xFED01331

Unique number used to distinguish the H5FD_SUBFILING driver from other HDF5 file drivers

◆ H5FD_SUBFILING_FILENAME_TEMPLATE

#define H5FD_SUBFILING_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 "_%0*d_of_%d"

The basic printf-style template for a H5FD_SUBFILING driver subfile filename. The format specifiers correspond to:

%s
base filename, e.g. "file.h5"
%PRIu64
file inode, e.g. 11273556
%0*d
number (starting at 1) signifying the Nth (out of total number of subfiles) subfile. Zero-padded according to the number of digits in the number of subfiles (calculated by log10(num_subfiles) + 1)
%d
number of subfiles

yielding filenames such as:

file.h5.subfile_11273556_01_of_10
file.h5.subfile_11273556_02_of_10
file.h5.subfile_11273556_10_of_10

◆ H5FD_SUBFILING_IOC_PER_NODE

#define H5FD_SUBFILING_IOC_PER_NODE   "H5FD_SUBFILING_IOC_PER_NODE"

Macro for name of the environment variable that specifies the number of MPI ranks per node to use as I/O concentrators

The value set for this environment variable is interpreted as a long value and must be > 0.

◆ H5FD_SUBFILING_IOC_SELECTION_CRITERIA

#define H5FD_SUBFILING_IOC_SELECTION_CRITERIA   "H5FD_SUBFILING_IOC_SELECTION_CRITERIA"

Macro for name of the environment variable that provides information for selection MPI ranks as I/O concentrators

The value set for this environment variable is interpreted differently, depending on the IOC selection method chosen.

For SELECT_IOC_ONE_PER_NODE, this value is ignored.

For SELECT_IOC_EVERY_NTH_RANK, this value is interpreted as a long value and must be > 0. The value will correspond to the N value when selecting every N-th MPI rank as an I/O concentrator.

For SELECT_IOC_WITH_CONFIG, this value is ignored as that particular IOC selection method is not currently supported.

For SELECT_IOC_TOTAL, this value is interpreted as a long value and must be > 0. The value will correspond to the total number of I/O concentrators to be used.

◆ H5FD_SUBFILING_NAME

#define H5FD_SUBFILING_NAME   "subfiling"

The canonical name for the H5FD_SUBFILING driver

◆ H5FD_SUBFILING_STRIPE_SIZE

#define H5FD_SUBFILING_STRIPE_SIZE   "H5FD_SUBFILING_STRIPE_SIZE"

Macro for name of the environment variable that specifies the size (in bytes) for data stripes in subfiles

The value set for this environment variable is interpreted as a long long value and must be > 0.

◆ H5FD_SUBFILING_SUBFILE_PREFIX

#define H5FD_SUBFILING_SUBFILE_PREFIX   "H5FD_SUBFILING_SUBFILE_PREFIX"

Macro for name of the environment variable that specifies a prefix to apply to the filenames generated for subfiles

The value set for this environment variable is interpreted as a pathname.

Enumeration Type Documentation

◆ H5FD_subfiling_ioc_select_t

This enum defines the various constants to allow different allocations of MPI ranks as I/O concentrators.

Enumerator
SELECT_IOC_ONE_PER_NODE 

Default selection method. One MPI rank per node is used as an I/O concentrator. If this selection method is used, the number of I/O concentrators per node can be adjusted with the H5FD_SUBFILING_IOC_PER_NODE environment variable.

SELECT_IOC_EVERY_NTH_RANK 

Starting with MPI rank 0, a stride of 'N' is applied to the MPI rank values to determine the next I/O concentrator. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the value desired for 'N'.

SELECT_IOC_WITH_CONFIG 

Currently unsupported. Use a configuration file to determine the mapping from MPI ranks to I/O concentrators. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the path to the configuration file.

SELECT_IOC_TOTAL 

Specifies that a total of 'N' I/O concentrators should be used. Starting with MPI rank 0, a stride of 'MPI comm size' / 'N' is applied to the MPI rank values to determine the next I/O concentrator. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the value desired for 'N'.

ioc_selection_options 

Unused. Sentinel value

Function Documentation

◆ H5FD_subfiling_init()

H5_DLL hid_t H5FD_subfiling_init ( void  )

Internal routine to initialize H5FD_SUBFILING driver. Not meant to be called directly by an HDF5 application.