[Top] [Prev] [Next] [Bottom]

3.5 Creating and Writing to Simple Scientific Data Sets

This section describes the routines needed to create and write simple SDSs. A "simple" SDS is defined here as one with no attributes or user-defined dimension scales.

In the SD interface, creating and writing data to an SDS are separate operations - each performed by one SD routine. The SD interface retains no definitions about the size, contents or rank of an SDS from one SDS to the next or from one file to the next.

3.5.1 Creating Scientific Data Sets: SDcreate

Creating a simple SDS or an SDS without attributes or scales involves the following steps:

1. Open a file and initialize the SD interface.
2. Define the characteristics of the SDS.
3. Perform optional operations on the SDS.
4. Terminate access to the data set.
5. Terminate access to the SD interface and close the file.
To create an SDS, the calling program must contain the following sequence of routine calls:

C:		sd_id = SDstart(filename, access_mode);
		sds_id = SDcreate(sd_id, name, number_type, rank, dim_sizes);
		<Optional operations>
		status = SDendaccess(sds_id);
		status = SDend(sd_id);

FORTRAN:	sd_id = sfstart(filename, access_mode)
		sds_id = sfcreate(sd_id, name, number_type, rank, dim_sizes)
		<Optional operations>
		status = sfendacc(sds_id)
		status = sfend(sd_id)

SDcreate defines a new SDS using the arguments name, number_type, rank and dim_sizes. Once a data set is created, you cannot change its name, data type or rank. SDcreate does not actually perform the write; it occurs only when SDend is called.

The SD interface will assign a name if one is not provided. In this situation, the filename parameter must be a null character string. The maximum length of an SDS name is defined by MAX_NC_NAME and the maximum rank of an SDS array is defined by MAX_VAR_DIMS. Both are defined in the "netcdf.h" header file. Tag names passed in the number_type parameter are prefaced by "DFNT".

When creating an SDS, it is necessary to specify the data type of the array data contained in the SDS. The "hntdefs.h" header file contains definitions of all valid data types, which are described in Chapter 2, titled HDF Fundamentals.

TABLE 3C SDcreate Parameter List
Routine Name

(Fortran-77)

Parameter

Data Type

Description

C

Fortran-77

SDcreate

(sfcreate)

file_id

int32

integer

File identifier.

name

char *

character* (*)

ASCII string containing the name of the data set.

data_type

int32

integer

Data type of the data set.

rank

int32

integer

Number of dimensions in the array.

dim_sizes

int32 []

integer (*)

Array defining the size of each dimension.

EXAMPLE 2. Creating an Empty SDS

If SDcreate is called but not written, an empty array is created. An "empty array" is an array that has been defined but not yet initialized with data. All array information passed into SDcreate through its parameters are stored in the file.

C:

#include "hdf.h"
#include "mfhdf.h"

#define X_LENGTH 5
#define Y_LENGTH 16

main( ) 
{

	int32 sd_id, sds_id, status;
	int32 dimsizes[2], rank;

	/* Create and open the file and initiate the SD interface. */
	sd_id = SDstart("Example2.hdf", DFACC_CREATE);

	/* Define the rank and dimensions of the array to be created. */
	rank = 2;
	dimsizes[0] = Y_LENGTH;
	dimsizes[1] = X_LENGTH;

	/* Create the array. */
	sds_id = SDcreate(sd_id, "Ex_array_1", DFNT_INT16, rank, dimsizes);

	/* Terminate access to the array. */
	status = SDendaccess(sds_id);

	/* Terminate access to the SD interface and close the file */
	status = SDend(sd_id);

}

FORTRAN:

	PROGRAM EMPTY ARRAY

      integer*4 sd_id, sds_id, dimsizes(2), rank
      integer sfstart, sfcreate, sfendacc, sfend
      integer*4 X_LENGTH, Y_LENGTH
      parameter (X_LENGTH = 16, Y_LENGTH = 5)

C     DFACC_CREATE and DFNT_INT16 are defined in hdf.h.
      integer*4 DFACC_CREATE, DFNT_INT16
      parameter (DFACC_CREATE = 4, DFNT_INT16 = 22)

C     Create and open the file and initiate the SD interface.
      sd_id = sfstart('Example2.hdf', DFACC_CREATE)

C     Define the rank and dimensions of the array to be created.
      rank = 2
      dimsizes(1) = Y_LENGTH
      dimsizes(2) = X_LENGTH

C     Create the array.
      sds_id = sfcreate(sd_id, 'Ex_array_1', DFNT_INT16, rank, 
     +                  dimsizes)

C     Terminate access to the array.
      status = sfendacc(sds_id)

C     Terminate access to the SD interface and close the file.
      status = sfend(sd_id)

      end

3.5.2 Writing Data to an SDS Array: SDwritedata

SDwritedata is the only routine that is needed to write data to an SDS array. It can completely or partially fill an SDS n-dimensional array or append data along one dimension defined to be of unlimited length. It can also skip a specified number of SDS array elements between write operations along each dimension.

Creating an SDS and writing one or more slabs to it involves the following steps:

1. Create an SDS.
2. Write a slab or series of slabs.
To do this, the calling program must contain the following sequence of routine calls:

C:		sds_id = SDcreate(sd_id, name, number_type, rank, dim_sizes);
		status = SDwritedata(sds_id, start, stride, edge, data);

FORTRAN:	sds_id = sfcreate(sd_id, name, number_type, rank, dim_sizes)
		status = sfwdata(sds_id, start, stride, edge, data)

A slab is an n-dimensional array whose dimensions are smaller than or equal to those of the SDS array into which it will be written. A slab is defined by the n-dimensional coordinate of its initial vertex and the lengths of each dimension.

SDwritedata takes five arguments: sds_id, start, stride, edge, and data. The sds_id argument is the SDS identifier returned by SDcreate or SDselect. The arguments start, stride, and edge respectively describe the n-dimensional coordinates the SD interface will begin the write operation in the data set, the number of locations the current SDS location will be moved forward after each write, and the length of each dimension of the n-dimensional slab to be written. If the SDS array is smaller than the data argument array, the amount of data written will be limited to the maximum size of the SDS array.

The argument start is an array specifying the location in the data set array the write operation will begin. The indices are relative to 0 so the first data value of an array would have index {0,0, ... 0}. The size of start must be the same as the number of dimensions in the SDS array. In addition, each value in start must be smaller than its corresponding SDS array dimension unless the dimension is unlimited. Violating any of these conditions causes a termination of the write operation and causes an error condition to be generated.

The argument stride is an array specifying, for each dimension, the interval between values to be written. For example, setting stride[0]=2 writes data to every other location along the first dimension.(See Figure 3b on page 30.) Setting stride[0]=1 writes data to every location along the first dimension. If stride is defined as NULL in C or 0 in Fortran-77, it is assumed to contain all ones. For better performance, is it advised that you define the value of stride as NULL rather than setting it equal to 1. The length of the stride array must be the same as the number of dimensions in the SDS array. Also, each value in stride must be smaller than or equal to its corresponding SDS array dimension unless the dimension is unlimited. A violation of any of these conditions terminates the write operation and causes an error condition to be generated.

FIGURE 3b Description of "Strides"

The argument edge is an array specifying the length of each dimension of the slab to be written. If the rank of the slab is less than that of the SDS data set, all additional dimensions must be specified as 1. Each value in the edge array must not be larger than the length of the corresponding dimension in the SDS data set. Attempting to read or write slabs larger than the size of the SDS data set will result in an error condition. The size of edge must be equal to the number of dimensions in the SDS array. In addition, the sum of each value in the edge array and the corresponding value in the start array must be smaller than or equal to its corresponding SDS array dimension unless the dimension is unlimited. A violation of any of these conditions terminates the write operation and results in an error condition.

Be aware that the mapping between the dimensions of a slab and the order in which the slab values are stored in memory is different between C and Fortran-77. In C the values are stored with the assumption that the last dimension of the slab varies fastest (or "row-major order" storage), but in Fortran-77 the first dimension varies fastest (or "column-major order" storage). These storage order conventions can cause some confusion when data written by a C program is read by a Fortran-77 program or vice versa.

There are two Fortran-77 versions of this routine: sfwdata and sfwcdata. The sfwdata routine writes numeric scientific data and sfwcdata writes character scientific data.

The parameters of SDwritedata are described in the following table. Note that, because there are two Fortran-77 versions of SDwritedata, there are correspondingly two entries in the "Data Type" field of the data parameter.

TABLE 3D SDwritedata Parameter List
Routine Name

(Fortran-77)

Parameter

Data Type

Description

C

Fortran-77

SDwritedata

(sfwdata/

sfwcdata)

sds_id

int32

integer

Data set identifier.

start

int32 []

integer (*)

Array containing the position the write will start for each dimension.

stride

int32 []

integer (*)

Array containing the number of data locations the current location is to be moved forward before the next write.

edge

int32 []

integer (*)

Array containing the number of data elements that will be written along each dimension.

data

VOIDP

<valid numeric data type>

Buffer for the data to be written.

3.5.2.1 Filling an Entire Array

Filling an array is a simple slab operation where the slab begins at the origin of the SDS array and fills every location in the array. SDwritedata will fill an SDS array with data when given the origin (start={0,0, ... 0}) as its starting coordinates, a stride value of NULL, and edge dimensions equal to the size of the SDS array (edge={dim_sizes[0], dim_sizes[1], ... dim_size[rank-1]}).

EXAMPLE 3. Creating and Writing to an SDS

These examples use SDcreate under the C interface and sfcreate under the Fortran-77 interface. The only difference between writing array data to an existing SDS and writing it to a newly created array is the use of SDselect instead of SDcreate.

C:

#include "hdf.h"
#include "mfhdf.h"

#define X_LENGTH 5
#define Y_LENGTH 16

main( ) 
{
	int32 sd_id, sds_id, status;
	int32 dims[2], start[2], edges[2], rank;
	int16 array_data[Y_LENGTH][X_LENGTH];
	intn i, j;

	/* Create and open the file and initiate the SD interface. */
	sd_id = SDstart("Example3.hdf", DFACC_CREATE);

	/* Define the rank and dimensions of the data set to be created. */
	rank = 2;
	dims[0] = Y_LENGTH;
	dims[1] = X_LENGTH;

	/* Create the array data set. */
	sds_id = SDcreate(sd_id, "Ex_array_3", DFNT_INT16, rank, dims);

	/* Fill the stored-data array with values. */
	for (j = 0; j < Y_LENGTH; j++) {
		for (i = 0; i < X_LENGTH; i++)
			array_data[j][i] = (i + j) + 1;
	}

	/* Define the location, pattern, and size of the data set */
	for (i = 0; i < rank; i++) {
		start[i] = 0;
		edges[i] = dims[i];
	}

	/* Write the stored data to the "Ex_Array_3" data set. The fifth 
	* argument must be explicitly cast to a generic pointer to conform 
	* to the HDF API definition for SDwritedata. */
	status = SDwritedata(sds_id, start, NULL, edges, (VOIDP)array_data);

	/* Terminate access to the array. */
	status = SDendaccess(sds_id);

	/* Terminate access to the SD interface and close the file. */
	status = SDend(sd_id);

}

FORTRAN:

	PROGRAM FILLED ARRAY

      integer*4 sd_id, sds_id, rank
      integer dims(2), start(2), edges(2), stride(2), status
      integer*2 i, j
      integer sfstart, sfcreate, sfwdata, sfendacc, sfend

C     DFACC_CREATE and DFNT_INT16 are defined in hdf.h.
      integer*4 DFACC_CREATE, DFNT_INT16
      integer*4 X_LENGTH, Y_LENGTH
      parameter (DFACC_CREATE = 4, DFNT_INT16 = 22, X_LENGTH = 5,
     +           Y_LENGTH = 16)
      integer*2 array_data(X_LENGTH, Y_LENGTH)

C     Create and open the file and initiate the SD interface.
      sd_id = sfstart('Example3.hdf', DFACC_CREATE)

C     Define the rank and dimensions of the data set to be created.
      rank = 2
      dims(1) = X_LENGTH
      dims(2) = Y_LENGTH

C     Create the data set.
      sds_id = sfcreate(sd_id, 'Ex_array_3', DFNT_INT16, rank, dims)

C     Fill the stored-data array with values.
      do 20 j = 1, Y_LENGTH
         do 10 i = 1, X_LENGTH
            array_data(i, j) = i + j - 1
10         continue
20    continue

C     Define the location, pattern, and size of the data set
C     that will be written to.
      start(1) = 0
      start(2) = 0
      edges(1) = X_LENGTH
      edges(2) = Y_LENGTH
      stride(1) = 1
      stride(2) = 1

C     Write the stored data to the "Ex_array_3" data set.
      status = sfwdata(sds_id, start, stride, edges, array_data)

C     Terminate access to the array.
      status = sfendacc(sds_id)

C     Terminate access to the SD interface and close the file.
      status = sfend(sd_id)

      end

3.5.2.2 Writing Slabs to an SDS Array

To allow preexisting data to be modified, the HDF library does not prevent SDwritedata from overwriting one slab with another. As a result, the calling program is responsible for managing any overlap when writing slabs. The HDF library will issue an error if a slab extends past the valid boundaries of the SDS data set, although appending data along an unlimited dimension is allowed.

EXAMPLE 4. Writing a Slab of Data to an SDS

These examples show how to write a series of slabs to a data set. The programs create a three-dimensional array with the size of the x-dimension being four elements, the y-dimension five elements and the z-dimension six elements. As the data set elements are written, the slabs are "sliced" along the y-dimension.

Note that the start, edge and stride arguments in SDwritedata refer to the coordinate locations in the file representation of the data set, not the location within the write_data array. Therefore, the actual slab to be written is first buffered in the zx_data array.

In most real-world HDF-based applications that perform slab writes, the write_data array would be a buffer area for data previously read in by SDreaddata. In this example, this data read step has been omitted so that this example can focus on the slab write procedure.

C:

#include "hdf.h"
#include "mfhdf.h"

#define X_LENGTH 4
#define Y_LENGTH 5
#define Z_LENGTH 6

main( ) 
{
	int32 sd_id, sds_id, rank, status;
	int32 dims[3], start[3], edges[3];
	int16 write_data[Z_LENGTH][Y_LENGTH][X_LENGTH];
	int16 zx_data[Z_LENGTH][X_LENGTH];
	intn i, j, k;

	/* Open the file. */
	sd_id = SDstart("Example4.hdf", DFACC_CREATE);

	/* Define the rank and dimensions of the array to be created. */
	rank = 3;
	dims[0] = Z_LENGTH;
	dims[1] = Y_LENGTH;
	dims[2] = X_LENGTH;

	/* Create the array. */
	sds_id = SDcreate(sd_id, "Ex_array_4", DFNT_INT16, rank, dims);

	/* Compute and store the values that will be written to the data 
	   set. Fill the write_data array along the x-axis first. */
	for (k = 0; k < Z_LENGTH; k++)
		for (j = 0; j < Y_LENGTH; j++)
			for (i = 0; i < X_LENGTH; i++)
				write_data[k][j][i] = (i + 1) + (j + 1) + (k + 1);

	/* Within each for loop, set the start and edge parameters to write \ 
	a 4-by-6 element slab of stored data from the write_data array to \ 
	the data set. */
	edges[0] = Z_LENGTH;
	edges[1] = 1;
	edges[2] = X_LENGTH;
	start[0] = start[2] = 0;

	for (j = Y_LENGTH; j > 0; j--) {
    		start[1] = j - 1;
    		for ( k = 0; k < Z_LENGTH; k++) {
    			for ( i = 0; i < X_LENGTH; i++) {
    				zx_data[k][i] = write_data[k][j-1][i];
    				status = SDwritedata(sds_id, start, NULL, edges,
    							         (VOIDP)zx_data);
                }
            }
	}

	/* Terminate access to the data set. */ 
	status = SDendaccess(sds_id);
    
	/* Terminate access to the SD interface and close the file. */
	status = SDend(sd_id);
    
}

FORTRAN:

	PROGRAM WRITE SLAB

      integer*4 sd_id, sds_id, rank
      integer dims(3), start(3), edges(3), stride(3)
      integer i, j, k, status
      integer sfstart, sfcreate, sfwdata, sfendacc, sfend

C     DFACC_CREATE and DFNT_INT16 are defined in hdf.h.
      integer*4 DFACC_CREATE, DFNT_INT16
      integer*4 X_LENGTH, Y_LENGTH, Z_LENGTH
      parameter (DFACC_CREATE = 4, DFNT_INT16 = 22, X_LENGTH = 4,
     +           Y_LENGTH = 5, Z_LENGTH = 6)

      integer*2 write_data(X_LENGTH, Y_LENGTH, Z_LENGTH)
      integer*2 zx_data(X_LENGTH, Z_LENGTH)

C     Create the file.
      sd_id = sfstart('Example4.hdf', DFACC_CREATE)

C     Define the rank and dimensions of the array to be created. 
      rank = 3
      dims(1) = X_LENGTH   
      dims(2) = Y_LENGTH   
      dims(3) = Z_LENGTH   

C     Create the array. 
      sds_id = sfcreate(sd_id, 'Ex_array_4', DFNT_INT16, rank, dims)

C     Compute and store the values that will be later written to the 
C     selected array data set. Fill the array_data array along the 
C     x-axis first.
      do 30 k = 1, Z_LENGTH
         do 20 j = 1, Y_LENGTH
            do 10 i = 1, X_LENGTH
               write_data(i, j, k) = i + j + k 
10            continue
20         continue
30    continue

C     Within each do loop, set the start and edge parameters so that 
C     a 4-by-6 element slab of stored data will be written from the 
C     array_data array to the data set. 
      edges(1) = X_LENGTH
      edges(2) = 1
      edges(3) = Z_LENGTH
      start(1) = 0 
      start(3) = 0
      stride(1) = 1
      stride(2) = 1
      stride(3) = 1

      do 60 j = Y_LENGTH, 0, -1
       start(2) = j - 1
       do 50 k = 1, Z_LENGTH
        do 40 i = 1, X_LENGTH
         zx_data(i, k) = write_data(i, j, k)
         status = sfwdata(sds_id, start, stride, edges, zx_data)
40      continue
50     continue
60    continue

C     Terminate access to the data set. 
      status = sfendacc(sds_id)

C     Terminate access to the SD interface and close the file. 
      status = sfend(sd_id)

      end

EXAMPLE 5. Altering Values Within an Array Data Set

These examples demonstrate the procedure for changing the value of one element, located in the second column and tenth row, of a data set.

This procedure can be used to alter the values of a group of elements within the data set by changing the start and edge arguments of the second call of the SDwritedata function. Notice that a NULL value is passed into SDwritedata in the C program instead of the stride array.

C:

#include "hdf.h"
#include "mfhdf.h"

#define X_LENGTH 5
#define Y_LENGTH 16

main( ) 
{
	int32 sd_id, sds_id, status;
	int32 start[2], edges[2];
	int16 write_data[Y_LENGTH][X_LENGTH], intval;
	intn i, j;

	/*	Open the file. */
	sd_id = SDstart("Example3.hdf", DFACC_RDWR);

	/*	Select the first data set. */
	sds_id = SDselect(sd_id, 0);

	/* Compute and store the values that will be written to the
	   selected array data set. */
	for (j = 0; j < Y_LENGTH; j++)
		for (i = 0; i < X_LENGTH; i++)
			write_data[j][i] = (i + 1) + (j + 1) * 10;

	/* Set up the start and edge parameters to write the buffered
	   data to the entire array data set. */
	start[0] = start[1] = 0;
	edges[0] = Y_LENGTH;
	edges[1] = X_LENGTH;

	/* Write the buffered data to the "Ex_array_3" data set. The fifth 
	   argument must be explicitly cast to a generic pointer to conform 
	   to the HDF API definition of SDwritedata.*/
	status = SDwritedata(sds_id, start, NULL, edges, (VOIDP)write_data);

	/* Alter a value within this data set. */
	intval = 15;
	start[0] = 10;
	start[1] = 1;
	edges[0] = 1;
	edges[1] = 1;
	status = SDwritedata(sds_id, start, NULL, edges, (VOIDP)&intval);

	/* Terminate access to the data set. */ 
	status = SDendaccess(sds_id);

	/* Terminate access to the SD interface and close the file. */
	status = SDend(sd_id);

}

FORTRAN:

	PROGRAM ALTER DATA

      integer*4 sd_id, sds_id
      integer start(2), edges(2), stride(2)
      integer i, j, status
      integer sfstart, sfselect, sfwdata, sfendacc, sfend

C     DFACC_RDWR is defined in hdf.h.
      integer*4 DFACC_RDWR
      integer*4 X_LENGTH, Y_LENGTH
      parameter (DFACC_RDWR = 3, X_LENGTH = 5, Y_LENGTH = 16) 

      integer*2 array_data(X_LENGTH, Y_LENGTH), intval

C     Open the file and initiate the SD interface.
      sd_id = sfstart('Example3.hdf', DFACC_RDWR)

C     Select the first data set.
      sds_id = sfselect(sd_id, 0)

C     Compute and store the values that will be later written to 
C     the data set.
      do 20 j = 1, Y_LENGTH
         do 10 i = 1, X_LENGTH
            array_data(i, j) = i + j * 10
10         continue
20    continue

C     Initialize the start, edge and stride parameters to write the
C     stored data to the data set.
      start(1) = 0
      start(2) = 0
      edges(1) = X_LENGTH
      edges(2) = Y_LENGTH
      stride(1) = 1
      stride(2) = 1

C     Write the stored data to the data set.
      status = sfwdata(sds_id, start, stride, edges, array_data)

C     Alter the value of the data set element in the first column,
C     tenth row to be 15.
      intval = 15
      start(1) = 1
      start(2) = 10
      edges(1) = 1
      edges(2) = 1
      status = sfwdata(sds_id, start, stride, edges, intval)

C     Terminate access to the data set.
      status = sfendacc(sds_id)

C     Terminate access to the SD interface and close the file.
      status = sfend(sd_id)

      end

3.5.2.3 Appending Data to an SDS Array Using the Unlimited Dimension

If one dimension of an SDS array must be extendable, it is possible to create an appendable SDS array. An SDS array is appendable if one dimension is specified as an "unlimited" when the array is created.

In C, if the SDcreate parameter dim_sizes[0] is assigned the value SD_UNLIMITED it is considered to be an unlimited dimension. This is the only dimension that may be specified as unlimited in a C program and can also be understood as the first dimension or the dimension of the lowest rank value, of the SDS. In Fortran-77, only the last dimension or the dimension of the highest rank value, can be unlimited. In other words, in Fortran-77 dim_sizes(rank) must be set to the value SD_UNLIMITED to make the array appendable.

To append data to an extensible data set without overwriting data, specify the appropriate start coordinates in the SDwritedata routine. For example, if the current coordinate boundary of an unlimited dimension is defined as 15, appending data to the array without overwriting existing data requires a start coordinate of 16. To append data by overwriting data, specify a start coordinate less than the current boundary of the unlimited dimension. In either case, all but the unlimited coordinate in start must fall within the boundaries of the original array definition.

Any time a write operation is attempted beyond the current boundary, the HDF library will automatically adjust the dimension record to the new length. If the new data both begins and ends past the boundary of the array, locations between the existing boundary and the beginning of the new data stream are initialized to the assigned fill value if there is one or the default fill value if none is assigned.

EXAMPLE 6. Appending Data to an SDS Array Using an Unlimited Dimension

In the C example, the length of the SDS array's y-dimension (or the first dimension) is set to the value SD_UNLIMITED, which defines the x-dimension (or the second and last dimension) as appendable. In the Fortran-77 version, the length of the y-dimension (or the second and last dimension) is set to the value SD_UNLIMITED. Again, this is because only the dimension of the highest rank (in this case the x-dimension is of rank 1 and the y-dimension is of rank 2) can be defined as appendable under the Fortran-77 interface.

C:

#include "hdf.h"
#include "mfhdf.h"

#define X_LENGTH 10
#define Y_LENGTH 10

main( ) 
{

	int32 sd_id, sds_id, sds_idx;
	int32 dims[2], rank;
	int16 array_data[Y_LENGTH][X_LENGTH], append_data[X_LENGTH];
	int32 start[2], edges[2];
	intn i = 0, j, status;

	/* Open the file and initiate the SD interface. */
	sd_id = SDstart("Example3.hdf", DFACC_RDWR);

	/* Define the rank and dimensions of the array. Make the first \
	   array dimension appendable by defining it's length to be \
	   unlimited.*/
	rank = 2;
	dims[0] = SD_UNLIMITED;
	dims[1] = X_LENGTH;

	/* Create the array data set. */
	sds_id = SDcreate(sd_id, "Ex_File_6", DFNT_INT16, rank, dims);

	/* Store the array values. */
	for (j = 0; j < Y_LENGTH; j++) {
		for (i = 0; i < X_LENGTH; i++)
			array_data[j][i] = (i + 1) + (j + 1);
	}

	/* Write the data to the array. */
	start[0] = start[1] = 0;
	edges[0] = Y_LENGTH;
	edges[1] = X_LENGTH;

	/* Perform the initial write to the array data set. */
	status = SDwritedata(sds_id, start, NULL, edges, (VOIDP)array_data);

	/* Terminate access to the array data set, terminate access \
	to the SD interface and close the file. */
	status = SDendaccess(sds_id);
	status = SDend(sd_id);

	/* Reopen the file and initiate the SD interface in preparation 
	for appending data to the data set. Then select the first 
	data set.*/
	sd_id = SDstart("Example3.hdf", DFACC_RDWR);
	sds_idx = SDnametoindex(sd_id, "Ex_File_6");
	sds_id = SDselect(sd_id, sds_idx);

	/* Store the array values to be appended to the data set. */
	for (i = 0; i < X_LENGTH; i++)
		append_data[i] = i + 1;

	/* Define the location of the append to start at the first column \
	of the sixteenth row of the data set and to stop at the end of the \
	fifth row. */
	start[0] = Y_LENGTH;
	start[1] = 0;
	edges[0] = 1;
	edges[1] = X_LENGTH;

	/* Append the stored data to the array data set. */
	status = SDwritedata(sds_id, start, NULL, edges, (VOIDP)append_data);

	/* Terminate access to the array data set. */
	status = SDendaccess(sds_id);

	/* Terminate access to the SD interface and close the file. */
	status = SDend(sd_id);

}

FORTRAN:

	PROGRAM APPEND DATA

      integer*4 sd_id, sds_id, sds_idx
      integer dims(2), rank
      integer start(2), edges(2), stride(2)
      integer status
      integer sfstart, sfcreate, sfwdata, sfn2index, sfselect 
      integer sfendacc, sfend

C     DFACC_RDWR and DFNT_INT16 are defined in hdf.h, SD_UNLIMITED
C     is defined in hdf.h.
      integer*4 DFACC_RDWR, DFNT_INT16, SD_UNLIMITED
      integer*4 X_LENGTH, Y_LENGTH
      parameter (DFACC_RDWR = 3, DFNT_INT16 = 22, SD_UNLIMITED = 0,
     +           X_LENGTH = 10, Y_LENGTH = 10)

      integer*2 array_data(X_LENGTH, Y_LENGTH)
      integer*2 append_data(X_LENGTH)
      integer*2 i, j

C     Open the file and initiate the SD interface. 
      sd_id = sfstart('Example3.hdf', DFACC_RDWR)

C     Define the rank and dimensions of the array. Make the
C     last dimension appendable by defining it's length as unlimited.
      rank = 2
      dims(1) = X_LENGTH
      dims(2) = SD_UNLIMITED

C     Create the array data set. 
      sds_id = sfcreate(sd_id, 'Ex_File_6', DFNT_INT16, rank, dims)

C     Store the array values. 
      do 20 j = 1, Y_LENGTH
         do 10 i = 1, X_LENGTH
            array_data(i, j) = i + j
10      continue
20    continue

C     Write the data to the array.
      start(1) = 0
      start(2) = 0
      edges(1) = X_LENGTH
      edges(2) = Y_LENGTH
      stride(1) = 1
      stride(2) = 1

C     Perform the initial write to the data set. 
      status = sfwdata(sds_id, start, stride, edges, array_data)

C     Terminate access to the data set, terminate access
C     to the SD interface and close the file. 
      status = sfendacc(sds_id)
      status = sfend(sd_id)

C     Reopen the file and initiate the SD interface in preparation
C     for appending data. Then select the first data set.
      sd_id = sfstart('Example3.hdf', DFACC_RDWR)
      sds_idx = sfn2index(sd_id, 'Ex_File_6')
      sds_id = sfselect(sd_id, sds_idx)

C     Store the array values to be appended to the data set. 
      do 30 i = 1, X_LENGTH
         append_data(i) = i 
30    continue

C     Define the location of the append to start at the first
C     column of the third row and to stop at the end of the third row.
      start(1) = 0
      start(2) = Y_LENGTH
      edges(1) = X_LENGTH
      edges(2) = 1

C     Append the stored data to the data set. 
      status = sfwdata(sds_id, start, stride, edges, append_data)

C     Terminate access to the array data set. 
      status = sfendacc(sds_id)

C     Terminate access to the SD interface and close the file. 
      status = sfend(sd_id)

      end

3.5.3 Compressing SD SDS Array Data: SDsetcompress

Uncompressed SDS array data is compressed, or new compressed data sets are created, by calling the SDsetcompress routine. SDsetcompress compresses the data set data at the time it is called (not during the next call to SDwritedata)and supports all standard HDF compression algorithms. These algorithms are:

In the future, the following algorithms may be included: Lempel/Ziv-78 dictionary coding, an arithmetic coder, and a faster Huffman algorithm.

The SDsetcompress routine is a simplified interface to the HCcreate routine, and should be used instead of HCcreate unless the user is familiar with working with the lower-level routines. All essential compression functionality is provided by SDsetcompress.

The syntax of the SDsetcompress routine is as follows:

C:	status = SDsetcompress(sds_id, comp_type, c_info);

The comp_type parameter is the compression type definition and is set to COMP_CODE_JPEG for JPEG compression, COMP_CODE_RLE for run-length encoding, COMP_CODE_DEFLATE for Gnu ZIP (or GZIP) compression, COMP_CODE_SKPHUFF for skipping Huffman or COMP_CODE_NONE for no compression.

The c_info parameter is a pointer to a union structure of type coder_info.If comp_type is set to COMP_CODE_NONE or COMP_CODE_RLE, this is unused and can be set to NULL. If it's set to COMP_CODE_SKPHUFF, the skphuff structure in the coder_info union must be provided information about the size, in bytes, of the data elements. If it's set to COMP_CODE_DEFLATE, the deflate structure in the coder_info union must be provided information about the compression "effort".

For example, to compress unsigned 16-bit integer data using the adaptive Huffman algorithm, the following definition and SDsetcompress call is used.


coder_info 	c_info;
c_info.skphuff.skp_size = sizeof(uint16);
status = SDsetcompress(sds_id, COMP_CODE_SKPHUFF, &c_info);

To compress a data set using the gzip deflation algorithm, with the maximum "effort" level designated, the following definition and SDsetcompress call is used.


coder_info 	c_info;
c_info.deflate_level = 9;
status = SDsetcompress(sds_id, COMP_CODE_DEFLATE, &c_info);

To compress a data set using the JPEG algorithm, with the quality level of 80 designated and the "force baseline" parameter enabled, the following definition and SDsetcompress call is used.


coder_info 	c_info;
c_info.quality = 80;
c_info.force_baseline = TRUE;
status = SDsetcompress(sds_id, COMP_CODE_JPEG, &c_info);

SDsetcompress functionality is currently limited to creating new datasets or appending new slabs onto existing datasets. Overwriting existing dataset data will be supported in the future.

Note that there is currently no Fortran-77 version of the SDsetcompress routine. It will be included in a future release.

TABLE 3E SDsetcompress Parameter List
Routine Name

Parameter

Data Type

Description

C

SDsetcompress

sds_id

int32

Data set identifier.

comp_type

char *

Compression method.

cinfo

comp_info *

Pointer to compression information structure.

3.5.3.1 Rules for Writing to a Compressed Data Set

Due to certain limitations in the way that compressed data sets are stored, they aren't writable in the way that uncompressed data sets are. The "rules" for writing to a compressed data set are the following:

1. Write the compressed data, in its entirety, to the data set. Build the data set in-core, then write it to the data set in a single write operation.
2. Append to a compressed data set. In other words, write to a compressed data set along it's unlimited dimension. If an unlimited dimension hasn't been defined for the data set, it can't be appended to in this way.
3. Write the compressed data, in its entirety, to any chunk in a chunked SDS.
These rules imply that it is impossible to overwrite subsets of non-chunked data sets. This is because the existing compression algorithms supported by HDF don't allow partial modification to a compressed datastream.

3.5.4 External File Operations

An external SDS array is an array stored in a file separate from the file containing the metadata for the array. With external arrays, it is possible to link data sets in the same HDF file to multiple external files or data sets in different HDF files to the same external file. Routines for writing external SDS arrays are only available in the C interface and can only be used with HDF files. Unidata-formatted netCDF files are not supported by these routines.

External arrays are functionally identical to arrays in the primary data file. During slab operations, the HDF library keeps track of the beginning of the data set and adds slabs at the appropriate position in the external file. When data is written or appended along a specified dimension, the HDF library writes along that dimension in the external file and updates the appropriate dimension record in the primary file.

There are two methods for creating external SDS arrays. The user can create a new data set in an external file or move data from an existing internal data set to an external file. In either case, only the array values are stored externally, all other data set information remains in the primary HDF file. When an external array is created, a sufficient amount of space is reserved in the external file for the entire data set. The data set will begin at the specified byte offset and extend the length of the data set. The write operation will overwrite the target locations in the external file. The external file may be of any format, provided the data types, byte ordering, and dimension ordering are supported by HDF. The primary file must be an HDF file.

3.5.4.1 Specifying the Directory Search Path of an External File: HXsetdir

There are three filesystem locations the HDF external file routines check when determining the location of an external file. They are, in order of precedence:

1. The directory path specified by the last call to the HXsetdir routine.
2. The directory path specified by the $HDFEXTDIR shell environment variable.
3. The locations searched by the standard open(3) routine.
Until the HXsetdir routine is called, either the directories specified by the $HDFEXTDIR environment variable are searched if the variable has been set, or Item 3 will be the search method used. If the HXsetdir routine hasn't been called and the $HDFEXTDIR variable hasn't been set, Item 3 will again be the search method used. Setting the $HDFEXTDIR environment variable effectively ensures that Item 3 will never be used as a default method as it can't be unset within a calling program.

HXsetdir has one argument, a string specifying the directory list to be searched. This list can consist of one directory name or a set of directory names separated by colons. If HXsetdir is passed a null string and the $HDFEXTDIR environment variable has been set, the directories specified by $HDFEXTDIR are searched, or Item 3 will be the search method used if $HDFEXTDIR hasn't been set.

If an error condition is encountered, HXsetdir leaves the directory search path unchanged. The directory search path specified by HXsetdir remains in effect throughout the scope of the calling program.

The parameters of HXsetdir are described further in the following table.

TABLE 3F HXsetdir Parameter List
Routine Name

(Fortran-77)

Parameter

Data Type

Description

C

Fortran-77

HXsetdir

(hxsdir)

dir_list

char *

character* (*)

Directory list to be searched.

dir_length

None

integer

Length of the dir_list string.

3.5.4.2 Specifying the Location of the Next External File to be Created: HXsetcreatedir

HXsetcreatedir specifies the directory location of the next external file to be created. It overrides the directory location specified by the $HDFEXTCREATEDIR and the locations searched by the open(3) call in the same manner as HXsetdir. Specifically, the precedence is:

1. The directory specified by the last call to the HXsetcreatedir routine.
2. The directory specified by the $HDFEXTCREATEDIR shell environment variable.
3. The locations searched by the standard open(3) routine.
HXsetcreatedir has one argument: the directory location of the next external file to be created. If an error is encountered, the directory location is left unchanged.

The parameters of HXsetcreatedir are described further in the following table.

TABLE 3G HXsetcreatedir Parameter List
Routine Name

(Fortran-77)

Parameter

Data Type

Description

C

Fortran-77

HXsetcreatedir

(hxsdir)

dir

char *

character* (*)

Directory location of the next external file to be created.

dir_length

None

integer

Length of the dir string.

3.5.4.3 Creating a Data Set in an External File: SDsetexternalfile

Creating a data set using external data involves the following steps:

1. Create the array.
2. Specify that an external data file is to be used.
3. Write data to the array.
4. Terminate access to the data set.
To create a data set containing an external file, the calling program must make the following calls.

C:		sds_id = SDcreate(sd_id, name, number_type, rank, dim_sizes);
		status = SDsetexternalfile(sds_id, filename, offset);
		status = SDwritedata(sds_id, start, stride, edge, data);
		status = SDendaccess(sds_id);
FORTRAN:	sds_id = sfcreate(sd_id, name, number_type, rank, dim_sizes)
		status = sfsextf(sds_id, filename, offset)
		status = sfwdata(sds_id, start, stride, edge, data)
		status = sfendacc(sds_id)

SDsetexternalfile marks the SDS identified by sds_id as one whose data is to be written to an external file. The parameter filename is the name of the external data file, and offset is the number of bytes from the beginning of the external file to the location where the first byte of data should be written.

When used in conjunction with SDcreate, SDsetexternalfile does not actually write data to an external file. Instead it marks the data set as an external data set for all subsequent SDwritedata operations. SDsetexternalfile can only be called once after a data set has been created.

If a file with the same name as filename exists in the current directory search path, HDF will access it as the external file. If the file does not exist, HDF will create one in the directory named in the last call to HXsetcreatefile. If an absolute pathname is specified, the external file will be created at the location specified by the pathname, overriding the location specified by the last call to HXsetcreatefile.

Once the name of an external file is established, it is impossible to change it without breaking the association between the data set's metadata and the data it describes.

Use caution when writing to existing external or primary files as the HDF library starts the write operation at the specified offset without checking if data is being overwritten.

For more information on the parameters used in SDsetexternalfile refer to the following table.

TABLE 3H SDsetexternalfile Parameter List
Routine Name

(Fortran-77)

Parameter

Data Type

Description

C

Fortran-77

SDsetexternalfile

(sfsextf)

sds_id

int32

integer

Data set identifier.

filename

char *

character* (*)

Name of the external data set.

offset

int32

integer

Offset in bytes from the beginning of the external file to the SDS data.

3.5.4.4 Moving Data to an External File

Data can be moved from a primary file to an external file. To do so requires the following steps:

1. Select the array.
2. Specify the external data file.
3. Terminate access to the data set.
To move data to an external file, the calling program must make the following calls:

C:		sds_id = SDselect(sd_id, sds_index);
		status = SDsetexternalfile(sds_id, filename, offset);
		status = SDendaccess(sds_id);

FORTRAN:	sds_id = sfcreate(sd_id, name, number_type, rank, dim_sizes)
		status = sfsextf(sds_id, filename, offset)
		status = sfendacc(sds_id)

When SDsetexternalfile is used in conjunction with SDselect, it will immediately write the existing data to the external file. Any data in the external file that occupies the space reserved for the external array will be overwritten as a result of this operation. Data can only be moved to an external file once after the parent data set has been created and the data set must first exist in the primary HDF file. During the operation, the data is written to the external file as a contiguous stream regardless of how it is stored in the primary file. Because data is moved "as is", any unwritten locations in the data set are preserved in the external file. Subsequent read and write operations performed on the data set will access the external file.

EXAMPLE 7. Writing SDS Data to an HDF File Starting at an Offset

A subset of values stored in an SDS can be moved from an HDF file to an external file, as shown in the following examples.

C:

#include "hdf.h"

main( ) 
{

	int32 sd_id, sds_id, offset, status;

	/* Open the file. */
	sd_id = SDstart("Example3.hdf", DFACC_RDWR);

	/* Get the identifier for the first data set. */
	sds_id = SDselect(sd_id, 0);

	/* Create a file named "subfile1" and move the data set values into
	 *	it, starting at byte location 24. */
	offset = 24;
	status = SDsetexternalfile(sds_id, "subfile1", offset);

	/* Terminate access to the data set, SD interface and file. */
	status = SDendaccess(sds_id);
	status = SDend(sd_id);

}

FORTRAN:

	PROGRAM WRITE EXTFILE

      integer*4 sd_id, sds_id, offset, DFACC_RDWR
      integer status
      integer sfstart, sfselect, sfsextf, sfendacc, sfend

C     DFACC_RDWR is defined in hdf.h.
      parameter (DFACC_RDWR = 3)

C     Open the HDF file.
      sd_id = sfstart('Example3.hdf', DFACC_RDWR)

C     Get the identifier of the first data set.
      sds_id = sfselect(sd_id, 0)

C     Create a file named "subfile1" and move the data set
C     into it, starting at byte location 24.
      offset = 24
      status = sfsextf(sds_id, 'subfile1', offset)

C     Dispose of the data set identifier to terminate access.
      status = sfendacc(sds_id)

C     Dispose of the file identifier to close the file.
      status = sfend(sd_id)

      end



[Top] [Prev] [Next] [Bottom]

hdfhelp@ncsa.uiuc.edu
HDF User's Guide - 06/04/97, NCSA HDF Development Group.