Navigate back: Main / HDF5 User Guide / Additional Resources

Introduction

The Read-Only S3 virtual file driver, H5FD_ROS3, reads HDF5 files from Amazon S3 and from S3-compatible object stores such as MinIO and Ceph RADOS Gateway. It is selected with H5Pset_fapl_ros3.

The driver takes its configuration from the same places as the AWS CLI and the AWS SDKs: the File Access Property List (FAPL), AWS environment variables, and the shared AWS configuration files in the ~/.aws folder. It does not honor every AWS setting, so this page states exactly what is and is not consulted.

Settings are resolved in this order, with the first value found winning:

The File Access Property List
Environment variables (HDF5-specific and AWS)
The shared AWS configuration file

The profile read from the shared configuration file is the default profile, unless the AWS_PROFILE environment variable names another one. The locations of the configuration files themselves can be changed with the AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE environment variables.

AWS Region

An AWS region is required, including when reading from an S3-compatible object store that has no concept of AWS regions; in that case, any value the store accepts will do. The driver looks for it in:

The aws_region field of H5FD_ros3_fapl_t, set with H5Pset_fapl_ros3
The AWS_REGION environment variable
The AWS_DEFAULT_REGION environment variable
The region setting of the active profile in ~/.aws/config

If no region can be determined, opening a file fails. No default region is assumed. This is deliberate, since guessing a region can cause unintended cross-region data transfer charges.

Credentials

If the authenticate field of H5FD_ros3_fapl_t is true, credentials are taken only from the FAPL: secret_id and secret_key must both be non-empty, and a session token, if needed, must be supplied separately with H5Pset_fapl_ros3_token. Nothing else is consulted in this case.

Otherwise, credentials are resolved by the default credential provider chain of the aws-c-auth library, the same chain used by the AWS CLI and the AWS SDKs:

The AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables
The active profile in the shared configuration files, which covers assuming a role (role_arn with source_profile or credential_source), an external credential helper program (credential_process), and static credentials (aws_access_key_id, aws_secret_access_key, aws_session_token)
STS AssumeRoleWithWebIdentity, using AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN, and AWS_ROLE_SESSION_NAME, or the web_identity_token_file profile setting, as used by IAM Roles for Service Accounts on EKS
ECS and EKS container credentials, when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI is set
EC2 instance metadata

If none of these yield credentials, the driver falls back to anonymous credentials and sends unsigned requests. This is how objects in public buckets are read. It also means that a misconfigured profile produces unsigned requests rather than an error, which usually surfaces later as an HTTP 403 response from the server.

Note: AWS IAM Identity Center (SSO) is not supported directly. A profile that relies on sso_session or sso_start_url alone is not part of the chain above and will fall back to anonymous credentials. SSO credentials can still be used by way of the credential_process setting.; Because EC2 instance metadata is attempted by default, opening a file on a machine that is not an EC2 instance may be delayed while that step times out. Set the AWS_EC2_METADATA_DISABLED environment variable to true to skip it.

Endpoint URL

An alternate endpoint URL is needed to read from an S3-compatible object store. It is looked for in:

The File Access Property List, set with H5Pset_fapl_ros3_endpoint
The AWS_ENDPOINT_URL_S3 environment variable
The AWS_ENDPOINT_URL environment variable
The endpoint_url entry under the s3 key of the [services ...] section named by the active profile's services entry
The endpoint_url entry of the active profile itself

The last two are read from the shared AWS configuration file, and follow the endpoint precedence documented by AWS. Either of these forms works:

# the "global endpoint" form
[default]
endpoint_url = https://minio.example.org:9000
 
# the service-specific form, which takes precedence over the above
[profile dev]
services = my-services
 
[services my-services]
s3 =
  endpoint_url = https://minio.example.org:9000

Note: Every source other than the File Access Property List can be suppressed by setting the AWS_IGNORE_CONFIGURED_ENDPOINT_URLS environment variable, or the ignore_configured_endpoint_urls entry of the active profile, to true.

Request Addressing Style

The driver uses virtual-hosted-style requests (https://bucket.host/key) by default where it can, and path-style requests (https://host/bucket/key) otherwise. It uses path-style requests when:

the HDF5_ROS3_VFD_FORCE_PATH_STYLE environment variable is set to anything other than false, off, or 0; or
an alternate endpoint URL is in effect; or
the bucket name contains a '.'

The addressing_style setting of the shared AWS configuration file is not consulted.

Debugging Output

Three environment variables control diagnostic output:

Variable	Effect
HDF5_ROS3_VFD_DEBUG	Prints the driver's own debugging information to `stderr`
HDF5_ROS3_VFD_LOG_LEVEL	Enables aws-c-s3 library logging at `error`, `info`, `debug`, or `trace` level
HDF5_ROS3_VFD_LOG_FILE	Destination for that logging: `stdout`, `stderr`, or a filename. Defaults to H5FD_ROS3_VFD_DEFAULT_LOG_FILE

Setting HDF5_ROS3_VFD_LOG_LEVEL to trace is the most direct way to see which credential provider answered and which endpoint and request style were used.

Building

The ROS3 VFD requires the aws-c-s3 library and is enabled with the CMake option HDF5_ENABLE_ROS3_VFD. See docs/INSTALL_S3.md in the HDF5 source distribution for instructions on obtaining aws-c-s3 and on running the ROS3 test suite.

Navigate back: Main / HDF5 User Guide / Additional Resources