Matrices and Arrays

HDF5 is an excellent format for storing large, multi-dimensional numerical arrays. h5lite simplifies the process of reading and writing matrices and arrays by handling the complex memory layout differences between R and HDF5 automatically.

This vignette covers writing matrices, preserving dimension names (dimnames), and understanding how h5lite manages dimension ordering.

library(h5lite)
file <- tempfile(fileext = ".h5")

Writing Matrices

In R, matrices are simply 2-dimensional arrays. You can write them directly using h5_write(). h5lite preserves the dimensions exactly as they appear in R.

# Create a 3x4 matrix
mat <- matrix(1:12, nrow = 3, ncol = 4)

# Write to file
h5_write(mat, file, "linear_algebra/mat_a")

# Read back
mat_in <- h5_read(file, "linear_algebra/mat_a")

# Verify
all.equal(mat, mat_in)
#> [1] TRUE

Writing N-Dimensional Arrays

The same logic applies to arrays with 3 or more dimensions.

# Create a 3D array (e.g., spatial data over time: x, y, time)
vol <- array(runif(24), dim = c(4, 3, 2))

h5_write(vol, file, "spatial/volume")

# Check dimensions without reading the full data
h5_dim(file, "spatial/volume")
#> [1] 4 3 2

Dimension Names (dimnames)

R objects often carry metadata in the form of dimnames (row names, column names, etc.). HDF5 does not have a native “row name” concept for numerical arrays, but it supports Dimension Scales.

h5lite automatically converts R dimnames into HDF5 Dimension Scales. This allows your row and column names to survive the round-trip to disk and back.

# Create a matrix with row and column names
data <- matrix(rnorm(6), nrow = 2)
rownames(data) <- c("Sample_A", "Sample_B")
colnames(data) <- c("Gene_1", "Gene_2", "Gene_3")

h5_write(data, file, "genetics/expression")

# Read back
data_in <- h5_read(file, "genetics/expression")

print(data_in)
#>             Gene_1     Gene_2     Gene_3
#> Sample_A 0.5069539  0.4831928 -0.6186387
#> Sample_B 0.3764785 -0.5090470 -0.8438659

Technical Note: In the HDF5 file, the names are stored as separate datasets (e.g., _rownames, _colnames) and linked to the main dataset using HDF5 Dimension Scale attributes.

Dimension Ordering (Row-Major vs. Column-Major)

One of the most confusing aspects of HDF5 for R users is dimension ordering.

  • R is Column-Major: The first dimension varies fastest.
  • HDF5 (and C/C++/Python) is Row-Major: The last dimension varies fastest.

How h5lite handles it

To ensure that a 3x4 matrix in R looks like a 3x4 dataset in HDF5 tools (like h5dump or HDFView), h5lite rearranges the data during read/write operations.

  1. Writing: h5lite converts R’s column-major memory layout to HDF5’s row-major layout.
  2. Reading: h5lite converts the data back to column-major for R.

This ensures that indexing is preserved. x[2, 1] in R refers to the exact same value after reading it back from HDF5.

Interoperability with Python

Because h5lite writes the data in C-order (Row-Major) to match the HDF5 specification, files created with h5lite are perfectly readable by Python (h5py or pandas).

  • R: Shape is (3, 4)
  • Python: Shape is (3, 4)

Note: Some other R packages create HDF5 files by swapping the dimensions (writing a 3x4 matrix as 4x3) to avoid the cost of transposing data. h5lite prioritizes correctness and interoperability over raw write speed.

Chunking and Compression

Matrices and arrays are compressed automatically. h5lite splits each one into chunks (tiles of about 1 MB) and compresses each chunk independently. By default, it uses gzip level 5. Use the compress argument to choose another codec, and h5_inspect() to see the chunk shape and the space saved:

# A large, mostly-zero matrix (highly compressible)
sparse_mat <- matrix(0, nrow = 1000, ncol = 1000)
sparse_mat[1:10, 1:10] <- 1

# Default compression (gzip level 5)
h5_write(sparse_mat, file, "compressed/matrix")

# Zstandard: faster than gzip, with a similar or better ratio
h5_write(sparse_mat, file, "compressed/matrix_zstd", compress = "zstd-5")

h5_inspect(file, "compressed/matrix_zstd")
#> <HDF5 Dataset Properties>
#>   Type:    uint8               Size:    976.56 KB
#>   Layout:  chunked             Disk:    103.00 B
#>   Chunks:  [1000 x 1000]       Ratio:   9708.74x
#>   Pipeline: zstd

To pick chunk shapes, h5lite repeatedly halves a matrix’s largest dimension. For tall matrices, this keeps entire rows together in each chunk, which makes reading a block of rows efficient.

For codec choices, lossy compression of numeric arrays (such as ZFP), and chunk-size tuning, see vignette('compression').

Partial Reading

You don’t need to read a whole matrix to get part of it. The start and count arguments of h5_read() read a block of rows directly from the file:

big <- matrix(1:50000, nrow = 10000, ncol = 5)
h5_write(big, file, "big_matrix")

# Read 3 rows, starting at row 5001
h5_read(file, "big_matrix", start = 5001, count = 3)
#>      [,1]  [,2]  [,3]  [,4]  [,5]
#> [1,] 5001 15001 25001 35001 45001
#> [2,] 5002 15002 25002 35002 45002
#> [3,] 5003 15003 25003 35003 45003

For arrays with 3 or more dimensions, a single start value selects whole matrices along the last dimension. See vignette('partial-reading') for the full rules, including how to target individual columns and elements.