---
title: "Matrices and Arrays"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Matrices and Arrays}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

HDF5 is an excellent format for storing large, multi-dimensional numerical arrays. `h5lite` simplifies the process of reading and writing matrices and arrays by handling the complex memory layout differences between R and HDF5 automatically.

This vignette covers writing matrices, preserving dimension names (`dimnames`), and understanding how `h5lite` manages dimension ordering.

```{r setup}
library(h5lite)
file <- tempfile(fileext = ".h5")
```

## Writing Matrices

In R, matrices are simply 2-dimensional arrays. You can write them directly using `h5_write()`. `h5lite` preserves the dimensions exactly as they appear in R.

```{r}
# Create a 3x4 matrix
mat <- matrix(1:12, nrow = 3, ncol = 4)

# Write to file
h5_write(mat, file, "linear_algebra/mat_a")

# Read back
mat_in <- h5_read(file, "linear_algebra/mat_a")

# Verify
all.equal(mat, mat_in)
```

## Writing N-Dimensional Arrays

The same logic applies to arrays with 3 or more dimensions.

```{r}
# Create a 3D array (e.g., spatial data over time: x, y, time)
vol <- array(runif(24), dim = c(4, 3, 2))

h5_write(vol, file, "spatial/volume")

# Check dimensions without reading the full data
h5_dim(file, "spatial/volume")
```

## Dimension Names (dimnames)

R objects often carry metadata in the form of `dimnames` (row names, column names, etc.). HDF5 does not have a native "row name" concept for numerical arrays, but it supports **Dimension Scales**.

`h5lite` automatically converts R `dimnames` into HDF5 Dimension Scales. This allows your row and column names to survive the round-trip to disk and back.

```{r}
# Create a matrix with row and column names
data <- matrix(rnorm(6), nrow = 2)
rownames(data) <- c("Sample_A", "Sample_B")
colnames(data) <- c("Gene_1", "Gene_2", "Gene_3")

h5_write(data, file, "genetics/expression")

# Read back
data_in <- h5_read(file, "genetics/expression")

print(data_in)
```

> **Technical Note:** In the HDF5 file, the names are stored as separate datasets (e.g., `_rownames`, `_colnames`) and linked to the main dataset using HDF5 Dimension Scale attributes.

## Dimension Ordering (Row-Major vs. Column-Major)

One of the most confusing aspects of HDF5 for R users is dimension ordering.

* **R** is **Column-Major**: The first dimension varies fastest.
* **HDF5** (and C/C++/Python) is **Row-Major**: The last dimension varies fastest.

### How h5lite handles it

To ensure that a `3x4` matrix in R looks like a `3x4` dataset in HDF5 tools (like `h5dump` or `HDFView`), `h5lite` **rearranges** the data during read/write operations.

1.  **Writing:** `h5lite` converts R's column-major memory layout to HDF5's row-major layout.
2.  **Reading:** `h5lite` converts the data back to column-major for R.

This ensures that **indexing is preserved**. `x[2, 1]` in R refers to the exact same value after reading it back from HDF5.

### Interoperability with Python

Because `h5lite` writes the data in C-order (Row-Major) to match the HDF5 specification, files created with `h5lite` are perfectly readable by Python (`h5py` or `pandas`).

* **R:** Shape is `(3, 4)`
* **Python:** Shape is `(3, 4)`

*Note: Some other R packages create HDF5 files by swapping the dimensions (writing a 3x4 matrix as 4x3) to avoid the cost of transposing data. `h5lite` prioritizes correctness and interoperability over raw write speed.*

## Chunking and Compression

Matrices and arrays are compressed automatically. `h5lite` splits each one into **chunks** (tiles of about 1 MB) and compresses each chunk independently. By default, it uses gzip level 5. Use the `compress` argument to choose another codec, and `h5_inspect()` to see the chunk shape and the space saved:

```{r}
# A large, mostly-zero matrix (highly compressible)
sparse_mat <- matrix(0, nrow = 1000, ncol = 1000)
sparse_mat[1:10, 1:10] <- 1

# Default compression (gzip level 5)
h5_write(sparse_mat, file, "compressed/matrix")

# Zstandard: faster than gzip, with a similar or better ratio
h5_write(sparse_mat, file, "compressed/matrix_zstd", compress = "zstd-5")

h5_inspect(file, "compressed/matrix_zstd")
```

To pick chunk shapes, `h5lite` repeatedly halves a matrix's largest dimension. For tall matrices, this keeps entire rows together in each chunk, which makes reading a block of rows efficient.

*For codec choices, lossy compression of numeric arrays (such as ZFP), and chunk-size tuning, see `vignette('compression')`.*

## Partial Reading

You don't need to read a whole matrix to get part of it. The `start` and `count` arguments of `h5_read()` read a block of rows directly from the file:

```{r}
big <- matrix(1:50000, nrow = 10000, ncol = 5)
h5_write(big, file, "big_matrix")

# Read 3 rows, starting at row 5001
h5_read(file, "big_matrix", start = 5001, count = 3)
```

For arrays with 3 or more dimensions, a single `start` value selects whole matrices along the last dimension. See `vignette('partial-reading')` for the full rules, including how to target individual columns and elements.

```{r, include=FALSE}
unlink(file)
```