Working with big.matrix Objects

Overview

bigalgebra is designed to interoperate with the bigmemory ecosystem. This vignette demonstrates how to create in-memory and file-backed big.matrix objects, interact with them via the package’s wrappers, and manage the underlying resources safely.

Creating in-memory `big.matrix` objects

In-memory matrices behave much like ordinary R matrices but reside in shared memory, allowing multiple R sessions to access the same data.

X <- big.matrix(3, 3, type = "double", init = 0)
X[,] <- matrix(1:9, nrow = 3)
X[]
#>      [,1] [,2] [,3]
#> [1,]    1    4    7
#> [2,]    2    5    8
#> [3,]    3    6    9

Once created, the objects can be passed directly to Level 1 helpers:

dvcal(ALPHA = 2, X = X, BETA = -1, Y = X)
X[]
#>      [,1] [,2] [,3]
#> [1,]    1    4    7
#> [2,]    2    5    8
#> [3,]    3    6    9

Working with file-backed matrices

File-backed matrices persist their contents on disk, making them suitable for data sets that exceed available RAM.

dir.create(tmp_fb <- tempfile())
Y <- filebacked.big.matrix(4, 2, type = "double",
                           backingpath = tmp_fb,
                           backingfile = "fb.bin",
                           descriptorfile = "fb.desc",
                           init = 0)
Y[,] <- matrix(runif(8), nrow = 4)
Y[]
#>           [,1]      [,2]
#> [1,] 0.8199440 0.9463776
#> [2,] 0.5380758 0.1504005
#> [3,] 0.9179665 0.5259782
#> [4,] 0.4985450 0.3954064

These objects participate in higher-level operations without being loaded into memory.

Z <- filebacked.big.matrix(4, 2, type = "double",
                           backingpath = tmp_fb,
                           backingfile = "res.bin",
                           descriptorfile = "res.desc",
                           init = 0)
dvcal(ALPHA = 1.5, X = Y, BETA = 0, Y = Z)
Z[]
#>           [,1]      [,2]
#> [1,] 1.2299160 1.4195664
#> [2,] 0.8071138 0.2256007
#> [3,] 1.3769498 0.7889673
#> [4,] 0.7478175 0.5931096

Sharing matrices between sessions

The descriptor file records the metadata needed to reopen a file-backed matrix in a new R session. The attach.big.matrix() helper reconstructs the object:

Y_desc <- dget(file.path(tmp_fb, "fb.desc"))
Y_again <- attach.big.matrix(Y_desc)
identical(Y[,], Y_again[,])
#> [1] TRUE

Any operations performed via bigalgebra update the shared backing file, allowing all attached references to observe the change.

dsub(X = Z, Y = Y_again)
Y_again[]
#>            [,1]        [,2]
#> [1,] -0.4099720 -0.47318880
#> [2,] -0.2690379 -0.07520024
#> [3,] -0.4589833 -0.26298910
#> [4,] -0.2492725 -0.19770320

Cleaning up backing files

File-backed matrices allocate resources on disk. Deleting the backing and descriptor files once they are no longer needed helps keep the workspace tidy.

unlink(file.path(tmp_fb, c("fb.bin", "fb.desc", "res.bin", "res.desc")))
unlink(tmp_fb, recursive = TRUE)