---
title: "mirai - Torch Integration"
vignette: >
  %\VignetteIndexEntry{mirai - Torch Integration}
  %\VignetteEngine{knitr::knitr}
  %\VignetteEncoding{UTF-8}
---


### Torch Integration

Custom serialization functions may be registered to handle external pointer type reference objects.

This allows tensors from the [`torch`](https://torch.mlverse.org/) package to be used seamlessly in 'mirai' computations.

#### Setup Steps

1. Set up dameons.

2. Create the serialization configuration, specifying 'class' as 'torch_tensor' and 'vec' as TRUE.

3. Use `everywhere()`, supplying the configuration to the '.serial' argument, and (optionally) making the `torch` package available on all daemons for convenience.


``` r
library(mirai)
library(torch)

daemons(1)
#> [1] 1

cfg <- serial_config(
  class = "torch_tensor",
  sfunc = torch:::torch_serialize,
  ufunc = torch::torch_load,
  vec = TRUE
)

everywhere(library(torch), .serial = cfg)
```
#### Example Usage

The below example creates a convolutional neural network using `torch::nn_module()`.

A set of model parameters is also specified.

The model specification and parameters are then passed to and initialized within a 'mirai'.


``` r
model <- nn_module(
  initialize = function(in_size, out_size) {
    self$conv1 <- nn_conv2d(in_size, out_size, 5)
    self$conv2 <- nn_conv2d(in_size, out_size, 5)
  },
  forward = function(x) {
    x <- self$conv1(x)
    x <- nnf_relu(x)
    x <- self$conv2(x)
    x <- nnf_relu(x)
    x
  }
)

params <- list(in_size = 1, out_size = 20)

m <- mirai(do.call(model, params), model = model, params = params)

m[]
#> An `nn_module` containing 1,040 parameters.
#> 
#> ── Modules ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> • conv1: <nn_conv2d> #520 parameters
#> • conv2: <nn_conv2d> #520 parameters
```
The returned model is an object containing many tensor elements.

``` r
m$data$parameters$conv1.weight
#> torch_tensor
#> (1,1,.,.) = 
#>   0.0893  0.1619  0.0775 -0.1471 -0.0133
#>   0.0879  0.0147 -0.0174  0.1419  0.1484
#>  -0.0018  0.1977  0.1937  0.1783  0.0327
#>   0.0323 -0.0875  0.0986 -0.0839 -0.1093
#>   0.0711  0.0961 -0.1584  0.1479 -0.1896
#> 
#> (2,1,.,.) = 
#>  -0.0290 -0.1286  0.1438 -0.1308 -0.1262
#>  -0.1404 -0.0477  0.1990  0.1335  0.1378
#>   0.1169  0.1017 -0.0328 -0.0272  0.1104
#>  -0.0706  0.0792 -0.1670 -0.1186  0.0755
#>   0.1990 -0.1033 -0.1669  0.1652 -0.0643
#> 
#> (3,1,.,.) = 
#>  -0.0433  0.1686  0.0648  0.1092  0.1720
#>  -0.1340  0.1695  0.1466  0.1327 -0.0690
#>  -0.1999 -0.1581  0.0452 -0.0751 -0.0983
#>   0.1954 -0.0091 -0.1512  0.1844  0.0244
#>   0.1317 -0.1157 -0.0928  0.0231  0.1436
#> 
#> (4,1,.,.) = 
#>  -0.1073 -0.1389 -0.0121  0.0768  0.0314
#>   0.1249 -0.1564 -0.1910 -0.1824 -0.1843
#>   0.0146 -0.1667  0.0274  0.0109 -0.1467
#>  -0.0004  0.1380  0.1295 -0.1585 -0.0241
#>  -0.0741 -0.0961 -0.1936  0.0706 -0.0795
#> 
#> (5,1,.,.) = 
#>   0.0961  0.1492  0.1756 -0.0292  0.0832
#> ... [the output was truncated (use n=-1 to disable)]
#> [ CPUFloatType{20,1,5,5} ][ requires_grad = TRUE ]
```
It is usual for model parameters to then be passed to an optimiser.

This can also be initialized within a 'mirai' process.

``` r
optim <- mirai(optim_rmsprop(params = params), params = m$data$parameters)

optim[]
#> <optim_rmsprop>
#>   Inherits from: <torch_optimizer>
#>   Public:
#>     add_param_group: function (param_group) 
#>     clone: function (deep = FALSE) 
#>     defaults: list
#>     initialize: function (params, lr = 0.01, alpha = 0.99, eps = 1e-08, weight_decay = 0, 
#>     load_state_dict: function (state_dict, ..., .refer_to_state_dict = FALSE) 
#>     param_groups: list
#>     state: State, R6
#>     state_dict: function () 
#>     step: function (closure = NULL) 
#>     zero_grad: function () 
#>   Private:
#>     step_helper: function (closure, loop_fun)

daemons(0)
#> [1] 0
```
Above, tensors and complex objects containing tensors were passed seamlessly between host and daemon processes, in the same way as any other R object.

The custom serialization in `mirai` leverages R's own native 'refhook' mechanism to allow such completely transparent usage. Designed to be fast and efficient, data copies are minimised and the 'official' serialization methods from the `torch` package are used directly.