Introduction to innsight

Step 1: Model creation and converting

The innsight package aims to be as flexible as possible and independent of any particular deep learning package in which the passed network was learned or defined. For this reason, there are several ways in this package to pass a neural network to the Converter object, but the call is always the same:

# Using the helper function `convert`
converter <- convert(model, ...)
# It simply passes all arguments to the initialization function of
# the corresponding R6 class, i.e., it is equivalent to
converter <- Converter$new(model, ...)

Except for a neuralnet model, no names of inputs or outputs are stored in the given model. If no further arguments are set for the Converter instance or convert() function, default labels are generated for the input (e.g. 'X1', 'X2', …) and output names ('Y1', 'Y2', … ). In the converter, however, there is the possibility with the optional arguments input_names and output_names to pass the names, which will then be used in all results and plots created by this object.

library(torch)
library(innsight)
torch_manual_seed(123)

# Create model
model <- nn_sequential(
  nn_linear(3, 10),
  nn_relu(),
  nn_linear(10, 2, bias = FALSE),
  nn_softmax(2)
)
# Convert the model
conv_dense <- convert(model, input_dim = c(3))
# Convert model with input and output names
conv_dense_with_names <-
  convert(model,
    input_dim = c(3),
    input_names = list(c("Price", "Weight", "Height")),
    output_names = list(c("Buy it!", "Don't buy it!"))
  )

library(keras)

# Create model
model <- keras_model_sequential()
model <- model %>%
  layer_conv_2d(4, c(5, 4), input_shape = c(10, 10, 3), activation = "softplus") %>%
  layer_max_pooling_2d(c(2, 2), strides = c(1, 1)) %>%
  layer_conv_2d(6, c(3, 3), activation = "relu", padding = "same") %>%
  layer_max_pooling_2d(c(2, 2)) %>%
  layer_conv_2d(4, c(2, 2), strides = c(2, 1), activation = "relu") %>%
  layer_flatten() %>%
  layer_dense(5, activation = "softmax")

# Convert the model
conv_cnn <- convert(model)

library(neuralnet)
data(iris)

# Create model
model <- neuralnet(Species ~ Petal.Length + Petal.Width, iris,
  linear.output = FALSE
)

# Convert model
conv_dense <- convert(model)

model <- list(
  input_dim = 2,
  input_names = list(c("X1", "Feat2")),
  input_nodes = 1,
  output_nodes = 2,
  layers = list(
    list(
      type = "Dense", weight = matrix(rnorm(10), 5, 2), bias = rnorm(5),
      activation_name = "relu", input_layers = 0, output_layers = 2
    ),
    list(
      type = "Dense", weight = matrix(rnorm(5), 1, 5), bias = rnorm(1),
      activation_name = "sigmoid", input_layers = 1, output_layers = -1
    )
  )
)

converter <- convert(model)

converter

Step 2: Apply selected method

The innsight package provides several tools for analyzing black box neural networks based on dense or convolution layers. For the sake of uniform usage, all implemented methods inherit from the InterpretingMethod super class (see ?InterpretingMethod for details) and differ in each case only by method-specific arguments and settings. Therefore, each method has the following initialization structure:

method <- Method$new(converter, data, # required arguments
  channels_first = TRUE, # optional settings
  output_idx = NULL, # .
  ignore_last_act = TRUE, # .
  ... # other args and method-specific args
)

However, you can also use the helper functions (e.g., run_grad(), run_deeplift(), etc.) for initializing a new object:

method <- run_method(converter, data, # required arguments
  channels_first = TRUE, # optional settings
  output_idx = NULL, # .
  ignore_last_act = TRUE, # .
  ... # other args and method-specific args
)

The most important arguments are explained below. For a complete and detailed explanation, however, we refer to the R documentation (see ?InterpretingMethod) or the vignette “In-depth explanation” (vignette("detailed_overview", package = "innsight")).

• converter: This is the converter object created in the first step.

• data: The data to which the method is to be applied. These must have the same format as the input data of the passed model to the converter object. This means either an array, data.frame, torch_tensor or array-like format of size (batchsize, input_dim), if e.g., the model has only one input layer, or a list of the respective input sizes for each of the input layers.

• channels_first: The channel position of the given data (argument data). If TRUE, the channel axis is placed at the second position between the batch size and the remaining input axes, e.g., c(10,3,32,32) for a batch of ten images with three channels and a height and width of 32 pixels. Otherwise (FALSE), the channel axis is at the last position, i.e., c(10,32,32,3). If the data has no channel axis, use the default value TRUE.

• output_idx: These indices specify the output nodes or classes for which the method is to be applied. If the model has only one output layer, the values correspond to the indices of the output nodes, e.g., c(1,3,4) for the first, third and fourth output node. If your model has more than one output layer, you can pass the respective output nodes in a list which is described in detail in the R documentation (see ?InterpretingMethod) or in the in-depth vignette

• output_label: These values specify the output nodes for which the method is to be applied and can be used as an alternative to the argument output_idx. Only values that were previously passed with the argument output_names in the converter can be used.

• ignore_last_act: Set this logical value to include the last activation functions for each output layer, or not (default: TRUE)

The package innsight now offers the following methods for interpreting your model. To use them, simply replace the name "Method" with one of the method’s names below. There are also method-specific arguments, but these are explained in detail along with the methods in the R documentation (e.g., ?Gradient or ?LRP) or in the in-depth vignette. Let x ∈ ℝ^p the input instance, i is the feature index of the input and c the index of the output node or class to be explained:

• Gradient : Calculation of the model output Gradients with respect to the model inputs including the attribution method Gradient×Input:

  $$ \begin{align} \text{Gradient}(x)_i^c &= \frac{\partial f(x)_c}{\partial x_i}\\ \text{Gradient$\times$Input}(x)_i^c &= x_i \cdot \text{Gradient}(x)_i^c \end{align} $$
  Examples

  # Apply method 'Gradient' for the dense network
  grad_dense <- Gradient$new(conv_dense, iris[-c(1, 2, 5)])
  
  # You can also use the helper function `run_grad`
  grad_dense <- run_grad(conv_dense, iris[-c(1, 2, 5)])
  
  # Apply method 'Gradient x Input' for CNN
  x <- torch_randn(c(10, 3, 10, 10))
  grad_cnn <- run_grad(conv_cnn, x, times_input = TRUE)

• SmoothGrad : Calculation of the smoothed model output gradients (SmoothGrad) with respect to the model inputs by averaging the gradients over number of inputs with added noise (including SmoothGrad×Input): $$ \text{SmoothGrad}(x)_i^c = \mathbb{E}_{\varepsilon \sim \mathcal{N}(0, \sigma^2)}\left[\frac{\partial f(x + \varepsilon)_c}{\partial (x + \varepsilon)_i}\right] \approx \frac{1}{n} \sum_{k=1}^n \frac{\partial f(x + \varepsilon_k)_c}{\partial (x + \varepsilon_k)_i} $$ with ε₁, …ε_n ∼ 𝒩(0, σ²).

  Examples

  # Apply method 'SmoothGrad' for the dense network
  smooth_dense <- run_smoothgrad(conv_dense, iris[-c(1, 2, 5)])
  
  # Apply method 'SmoothGrad x Input' for CNN
  x <- torch_randn(c(10, 3, 10, 10))
  smooth_cnn <- run_smoothgrad(conv_cnn, x, times_input = TRUE)

• IntegratedGradient : Calculation of the integrated gradients (Sundararajan et al. (2017)) with respect to a reference input x̃: $$ \text{IntGrad}(x)_i^c = (x - \tilde{x}) \int_{\alpha = 0}^1 \frac{\partial f(\tilde{x} + \alpha (x - \tilde{x}))}{\partial x} d\alpha. $$

  Examples

  # Apply method 'IntegratedGradient' for the dense network
  intgrad_dense <- run_intgrad(conv_dense, iris[-c(1, 2, 5)])
  
  # Apply method 'IntegratedGradient' for CNN with the average baseline
  x <- torch_randn(c(10, 3, 10, 10))
  x_ref <- x$mean(1, keepdim = TRUE)
  intgrad_cnn <- run_intgrad(conv_cnn, x, x_ref = x_ref)

• ExpectedGradient : Calculation of the integrated gradients (Erion et al., 2021) with respect to a whole reference dataset X̃ ∼ x̃: $$ \text{ExpGrad}(x)_i^c = \mathbb{E}_{\tilde{x}\sim \tilde{X}, \alpha \sim U(0,1)} \left[(x - \tilde{x}) \times \frac{\partial f(\tilde{x} + \alpha (x - \tilde{x}))}{\partial x} \right] $$

  Examples

  # Apply method 'ExpectedGradient' for the dense network
  expgrad_dense <- run_expgrad(conv_dense, iris[-c(1, 2, 5)],
                               data_ref = iris[-c(1, 2, 5)])
  
  # Apply method 'ExpectedGradient' for CNN
  x <- torch_randn(c(10, 3, 10, 10))
  data_ref <- torch_randn(c(20, 3, 10, 10))
  expgrad_cnn <- run_expgrad(conv_cnn, x, data_ref = data_ref)

• LRP : Back-propagating the model output to the model input neurons to obtain relevance scores for the model prediction which is known as Layer-wise Relevance Propagation: $$ f(x)_c \approx \sum_{i=1}^d R_i $$ with R_i relevance score for input neuron i.

  Examples

  # Apply method 'LRP' for the dense network
  lrp_dense <- run_lrp(conv_dense, iris[-c(1, 2, 5)])
  
  # Apply method 'LRP' for CNN with alpha-beta-rule
  x <- torch_randn(c(10, 10, 10, 3))
  lrp_cnn <- run_lrp(conv_cnn, x,
    rule_name = "alpha_beta", rule_param = 1,
    channels_first = FALSE
  )

• DeepLift : Calculation of a decomposition of the model output with respect to the model inputs and a reference input which is known as Deep Learning Important Features (DeepLift): $$ \Delta y_c = f(x)_c - f(x_\text{ref})_c = \sum_{i=1}^d C_i $$ with C_i contribution score for input neuron i to the difference-from-reference model output Δy_c.

  Examples

  # Define reference value
  x_ref <- array(colMeans(iris[-c(1, 2, 5)]), dim = c(1, 2))
  # Apply method 'DeepLift' for the dense network
  deeplift_dense <- run_deeplift(conv_dense, iris[-c(1, 2, 5)], x_ref = x_ref)
  
  # Apply method 'DeepLift' for CNN (default is a zero baseline)
  x <- torch_randn(c(10, 3, 10, 10))
  deeplift_cnn <- run_deeplift(conv_cnn, x)

• ConnectionWeights : This is a naive and old approach by calculating the product of all weights from an input to an output neuron and then adding them up (see Connection Weights).

  Examples

  # Apply global method 'ConnectionWeights' for a dense network
  connectweights_dense <- run_cw(conv_dense)
  
  # Apply local method 'ConnectionWeights' for a CNN
  # Note: This variant requires input data
  x <- torch_randn(c(10, 3, 10, 10))
  connectweights_cnn <- run_cw(conv_cnn, x, times_input = TRUE)

• Additionally, the method DeepSHAP and the model-agnostic methods LIME and SHAP are implemented (by the functions run_deepshap(), run_lime() and run_shap()). For details, we refer to our vignette “In-depth explanation”.

📝 Notes

• By default, the last activation function is not taken into account for all data-based methods. Because often, this is a sigmoid/logistic or softmax function, which has increasingly smaller gradients with a growing distance from 0, which leads to the so-called saturation problem. But if you still want to consider the last activation function, use the argument ignore_last_act = FALSE.

• For data with channels, it is impossible to determine exactly on which axis the channels are located. Internally, all data and the converted model are in the data format “channels first”, i.e., directly after the batch dimension (batchsize, channels, input_dim). In case you want to pass data with “channels last” (e.g., for MNIST-data (batchsize, 28, 28, 3)), you have to indicate that with argument channels_first in the applied method.

• It can happen with very large and deep neural networks that the calculation for all outputs requires the entire memory and takes a very long time. But often, the results are needed only for certain output nodes. By default, only the results for the first 10 outputs are calculated, which can be adjusted individually with the argument output_idx by passing the relevant output indices.

Similar to the instances of the Converter class, the default print() function for R6 classes was also overridden for each method object, so that all important contents of the corresponding method are displayed:

smooth_cnn

Step 3: Show and plot the results

Once a method object has been created, the results can be returned as an array, data.frame, or torch_tensor, and can be further processed as desired. In addition, for each of the three sizes of the inputs (tabular, 1D signals or 2D images) suitable plot and boxplot functions based on ggplot2 are implemented. Due to the complexity of higher dimensional inputs, these plots and boxplots can also be displayed as an interactive plotly plots by using the argument as_plotly.

# Get the result with the class method
method$get_result(type = "array")

# or use the S3 function
get_result(method, type = "array")

# Get result (make sure 'grad_dense' is defined!)
result_array <- grad_dense$get_result()

# or with the S3 method
result_array <- get_result(grad_dense)

# Show the result for data point 1 and 71
result_array[c(1, 71), , ]

# Get result as data.frame (make sure 'lrp_cnn' is defined!)
result_data.frame <- lrp_cnn$get_result("data.frame")

# or with the S3 method
result_data.frame <- get_result(lrp_cnn, "data.frame")

# Show the first 5 rows
head(result_data.frame, 5)

# Get result (make sure 'deeplift_dense' is defined!)
result_torch <- deeplift_dense$get_result("torch_tensor")

# or with the S3 method
result_torch <- get_result(deeplift_dense, "torch_tensor")

# Show for datapoint 1 and 71 the result
result_torch[c(1, 71), , ]

# Create a plot for single data points
plot(method,
  data_idx = 1, # the data point to be plotted
  output_idx = NULL, # the indices of the output nodes/classes to be plotted
  output_label = NULL, # the class labels to be plotted
  aggr_channels = "sum",
  as_plotly = FALSE, # create an interactive plot
  ... # other arguments
)

# Create a plot with summarized results
plot_global(method,
  output_idx = NULL, # the indices of the output nodes/classes to be plotted
  output_label = NULL, # the class labels to be plotted
  ref_data_idx = NULL, # the index of an reference data point to be plotted
  aggr_channels = "sum",
  as_plotly = FALSE, # create an interactive plot
  ... # other arguments
)

# Alias for `plot_global` for tabular and signal data
boxplot(...)

# Plot the result of the first data point (default) for the output classes '1', '2' and '3'
plot(smooth_dense, output_idx = 1:3)
# You can plot several data points at once
plot(smooth_dense, data_idx = c(1, 144), output_idx = 1:3)
# Plot result for the first data point and first and fourth output classes
# and aggregate the channels by taking the Euclidean norm
plot(lrp_cnn, aggr_channels = "norm", output_idx = c(1, 4))

# Create a plotly plot for the first output
plot(lrp_cnn, aggr_channels = "norm", output_idx = c(1), as_plotly = TRUE)

# Create boxplot for the first two output classes
plot_global(smooth_dense, output_idx = 1:2)
# Use no preprocess function (default: abs) and plot a reference data point
plot_global(smooth_dense,
  output_idx = 1:3, preprocess_FUN = identity,
  ref_data_idx = c(55)
)

# You can do the same with the plotly-based plots
plot_global(smooth_dense,
  output_idx = 1:3, preprocess_FUN = identity,
  ref_data_idx = c(55), as_plotly = TRUE
)