Package 'mlr3'

Description

Efficient, object-oriented programming on the building blocks of machine learning. Provides 'R6' objects for tasks, learners, resamplings, and measures. The package is geared towards scalability and larger datasets by supporting parallelization and out-of-memory data-backends like databases. While 'mlr3' focuses on the core computational operations, add-on packages provide additional functionality.

Learn mlr3

• Book on mlr3: https://mlr3book.mlr-org.com

• Use cases and examples gallery: https://mlr3gallery.mlr-org.com

• Cheat Sheets: https://github.com/mlr-org/mlr3cheatsheets

mlr3 extensions

• Preprocessing and machine learning pipelines: mlr3pipelines

• Analysis of benchmark experiments: mlr3benchmark

• More classification and regression tasks: mlr3data

• Connector to OpenML: mlr3oml

• Solid selection of good classification and regression learners: mlr3learners

• Even more learners: https://github.com/mlr-org/mlr3extralearners

• Tuning of hyperparameters: mlr3tuning

• Hyperband tuner: mlr3hyperband

• Visualizations for many mlr3 objects: mlr3viz

• Survival analysis and probabilistic regression: mlr3proba

• Cluster analysis: mlr3cluster

• Feature selection filters: mlr3filters

• Feature selection wrappers: mlr3fselect

• Interface to real (out-of-memory) data bases: mlr3db

• Performance measures as plain functions: mlr3measures

• Resampling methods for spatiotemporal data: mlr3spatiotempcv

• Data storage and prediction support for spatial objects: mlr3spatial

Suggested packages

• Parallelization framework: future

• Progress bars: progressr

• Encapsulated evaluation: evaluate, callr (external process)

Package Options

• "mlr3.exec_random": Randomize the order of execution in resample() and benchmark() during parallelization with future. Defaults to TRUE. Note that this does not affect the order of results.

• "mlr3.exec_chunk_size": Number of iterations to perform in a single future::future() during parallelization with future. Defaults to 1.

• "mlr3.exec_chunk_bins": Number of bins to split the iterations into. If set, "mlr3.exec_chunk_size" is ignored.

• "mlr3.debug": If set to TRUE, parallelization via future is disabled to simplify debugging and provide more concise tracebacks. Note that results computed in debug mode use a different seeding mechanism and are not reproducible.

• "mlr3.allow_utf8_names": If set to TRUE, checks on the feature names are relaxed, allowing non-ascii characters in column names. This is an experimental and temporal option to pave the way for text analysis, and will likely be removed in a future version of the package. analysis.

• "mlr3.warn_version_mismatch": Set to FALSE to silence warnings raised during predict if a learner has been trained with a different version version of mlr3.

Author(s)

Maintainer: Marc Becker [email protected] (ORCID)

Authors:

• Michel Lang [email protected] (ORCID)

• Bernd Bischl [email protected] (ORCID)

• Jakob Richter [email protected] (ORCID)

• Patrick Schratz [email protected] (ORCID)

• Martin Binder [email protected]

• Florian Pfisterer [email protected] (ORCID)

• Raphael Sonabend [email protected] (ORCID)

• Sebastian Fischer [email protected] (ORCID)

Other contributors:

• Giuseppe Casalicchio [email protected] (ORCID) [contributor]

• Stefan Coors [email protected] (ORCID) [contributor]

• Quay Au [email protected] (ORCID) [contributor]

• Lennart Schneider [email protected] (ORCID) [contributor]

• Lona Koers [email protected] [contributor]

References

Lang M, Binder M, Richter J, Schratz P, Pfisterer F, Coors S, Au Q, Casalicchio G, Kotthoff L, Bischl B (2019). “mlr3: A modern object-oriented machine learning framework in R.” Journal of Open Source Software. doi:10.21105/joss.01903, https://joss.theoj.org/papers/10.21105/joss.01903.

See Also

Useful links:

• https://mlr3.mlr-org.com

• https://github.com/mlr-org/mlr3

• Report bugs at https://github.com/mlr-org/mlr3/issues

Description

Convert object to a BenchmarkResult.

Usage

as_benchmark_result(x, ...)

## S3 method for class 'BenchmarkResult'
as_benchmark_result(x, ...)

## S3 method for class 'ResampleResult'
as_benchmark_result(x, ...)

Arguments

Value

(BenchmarkResult).

Description

Wraps a DataBackend around data. mlr3 ships with methods for data.frame (converted to a DataBackendDataTable and Matrix from package Matrix (converted to a DataBackendMatrix).

Additional methods are implemented in the package mlr3db, e.g. to connect to real DBMS like PostgreSQL (via dbplyr) or DuckDB (via DBI/duckdb).

Usage

## S3 method for class 'Matrix'
as_data_backend(data, primary_key = NULL, dense = NULL, ...)

as_data_backend(data, primary_key = NULL, ...)

## S3 method for class 'data.frame'
as_data_backend(data, primary_key = NULL, keep_rownames = FALSE, ...)

Arguments

Value

DataBackend.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackend, DataBackendDataTable, DataBackendMatrix

Examples

# create a new backend using the penguins data:
as_data_backend(palmerpenguins::penguins)

Description

Convert object to a Learner or a list of Learner.

Usage

as_learner(x, ...)

## S3 method for class 'Learner'
as_learner(x, clone = FALSE, discard_state = FALSE, ...)

as_learners(x, ...)

## Default S3 method:
as_learners(x, ...)

## S3 method for class 'list'
as_learners(x, ...)

Arguments

Value

Learner.

Description

Convert object to a Measure or a list of Measure.

Usage

as_measure(x, ...)

## S3 method for class ''NULL''
as_measure(x, task_type = NULL, ...)

## S3 method for class 'Measure'
as_measure(x, clone = FALSE, ...)

as_measures(x, ...)

## Default S3 method:
as_measures(x, ...)

## S3 method for class ''NULL''
as_measures(x, task_type = NULL, ...)

## S3 method for class 'list'
as_measures(x, ...)

Arguments

Value

Measure.

Description

Convert object to a Prediction or a list of Prediction.

Usage

as_prediction(x, check = FALSE, ...)

## S3 method for class 'Prediction'
as_prediction(x, check = FALSE, ...)

## S3 method for class 'PredictionDataClassif'
as_prediction(x, check = FALSE, ...)

## S3 method for class 'PredictionDataRegr'
as_prediction(x, check = FALSE, ...)

as_predictions(x, predict_sets = "test", ...)

## S3 method for class 'list'
as_predictions(x, predict_sets = "test", ...)

Arguments

Value

Prediction.

Description

Convert object to a PredictionClassif.

Usage

as_prediction_classif(x, ...)

## S3 method for class 'PredictionClassif'
as_prediction_classif(x, ...)

## S3 method for class 'data.frame'
as_prediction_classif(x, ...)

Arguments

Value

PredictionClassif.

Examples

# create a prediction object
task = tsk("penguins")
learner = lrn("classif.rpart", predict_type = "prob")
learner$train(task)
p = learner$predict(task)

# convert to a data.table
tab = as.data.table(p)

# convert back to a Prediction
as_prediction_classif(tab)

# split data.table into a list of data.tables
tabs = split(tab, tab$truth)

# convert back to list of predictions
preds = lapply(tabs, as_prediction_classif)

# calculate performance in each group
sapply(preds, function(p) p$score())

Description

Convert object to a PredictionData or a list of PredictionData.

Usage

as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'Prediction'
as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'PredictionData'
as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'list'
as_prediction_data(
  x,
  task,
  row_ids = task$row_ids,
  check = TRUE,
  ...,
  train_task
)

Arguments

Value

PredictionData.

Description

Convert object to a PredictionRegr.

Usage

as_prediction_regr(x, ...)

## S3 method for class 'PredictionRegr'
as_prediction_regr(x, ...)

## S3 method for class 'data.frame'
as_prediction_regr(x, ...)

Arguments

Value

PredictionRegr.

Examples

# create a prediction object
task = tsk("mtcars")
learner = lrn("regr.rpart")
learner$train(task)
p = learner$predict(task)

# convert to a data.table
tab = as.data.table(p)

# convert back to a Prediction
as_prediction_regr(tab)

# split data.table into a list of data.tables
tabs = split(tab, cut(tab$truth, 3))

# convert back to list of predictions
preds = lapply(tabs, as_prediction_regr)

# calculate performance in each group
sapply(preds, function(p) p$score())

Description

Convert object to a ResampleResult.

The S3 method for list expects argument x to be a list of Prediction objects and all other relevant objects (Task, Learners, and instantiated Resampling) must be provided, too. A more flexible way to manually create a ResampleResult is implemented in as_result_data().

Usage

as_resample_result(x, ...)

## S3 method for class 'ResampleResult'
as_resample_result(x, ...)

## S3 method for class 'ResultData'
as_resample_result(x, view = NULL, ...)

## S3 method for class 'list'
as_resample_result(x, task, learners, resampling, store_backends = TRUE, ...)

Arguments

Value

(ResampleResult).

Description

Convert object to a Resampling or a list of Resampling.

Usage

as_resampling(x, ...)

## S3 method for class 'Resampling'
as_resampling(x, clone = FALSE, ...)

as_resamplings(x, ...)

## Default S3 method:
as_resamplings(x, ...)

## S3 method for class 'list'
as_resamplings(x, ...)

Arguments

Description

This function allows to construct or convert to a ResultData object, the result container used by ResampleResult and BenchmarkResult. A ResampleResult or BenchmarkResult can be initialized with the returned object. Note that ResampleResults can be converted to a BenchmarkResult with as_benchmark_result() and multiple BenchmarkResults can be combined to a larger BenchmarkResult with the ⁠$combine()⁠ method of BenchmarkResult.

Usage

as_result_data(
  task,
  learners,
  resampling,
  iterations,
  predictions,
  learner_states = NULL,
  store_backends = TRUE
)

Arguments

Value

ResultData object which can be passed to the constructor of ResampleResult.

Examples

task = tsk("penguins")
learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 2)$instantiate(task)
iterations = seq_len(resampling$iters)

# manually train two learners.
# store learners and predictions
learners = list()
predictions = list()
for (i in iterations) {
  l = learner$clone(deep = TRUE)
  learners[[i]] = l$train(task, row_ids = resampling$train_set(i))
  predictions[[i]] = list(test = l$predict(task, row_ids = resampling$test_set(i)))
}

rdata = as_result_data(task, learners, resampling, iterations, predictions)
ResampleResult$new(rdata)

Description

Convert object to a Task or a list of Task.

Usage

as_task(x, ...)

## S3 method for class 'Task'
as_task(x, clone = FALSE, ...)

as_tasks(x, ...)

## Default S3 method:
as_tasks(x, ...)

## S3 method for class 'list'
as_tasks(x, ...)

Arguments

Description

Convert object to a TaskClassif. This is a S3 generic. mlr3 ships with methods for the following objects:

1. TaskClassif: ensure the identity

2. formula, data.frame(), matrix(), Matrix::Matrix() and DataBackend: provides an alternative to the constructor of TaskClassif.

3. TaskRegr: Calls convert_task().

Note that the target column will be converted to a factor(), if possible.

Usage

as_task_classif(x, ...)

## S3 method for class 'TaskClassif'
as_task_classif(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_classif(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  positive = NULL,
  label = NA_character_,
  ...
)

## S3 method for class 'matrix'
as_task_classif(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'Matrix'
as_task_classif(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_classif(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  positive = NULL,
  label = NA_character_,
  ...
)

## S3 method for class 'TaskRegr'
as_task_classif(
  x,
  target = NULL,
  drop_original_target = FALSE,
  drop_levels = TRUE,
  ...
)

## S3 method for class 'formula'
as_task_classif(
  x,
  data,
  id = deparse1(substitute(data)),
  positive = NULL,
  label = NA_character_,
  ...
)

Arguments

Value

TaskClassif.

Examples

as_task_classif(palmerpenguins::penguins, target = "species")

Description

Convert object to a TaskRegr. This is a S3 generic. mlr3 ships with methods for the following objects:

1. TaskRegr: ensure the identity

2. formula, data.frame(), matrix(), Matrix::Matrix() and DataBackend: provides an alternative to the constructor of TaskRegr.

3. TaskClassif: Calls convert_task().

Usage

as_task_regr(x, ...)

## S3 method for class 'TaskRegr'
as_task_regr(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_regr(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'matrix'
as_task_regr(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'Matrix'
as_task_regr(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_regr(
  x,
  target = NULL,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'TaskClassif'
as_task_regr(
  x,
  target = NULL,
  drop_original_target = FALSE,
  drop_levels = TRUE,
  ...
)

## S3 method for class 'formula'
as_task_regr(
  x,
  data,
  id = deparse1(substitute(data)),
  label = NA_character_,
  ...
)

Arguments

Value

TaskRegr.

Examples

as_task_regr(datasets::mtcars, target = "mpg")

Description

Convert object to a TaskUnsupervised or a list of TaskUnsupervised.

Usage

as_task_unsupervised(x, ...)

## S3 method for class 'Task'
as_task_unsupervised(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_unsupervised(
  x,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_unsupervised(
  x,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

as_tasks_unsupervised(x, ...)

## S3 method for class 'list'
as_tasks_unsupervised(x, clone = FALSE, ...)

## S3 method for class 'Task'
as_tasks_unsupervised(x, clone = FALSE, ...)

Arguments

Description

Runs a benchmark on arbitrary combinations of tasks (Task), learners (Learner), and resampling strategies (Resampling), possibly in parallel.

For large-scale benchmarking we recommend to use the mlr3batchmark package. This package runs benchmark experiments on high-performance computing clusters and handles failed experiments.

Usage

benchmark(
  design,
  store_models = FALSE,
  store_backends = TRUE,
  encapsulate = NA_character_,
  allow_hotstart = FALSE,
  clone = c("task", "learner", "resampling"),
  unmarshal = TRUE
)

Arguments

Value

BenchmarkResult.

Predict Sets

If you want to compare the performance of a learner on the training with the performance on the test set, you have to configure the Learner to predict on multiple sets by setting the field predict_sets to c("train", "test") (default is "test"). Each set yields a separate Prediction object during resampling. In the next step, you have to configure the measures to operate on the respective Prediction object:

m1 = msr("classif.ce", id = "ce.train", predict_sets = "train")
m2 = msr("classif.ce", id = "ce.test", predict_sets = "test")

The (list of) created measures can finally be passed to ⁠$aggregate()⁠ or ⁠$score()⁠.

Parallelization

This function can be parallelized with the future package. One job is one resampling iteration, and all jobs are send to an apply function from future.apply in a single batch. To select a parallel backend, use future::plan(). More on parallelization can be found in the book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html

Progress Bars

This function supports progress bars via the package progressr. Simply wrap the function call in progressr::with_progress() to enable them. Alternatively, call progressr::handlers() with global = TRUE to enable progress bars globally. We recommend the progress package as backend which can be enabled with progressr::handlers("progress").

Logging

The mlr3 uses the lgr package for logging. lgr supports multiple log levels which can be queried with getOption("lgr.log_levels").

To suppress output and reduce verbosity, you can lower the log from the default level "info" to "warn":

lgr::get_logger("mlr3")$set_threshold("warn")

To get additional log output for debugging, increase the log level to "debug" or "trace":

lgr::get_logger("mlr3")$set_threshold("debug")

To log to a file or a data base, see the documentation of lgr::lgr-package.

Note

The fitted models are discarded after the predictions have been scored in order to reduce memory consumption. If you need access to the models for later analysis, set store_models to TRUE.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: BenchmarkResult, benchmark_grid()

Examples

# benchmarking with benchmark_grid()
tasks = lapply(c("penguins", "sonar"), tsk)
learners = lapply(c("classif.featureless", "classif.rpart"), lrn)
resamplings = rsmp("cv", folds = 3)

design = benchmark_grid(tasks, learners, resamplings)
print(design)

set.seed(123)
bmr = benchmark(design)

## Data of all resamplings
head(as.data.table(bmr))

## Aggregated performance values
aggr = bmr$aggregate()
print(aggr)

## Extract predictions of first resampling result
rr = aggr$resample_result[[1]]
as.data.table(rr$prediction())

# Benchmarking with a custom design:
# - fit classif.featureless on penguins with a 3-fold CV
# - fit classif.rpart on sonar using a holdout
tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv", folds = 3), rsmp("holdout"))

design = data.table::data.table(
  task = tasks,
  learner = learners,
  resampling = resamplings
)

## Instantiate resamplings
design$resampling = Map(
  function(task, resampling) resampling$clone()$instantiate(task),
  task = design$task, resampling = design$resampling
)

## Run benchmark
bmr = benchmark(design)
print(bmr)

## Get the training set of the 2nd iteration of the featureless learner on penguins
rr = bmr$aggregate()[learner_id == "classif.featureless"]$resample_result[[1]]
rr$resampling$train_set(2)

Description

Takes a lists of Task, a list of Learner and a list of Resampling to generate a design in an expand.grid() fashion (a.k.a. cross join or Cartesian product).

There are two modes of operation, depending on the flag paired.

• With paired set to FALSE (default), resampling strategies are not allowed to be instantiated, and instead will be instantiated per task internally. The only exception to this rule applies if all tasks have exactly the same number of rows, and the resamplings are all instantiated for such tasks. The grid will be generated based on the Cartesian product of tasks, learners, and resamplings. Because the resamplings are instantiated on the tasks, reproducibility requires a seed to be set before calling this function, as this process is stochastic.

• With paired set to TRUE, tasks and resamplings are treated as pairs. I.e., you must provide as many tasks as corresponding instantiated resamplings. The grid will be generated based on the Cartesian product of learners and pairs.

Usage

benchmark_grid(
  tasks,
  learners,
  resamplings,
  param_values = NULL,
  paired = FALSE
)

Arguments

Value

(data.table::data.table()) with the cross product of the input vectors.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: BenchmarkResult, benchmark()

Examples

tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv"), rsmp("subsampling"))

# Set a seed to ensure reproducibility of the resampling instantiation
set.seed(123)
grid = benchmark_grid(tasks, learners, resamplings)
# the resamplings are now instantiated
head(grid$resampling[[1]]$instance)
print(grid)
## Not run: 
benchmark(grid)

## End(Not run)

# paired
learner = lrn("classif.rpart")
task1 = tsk("penguins")
task2 = tsk("german_credit")
res1 = rsmp("holdout")
res2 = rsmp("holdout")
res1$instantiate(task1)
res2$instantiate(task2)
design = benchmark_grid(list(task1, task2), learner, list(res1, res2), paired = TRUE)
print(design)

# manual construction of the grid with data.table::CJ()
grid = data.table::CJ(task = tasks, learner = learners,
  resampling = resamplings, sorted = FALSE)

# manual instantiation (not suited for a fair comparison of learners!)
Map(function(task, resampling) {
  resampling$instantiate(task)
}, task = grid$task, resampling = grid$resampling)
## Not run: 
benchmark(grid)

## End(Not run)

Description

This is the result container object returned by benchmark(). A BenchmarkResult consists of the data of multiple ResampleResults. The contents of a BenchmarkResult and ResampleResult are almost identical and the stored ResampleResults can be extracted via the ⁠$resample_result(i)⁠ method, where i is the index of the performed resample experiment. This allows us to investigate the extracted ResampleResult and individual resampling iterations, as well as the predictions and models from each fold.

BenchmarkResults can be visualized via mlr3viz's autoplot() function.

For statistical analysis of benchmark results and more advanced plots, see mlr3benchmark.

S3 Methods

• as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test", task_characteristics = FALSE)
  BenchmarkResult -> data.table::data.table()
  Returns a tabular view of the internal data.

• c(...)
  (BenchmarkResult, ...) -> BenchmarkResult
  Combines multiple objects convertible to BenchmarkResult into a new BenchmarkResult.

Active bindings

Methods

Public methods

• BenchmarkResult$new()

• BenchmarkResult$help()

• BenchmarkResult$format()

• BenchmarkResult$print()

• BenchmarkResult$combine()

• BenchmarkResult$marshal()

• BenchmarkResult$unmarshal()

• BenchmarkResult$score()

• BenchmarkResult$obs_loss()

• BenchmarkResult$aggregate()

• BenchmarkResult$filter()

• BenchmarkResult$resample_result()

• BenchmarkResult$discard()

• BenchmarkResult$clone()

Method new()

Creates a new instance of this R6 class.

Usage

BenchmarkResult$new(data = NULL)

Arguments

Method help()

Opens the help page for this object.

Usage

BenchmarkResult$help()

Method format()

Helper for print outputs.

Usage

BenchmarkResult$format(...)

Arguments

Method print()

Printer.

Usage

BenchmarkResult$print()

Method combine()

Fuses a second BenchmarkResult into itself, mutating the BenchmarkResult in-place. If the second BenchmarkResult bmr is NULL, simply returns self. Note that you can alternatively use the combine function c() which calls this method internally.

Usage

BenchmarkResult$combine(bmr)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Method marshal()

Marshals all stored models.

Usage

BenchmarkResult$marshal(...)

Arguments

Method unmarshal()

Unmarshals all stored models.

Usage

BenchmarkResult$unmarshal(...)

Arguments

Method score()

Returns a table with one row for each resampling iteration, including all involved objects: Task, Learner, Resampling, iteration number (integer(1)), and Prediction. If ids is set to TRUE, character column of extracted ids are added to the table for convenient filtering: "task_id", "learner_id", and "resampling_id".

Additionally calculates the provided performance measures and binds the performance scores as extra columns. These columns are named using the id of the respective Measure.

Usage

BenchmarkResult$score(
  measures = NULL,
  ids = TRUE,
  conditions = FALSE,
  predictions = TRUE
)

Arguments

Returns

data.table::data.table().

Method obs_loss()

Calculates the observation-wise loss via the loss function set in the Measure's field obs_loss. Returns a data.table() with the columns row_ids, truth, response and one additional numeric column for each measure, named with the respective measure id. If there is no observation-wise loss function for the measure, the column is filled with NA values. Note that some measures such as RMSE, do have an ⁠$obs_loss⁠, but they require an additional transformation after aggregation, in this example taking the square-root.

Usage

BenchmarkResult$obs_loss(measures = NULL, predict_sets = "test")

Arguments

Method aggregate()

Returns a result table where resampling iterations are combined into ResampleResults. A column with the aggregated performance score is added for each Measure, named with the id of the respective measure.

The method for aggregation is controlled by the Measure, e.g. micro aggregation, macro aggregation or custom aggregation. Most measures default to macro aggregation.

Note that the aggregated performances just give a quick impression which approaches work well and which approaches are probably underperforming. However, the aggregates do not account for variance and cannot replace a statistical test. See mlr3viz to get a better impression via boxplots or mlr3benchmark for critical difference plots and significance tests.

For convenience, different flags can be set to extract more information from the returned ResampleResult.

Usage

BenchmarkResult$aggregate(
  measures = NULL,
  ids = TRUE,
  uhashes = FALSE,
  params = FALSE,
  conditions = FALSE
)

Arguments

Returns

data.table::data.table().

Method filter()

Subsets the benchmark result. If task_ids is not NULL, keeps all tasks with provided task ids and discards all others tasks. Same procedure for learner_ids and resampling_ids.

Usage

BenchmarkResult$filter(
  task_ids = NULL,
  task_hashes = NULL,
  learner_ids = NULL,
  learner_hashes = NULL,
  resampling_ids = NULL,
  resampling_hashes = NULL
)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keeps the object in its previous state.

Method resample_result()

Retrieve the i-th ResampleResult, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

BenchmarkResult$resample_result(i = NULL, uhash = NULL)

Arguments

Returns

ResampleResult.

Method discard()

Shrinks the BenchmarkResult by discarding parts of the internally stored data. Note that certain operations might stop work, e.g. extracting importance values from learners or calculating measures requiring the task's data.

Usage

BenchmarkResult$discard(backends = FALSE, models = FALSE)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keeps the object in its previous state.

Method clone()

The objects of this class are cloneable with this method.

Usage

BenchmarkResult$clone(deep = FALSE)

Arguments

Note

All stored objects are accessed by reference. Do not modify any extracted object without cloning it first.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: benchmark(), benchmark_grid()

Examples

set.seed(123)
learners = list(
  lrn("classif.featureless", predict_type = "prob"),
  lrn("classif.rpart", predict_type = "prob")
)

design = benchmark_grid(
  tasks = list(tsk("sonar"), tsk("penguins")),
  learners = learners,
  resamplings = rsmp("cv", folds = 3)
)
print(design)

bmr = benchmark(design)
print(bmr)

bmr$tasks
bmr$learners

# first 5 resampling iterations
head(as.data.table(bmr, measures = c("classif.acc", "classif.auc")), 5)

# aggregate results
bmr$aggregate()

# aggregate results with hyperparameters as separate columns
mlr3misc::unnest(bmr$aggregate(params = TRUE), "params")

# extract resample result for classif.rpart
rr = bmr$aggregate()[learner_id == "classif.rpart", resample_result][[1]]
print(rr)

# access the confusion matrix of the first resampling iteration
rr$predictions()[[1]]$confusion

# reduce to subset with task id "sonar"
bmr$filter(task_ids = "sonar")
print(bmr)

Description

A regression task to predict the median house value in California.

Contains 9 features and 20640 observations. Target column is "median_house_value".

Format

R6::R6Class inheriting from TaskRegr.

Construction

mlr_tasks$get("california_housing")
tsk("california_housing")

Meta Information

• Task type: “regr”

• Dimensions: 20640x10

• Properties: -

• Has Missings: TRUE

• Target: “median_house_value”

• Features: “households”, “housing_median_age”, “latitude”, “longitude”, “median_income”, “ocean_proximity”, “population”, “total_bedrooms”, “total_rooms”

Source

https://www.kaggle.com/datasets/camnugent/california-housing-prices

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html

• Package mlr3data for more toy tasks.

• Package mlr3oml for downloading tasks from https://www.openml.org.

• Package mlr3viz for some generic visualizations.

• Dictionary of Tasks: mlr_tasks

• as.data.table(mlr_tasks) for a table of available Tasks in the running session (depending on the loaded packages).

• mlr3fselect and mlr3filters for feature selection and feature filtering.

• Extension packages for additional task types:

  • Unsupervised clustering: mlr3cluster

  • Probabilistic supervised regression and survival analysis: https://mlr3proba.mlr-org.com/.

Other Task: Task, TaskClassif, TaskRegr, TaskSupervised, TaskUnsupervised, mlr_tasks, mlr_tasks_breast_cancer, mlr_tasks_german_credit, mlr_tasks_iris, mlr_tasks_mtcars, mlr_tasks_penguins, mlr_tasks_pima, mlr_tasks_sonar, mlr_tasks_spam, mlr_tasks_wine, mlr_tasks_zoo

Description

The task's target is replaced by a different column from the data.

Usage

convert_task(
  intask,
  target = NULL,
  new_type = NULL,
  drop_original_target = FALSE,
  drop_levels = TRUE
)

Arguments

Value

Task of requested type.

Description

This is the abstract base class for data backends.

Data backends provide a layer of abstraction for various data storage systems. It is not recommended to work directly with the DataBackend. Instead, all data access is handled transparently via the Task.

This package comes with two implementations for backends:

• DataBackendDataTable which stores the data as data.table::data.table().

• DataBackendMatrix which stores the data as sparse Matrix::sparseMatrix().

To connect to out-of-memory database management systems such as SQL servers, see the extension package mlr3db.

Details

The required set of fields and methods to implement a custom DataBackend is listed in the respective sections (see DataBackendDataTable or DataBackendMatrix for exemplary implementations of the interface).

Public fields

Active bindings

Methods

Public methods

• DataBackend$new()

• DataBackend$format()

• DataBackend$print()

Method new()

Creates a new instance of this R6 class.

Note: This object is typically constructed via a derived classes, e.g. DataBackendDataTable or DataBackendMatrix, or via the S3 method as_data_backend().

Usage

DataBackend$new(data, primary_key, data_formats)

Arguments

Method format()

Helper for print outputs.

Usage

DataBackend$format(...)

Arguments

Method print()

Printer.

Usage

DataBackend$print()

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackendDataTable, DataBackendMatrix, as_data_backend.Matrix()

Examples

data = data.table::data.table(id = 1:5, x = runif(5),
  y = sample(letters[1:3], 5, replace = TRUE))

b = DataBackendDataTable$new(data, primary_key = "id")
print(b)
b$head(2)
b$data(rows = 1:2, cols = "x")
b$distinct(rows = b$rownames, "y")
b$missings(rows = b$rownames, cols = names(data))

Description

DataBackend for data.table which serves as an efficient in-memory data base.

Super class

mlr3::DataBackend -> DataBackendDataTable

Public fields

Active bindings

Methods

Public methods

• DataBackendDataTable$new()

• DataBackendDataTable$data()

• DataBackendDataTable$head()

• DataBackendDataTable$distinct()

• DataBackendDataTable$missings()

Method new()

Creates a new instance of this R6 class.

Note that DataBackendDataTable does not copy the input data, while as_data_backend() calls data.table::copy(). as_data_backend() also takes care about casting to a data.table() and adds a primary key column if necessary.

Usage

DataBackendDataTable$new(data, primary_key)

Arguments

Method data()

Returns a slice of the data in the specified format. Currently, the only supported formats are "data.table" and "Matrix". The rows must be addressed as vector of primary key values, columns must be referred to via column names. Queries for rows with no matching row id and queries for columns with no matching column name are silently ignored. Rows are guaranteed to be returned in the same order as rows, columns may be returned in an arbitrary order. Duplicated row ids result in duplicated rows, duplicated column names lead to an exception.

Usage

DataBackendDataTable$data(rows, cols, data_format)

Arguments

Method head()

Retrieve the first n rows.

Usage

DataBackendDataTable$head(n = 6L)

Arguments

Returns

data.table::data.table() of the first n rows.

Method distinct()

Returns a named list of vectors of distinct values for each column specified. If na_rm is TRUE, missing values are removed from the returned vectors of distinct values. Non-existing rows and columns are silently ignored.

Usage

DataBackendDataTable$distinct(rows, cols, na_rm = TRUE)

Arguments

Returns

Named list() of distinct values.

Method missings()

Returns the number of missing values per column in the specified slice of data. Non-existing rows and columns are silently ignored.

Usage

DataBackendDataTable$missings(rows, cols)

Arguments

Returns

Total of missing values per column (named numeric()).

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackend, DataBackendMatrix, as_data_backend.Matrix()

Examples

data = as.data.table(palmerpenguins::penguins)
data$id = seq_len(nrow(palmerpenguins::penguins))
b = DataBackendDataTable$new(data = data, primary_key = "id")
print(b)
b$head()
b$data(rows = 100:101, cols = "species")

b$nrow
head(b$rownames)

b$ncol
b$colnames

# alternative construction
as_data_backend(palmerpenguins::penguins)

Description

DataBackend for Matrix. Data is split into a (numerical) sparse part and an optional dense part. These parts are automatically merged to a sparse format during ⁠$data()⁠. Note that merging both parts potentially comes with a data loss, as all dense columns are converted to numeric columns.

Super class

mlr3::DataBackend -> DataBackendMatrix

Active bindings

Methods

Public methods

• DataBackendMatrix$new()

• DataBackendMatrix$data()

• DataBackendMatrix$head()

• DataBackendMatrix$distinct()

• DataBackendMatrix$missings()

Method new()

Creates a new instance of this R6 class.

Usage

DataBackendMatrix$new(data, dense, primary_key = NULL)

Arguments

Method data()

Returns a slice of the data as "data.table". The rows must be addressed as vector of primary key values, columns must be referred to via column names. Queries for rows with no matching row id and queries for columns with no matching column name are silently ignored. Rows are guaranteed to be returned in the same order as rows, columns may be returned in an arbitrary order. Duplicated row ids result in duplicated rows, duplicated column names lead to an exception.

Usage

DataBackendMatrix$data(rows, cols, data_format)

Arguments

Method head()

Retrieve the first n rows.

Usage

DataBackendMatrix$head(n = 6L)

Arguments

Returns

data.table::data.table() of the first n rows.

Method distinct()

Returns a named list of vectors of distinct values for each column specified. If na_rm is TRUE, missing values are removed from the returned vectors of distinct values. Non-existing rows and columns are silently ignored.

Usage

DataBackendMatrix$distinct(rows, cols, na_rm = TRUE)

Arguments

Returns

Named list() of distinct values.

Method missings()

Returns the number of missing values per column in the specified slice of data. Non-existing rows and columns are silently ignored.

Usage

DataBackendMatrix$missings(rows, cols)

Arguments

Returns

Total of missing values per column (named numeric()).

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackend, DataBackendDataTable, as_data_backend.Matrix()

Examples

requireNamespace("Matrix")
data = Matrix::Matrix(sample(0:1, 20, replace = TRUE), ncol = 2)
colnames(data) = c("x1", "x2")
dense = data.frame(
  ..row_id = 1:10,
  num = runif(10),
  fact = factor(sample(c("a", "b"), 10, replace = TRUE), levels = c("a", "b"))
)

b = as_data_backend(data, dense = dense, primary_key = "..row_id")
b$head()
b$data(1:3, b$colnames)

Description

Create a fallback learner for a given learner. The function searches for a suitable fallback learner based on the task type. Additional checks are performed to ensure that the fallback learner supports the predict type.

Usage

default_fallback(learner, ...)

## S3 method for class 'Learner'
default_fallback(learner, ...)

## S3 method for class 'LearnerClassif'
default_fallback(learner, ...)

## S3 method for class 'LearnerRegr'
default_fallback(learner, ...)

Arguments

Value

Learner

Description

Gets the default measures using the information in mlr_reflections$default_measures:

• "classif.ce" for classification ("classif").

• "regr.mse" for regression ("regr").

• Add-on package may register additional default measures for their own task types.

Usage

default_measures(task_type)

Arguments

Value

list of Measure.

Examples

default_measures("classif")
default_measures("regr")

Description

This class stores learners for hot starting training, i.e. resuming or continuing from an already fitted model. We assume that hot starting is only possible if a single hyperparameter (also called the fidelity parameter, usually controlling the complexity or expensiveness) is altered and all other hyperparameters are identical.

The HotstartStack stores trained learners which can be potentially used to hot start a learner. Learner automatically hot start while training if a stack is attached to the ⁠$hotstart_stack⁠ field and the stack contains a suitable learner.

For example, if you want to train a random forest learner with 1000 trees but already have a random forest learner with 500 trees (hot start learner), you can add the hot start learner to the HotstartStack of the expensive learner with 1000 trees. If you now call the train() method (or resample() or benchmark()), a random forest with 500 trees will be fitted and combined with the 500 trees of the hotstart learner, effectively saving you to fit 500 trees.

Hot starting is only supported by learners which have the property "hotstart_forward" or "hotstart_backward". For example, an xgboost model (in mlr3learners) can hot start forward by adding more boosting iterations, and a random forest can go backwards by removing trees. The fidelity parameters are tagged with "hotstart" in learner's parameter set.

Public fields

Methods

Public methods

• HotstartStack$new()

• HotstartStack$add()

• HotstartStack$start_cost()

• HotstartStack$format()

• HotstartStack$print()

• HotstartStack$clone()

Method new()

Creates a new instance of this R6 class.

Usage

HotstartStack$new(learners = NULL, hotstart_threshold = NULL)

Arguments

Method add()

Add learners to hot start stack.

Usage

HotstartStack$add(learners)

Arguments

Returns

self (invisibly).

Method start_cost()

Calculates the cost for each learner of the stack to hot start the target learner.

The following cost values can be returned:

• NA_real_: Learner is unsuitable to hot start target learner.

• -1: Hotstart learner in the stack and target learner are identical.

• 0 Cost for hot starting backwards is always 0.

• ⁠> 0⁠ Cost for hot starting forward.

Usage

HotstartStack$start_cost(learner, task_hash)

Arguments

Method format()

Helper for print outputs.

Usage

HotstartStack$format(...)

Arguments

Method print()

Printer.

Usage

HotstartStack$print(...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

HotstartStack$clone(deep = FALSE)

Arguments

Examples

# train learner on pima task
task = tsk("pima")
learner = lrn("classif.debug", iter = 1)
learner$train(task)

# initialize stack with previously fitted learner
hot = HotstartStack$new(list(learner))

# retrieve learner with increased fidelity parameter
learner = lrn("classif.debug", iter = 2)

# calculate cost of hot starting
hot$start_cost(learner, task$hash)

# add stack with hot start learner
learner$hotstart_stack = hot

# train automatically uses hot start learner while fitting the model
learner$train(task)

Description

extract_pkgs() extracts required package from various objects, including TaskGenerator, Learner, Measure and objects from extension packages such as mlr3pipelines or mlr3filters. If applied on a list, the function is called recursively on all elements.

install_pkgs() calls extract_pkgs() internally and proceeds with the installation of extracted packages.

Usage

install_pkgs(x, ...)

extract_pkgs(x)

## S3 method for class 'character'
extract_pkgs(x)

## S3 method for class 'R6'
extract_pkgs(x)

## S3 method for class 'list'
extract_pkgs(x)

## S3 method for class 'ResampleResult'
extract_pkgs(x)

## S3 method for class 'BenchmarkResult'
extract_pkgs(x)

Arguments

Details

If a package contains a forward slash ('/'), it is assumed to be a package hosted on GitHub in "<user>/<repo>" format, and the string will be passed to remotes::install_github(). Otherwise, the package name will be passed to remotes::install_cran().

Value

extract_pkgs() returns a character() of package strings, install_pkgs() returns the names of extracted packages invisibly.

Examples

extract_pkgs(lrns(c("regr.rpart", "regr.featureless")))

Description

This is the abstract base class for learner objects like LearnerClassif and LearnerRegr.

Learners are build around the three following key parts:

• Methods ⁠$train()⁠ and ⁠$predict()⁠ which call internal methods or private methods ⁠$.train()⁠/⁠$.predict()⁠).

• A paradox::ParamSet which stores meta-information about available hyperparameters, and also stores hyperparameter settings.

• Meta-information about the requirements and capabilities of the learner.

• The fitted model stored in field ⁠$model⁠, available after calling ⁠$train()⁠.

Predefined learners are stored in the dictionary mlr_learners, e.g. classif.rpart or regr.rpart.

More classification and regression learners are implemented in the add-on package mlr3learners. Learners for survival analysis (or more general, for probabilistic regression) can be found in mlr3proba. Unsupervised cluster algorithms are implemented in mlr3cluster. The dictionary mlr_learners gets automatically populated with the new learners as soon as the respective packages are loaded.

More (experimental) learners can be found in the GitHub repository: https://github.com/mlr-org/mlr3extralearners. A guide on how to extend mlr3 with custom learners can be found in the mlr3book.

To combine the learner with preprocessing operations like factor encoding, mlr3pipelines is recommended. Hyperparameters stored in the param_set can be tuned with mlr3tuning.

Optional Extractors

Specific learner implementations are free to implement additional getters to ease the access of certain parts of the model in the inherited subclasses.

For the following operations, extractors are standardized:

• importance(...): Returns the feature importance score as numeric vector. The higher the score, the more important the variable. The returned vector is named with feature names and sorted in decreasing order. Note that the model might omit features it has not used at all. The learner must be tagged with property "importance". To filter variables using the importance scores, see package mlr3filters.

• selected_features(...): Returns a subset of selected features as character(). The learner must be tagged with property "selected_features".

• oob_error(...): Returns the out-of-bag error of the model as numeric(1). The learner must be tagged with property "oob_error".

• internal_valid_scores: Returns the internal validation score(s) of the model as a named list(). Only available for Learners with the "validation" property. If the learner is not trained yet, this returns NULL.

• internal_tuned_values: Returns the internally tuned hyperparameters of the model as a named list(). Only available for Learners with the "internal_tuning" property. If the learner is not trained yet, this returns NULL.

Setting Hyperparameters

All information about hyperparameters is stored in the slot param_set which is a paradox::ParamSet. The printer gives an overview about the ids of available hyperparameters, their storage type, lower and upper bounds, possible levels (for factors), default values and assigned values. To set hyperparameters, assign a named list to the subslot values:

lrn = lrn("classif.rpart")
lrn$param_set$values = list(minsplit = 3, cp = 0.01)

Note that this operation replaces all previously set hyperparameter values. If you only intend to change one specific hyperparameter value and leave the others as-is, you can use the helper function mlr3misc::insert_named():

lrn$param_set$values = mlr3misc::insert_named(lrn$param_set$values, list(cp = 0.001))

If the learner has additional hyperparameters which are not encoded in the ParamSet, you can easily extend the learner. Here, we add a factor hyperparameter with id "foo" and possible levels "a" and "b":

lrn$param_set$add(paradox::ParamFct$new("foo", levels = c("a", "b")))

Implementing Validation

Some Learners, such as XGBoost, other boosting algorithms, or deep learning models (mlr3torch), utilize validation data during the training to prevent overfitting or to log the validation performance. It is possible to configure learners to be able to receive such an independent validation set during training. To do so, one must:

• annotate the learner with the "validation" property

• implement the active binding ⁠$internal_valid_scores⁠ (see section Optional Extractors), as well as the private method ⁠$.extract_internal_valid_scores()⁠ which returns the (final) internal validation scores from the model of the Learner and returns them as a named list() of numeric(1). If the model is not trained yet, this method should return NULL.

• Add the validate parameter, which can be either NULL, a ratio in $(0, 1)$, "test", or "predefined":

  • NULL: no validation

  • ratio: only proportion 1 - ratio of the task is used for training and ratio is used for validation.

  • "test" means that the "test" task is used. Warning: This can lead to biased performance estimation. This option is only available if the learner is being trained via resample(), benchmark() or functions that internally use them, e.g. tune() of mlr3tuning or batchmark() of mlr3batchmark. This is especially useful for hyperparameter tuning, where one might e.g. want to use the same validation data for early stopping and model evaluation.

  • "predefined" means that the task's (manually set) ⁠$internal_valid_task⁠ is used. See the Task documentation for more information.

For an example how to do this, see LearnerClassifDebug. Note that in .train(), the ⁠$internal_valid_task⁠ will only be present if the ⁠$validate⁠ field of the Learner is set to a non-NULL value.

Implementing Internal Tuning

Some learners such as XGBoost or cv.glmnet can internally tune hyperparameters. XGBoost, for example, can tune the number of boosting rounds based on the validation performance. CV Glmnet, on the other hand, can tune the regularization parameter based on an internal cross-validation. Internal tuning can therefore rely on the internal validation data, but does not necessarily do so.

In order to be able to combine this internal hyperparamer tuning with the standard hyperparameter optimization implemented via mlr3tuning, one most:

• annotate the learner with the "internal_tuning" property

• implement the active binding ⁠$internal_tuned_values⁠ (see section Optional Extractors) as well as the private method ⁠$.extract_internal_tuned_values()⁠ which extracts the internally tuned values from the Learner's model and returns them as a named list(). If the model is not trained yet, this method should return NULL.

• Have at least one parameter tagged with "internal_tuning", which requires to also provide a in_tune_fn and disable_tune_fn, and should also include a default aggregation function.

For an example how to do this, see LearnerClassifDebug.

Implementing Marshaling

Some Learners have models that cannot be serialized as they e.g. contain external pointers. In order to still be able to save them, use them with parallelization or callr encapsulation it is necessary to implement how they should be (un)-marshaled. See marshaling for how to do this.

Public fields

Active bindings

Methods

Public methods

• Learner$new()

• Learner$format()

• Learner$print()

• Learner$help()

• Learner$train()

• Learner$predict()

• Learner$predict_newdata()

• Learner$reset()

• Learner$base_learner()

• Learner$encapsulate()

• Learner$clone()

Method new()

Creates a new instance of this R6 class.

Note that this object is typically constructed via a derived classes, e.g. LearnerClassif or LearnerRegr.

Usage

Learner$new(
  id,
  task_type,
  param_set = ps(),
  predict_types = character(),
  feature_types = character(),
  properties = character(),
  data_formats,
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method format()

Helper for print outputs.

Usage

Learner$format(...)

Arguments

Method print()

Printer.

Usage

Learner$print(...)

Arguments

Method help()

Opens the corresponding help page referenced by field ⁠$man⁠.

Usage

Learner$help()

Method train()

Train the learner on a set of observations of the provided task. Mutates the learner by reference, i.e. stores the model alongside other information in field ⁠$state⁠.

Usage

Learner$train(task, row_ids = NULL)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keeps the object in its previous state.

Method predict()

Uses the information stored during ⁠$train()⁠ in ⁠$state⁠ to create a new Prediction for a set of observations of the provided task.

Usage

Learner$predict(task, row_ids = NULL)

Arguments

Returns

Prediction.

Method predict_newdata()

Uses the model fitted during ⁠$train()⁠ to create a new Prediction based on the new data in newdata. Object task is the task used during ⁠$train()⁠ and required for conversion of newdata. If the learner's ⁠$train()⁠ method has been called, there is a (size reduced) version of the training task stored in the learner. If the learner has been fitted via resample() or benchmark(), you need to pass the corresponding task stored in the ResampleResult or BenchmarkResult, respectively.

Usage

Learner$predict_newdata(newdata, task = NULL)

Arguments

Returns

Prediction.

Method reset()

Reset the learner, i.e. un-train by resetting the state.

Usage

Learner$reset()

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keeps the object in its previous state.

Method base_learner()

Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines or AutoTuner in mlr3tuning. Returns the Learner itself for regular learners.

Usage

Learner$base_learner(recursive = Inf)

Arguments

Returns

Learner.

Method encapsulate()

Sets the encapsulation method and fallback learner for the train and predict steps. There are currently four different methods implemented:

• "none": Just runs the learner in the current session and measures the elapsed time. Does not keep a log, output is printed directly to the console. Works well together with traceback().

• "try": Similar to "none", but catches error. Output is printed to the console and not logged.

• "evaluate": Uses the package evaluate to call the learner, measure time and do the logging.

• "callr": Uses the package callr to call the learner, measure time and do the logging. This encapsulation spawns a separate R session in which the learner is called. While this comes with a considerable overhead, it also guards your session from being teared down by segfaults.

The fallback learner is fitted to create valid predictions in case that either the model fitting or the prediction of the original learner fails. If the training step or the predict step of the original learner fails, the fallback is used completely to predict predictions sets. If the original learner only partially fails during predict step (usually in the form of missing to predict some observations or producing some ⁠NA`` predictions), these missing predictions are imputed by the fallback. Note that the fallback is always trained, as we do not know in advance whether prediction will fail. If the training step fails, the ⁠$model⁠field of the original learner is⁠NULL'.

Also see the section on error handling the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling

Usage

Learner$encapsulate(method, fallback = NULL)

Arguments

Returns

self (invisibly).

Method clone()

The objects of this class are cloneable with this method.

Usage

Learner$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: LearnerClassif, LearnerRegr, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Description

This Learner specializes Learner for classification problems:

• task_type is set to "classif".

• Creates Predictions of class PredictionClassif.

• Possible values for predict_types are:

  • "response": Predicts a class label for each observation in the test set.

  • "prob": Predicts the posterior probability for each class for each observation in the test set.

• Additional learner properties include:

  • "twoclass": The learner works on binary classification problems.

  • "multiclass": The learner works on multiclass classification problems.

Predefined learners can be found in the dictionary mlr_learners. Essential classification learners can be found in this dictionary after loading mlr3learners. Additional learners are implement in the Github package https://github.com/mlr-org/mlr3extralearners.

Super class

mlr3::Learner -> LearnerClassif

Methods

Public methods

• LearnerClassif$new()

• LearnerClassif$clone()

Method new()

Creates a new instance of this R6 class.

Usage

LearnerClassif$new(
  id,
  param_set = ps(),
  predict_types = "response",
  feature_types = character(),
  properties = character(),
  data_formats,
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerClassif$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: Learner, LearnerRegr, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Examples

# get all classification learners from mlr_learners:
lrns = mlr_learners$mget(mlr_learners$keys("^classif"))
names(lrns)

# get a specific learner from mlr_learners:
lrn = lrn("classif.rpart")
print(lrn)

# train the learner:
task = tsk("penguins")
lrn$train(task, 1:200)

# predict on new observations:
lrn$predict(task, 201:344)$confusion

Description

This Learner specializes Learner for regression problems:

• task_type is set to "regr".

• Creates Predictions of class PredictionRegr.

• Possible values for predict_types are:

  • "response": Predicts a numeric response for each observation in the test set.

  • "se": Predicts the standard error for each value of response for each observation in the test set.

  • "distr": Probability distribution as VectorDistribution object (requires package distr6, available via repository https://raphaels1.r-universe.dev).

Predefined learners can be found in the dictionary mlr_learners. Essential regression learners can be found in this dictionary after loading mlr3learners. Additional learners are implement in the Github package https://github.com/mlr-org/mlr3extralearners.

Super class

mlr3::Learner -> LearnerRegr

Active bindings

Methods

Public methods

• LearnerRegr$new()

• LearnerRegr$clone()

Method new()

Creates a new instance of this R6 class.

Usage

LearnerRegr$new(
  id,
  param_set = ps(),
  predict_types = "response",
  feature_types = character(),
  properties = character(),
  data_formats,
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerRegr$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: Learner, LearnerClassif, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Examples

# get all regression learners from mlr_learners:
lrns = mlr_learners$mget(mlr_learners$keys("^regr"))
names(lrns)

# get a specific learner from mlr_learners:
mlr_learners$get("regr.rpart")
lrn("classif.featureless")

Description

Marshaling is the process of processing the model of a trained Learner so it an be successfully serialized and deserialized. The naming is inspired by the marshal package and we plan to fully migrate to this package once it is on CRAN. The current implementation should therfore be considered as a temporary solution and is likely to change in the future.

The central functions (and the only methods that are used by mlr3 internally) are:

• the S3 generic marshal_model(model, inplace, ...). Which takes in a model and returns it in marshaled form. This means, that the resulting object can be serialized and de-serialzed without loss of information. If a model is serializable anyway, nothing has to be implemented and the generic will fall back to the default implementation of marshal_model, which is to return the object as-is. Otherwise, the marshaled object should be a list with named elements marshaled and packages, where the former contains the marshaled object, and the latter the package that contains the packages required to unmarshal. Most importantly, this list should contain the package that contains the unmarshal_model method. The returned object should have the classes of the original object with the suffix "_marshaled" appended and the root class should be set to "marshaled".

• the S3 generic ⁠unmarshal_model(model, inplace ...)⁠. Which takes in the marshaled model and returns it in unmarshaled form. The generic takes care that the packages specified during "marshal" are loaded, and errs if they are not availabe. Calling this function on a marshaled model should reconstruct the original model, i.e. unmarshal_model(marshal_model(x)) should return x. The default implementation of this generic returns x as-is.

• the function is_marshaled_model(model). This (helper) function returns TRUE if the model inherits from class "marshaled" and FALSE otherwise. Note that it is not guarateed that is_marshaled_model(marshal_model(x)) returns TRUE. This is because the default marshal_model(x) returns x as-is.

For both marshal_model and unmarshal_model, the inplace argument determines whether in-place marshaling should be performed. This is especially relevant in the context of references semantics. If inplace is FALSE, the original input should not be modified, otherwise this is allowed. Note that the input and output can still share references, even when inplace is FALSE.

Usage

learner_unmarshal(.learner, ...)

learner_marshal(.learner, ...)

learner_marshaled(.learner)

marshal_model(model, inplace = FALSE, ...)

unmarshal_model(model, inplace = FALSE, ...)

is_marshaled_model(model)

Arguments

Implementing Marshaling

In order to implement marshaling for a Learner, you need to overload the marshal_model and unmarshal_model methods for the class of the learner's model and tag the learner with the "marshal" property. To make marshaling accessible in an R6-manner, you should also add the public methods ⁠$marshal()⁠, ⁠$unmarshal()⁠ and the active binding ⁠$marshaled⁠. To make this as convenient as possible, the functions learner_marshal(.learner, ...), learner_unmarshal(.learner, ...) and learner_marshaled(.learner) are provided and can be called from the public methods.

You can verify whether you have correctly implemented marshaling by using the internal test helper expect_marshalable_learner(learner, task). This is also run by expect_learner() if a task is provided.

For a concrete example on how to implement marshaling, see LearnerClassifDebug.

Description

This is the abstract base class for measures like MeasureClassif and MeasureRegr.

Measures are classes tailored around two functions doing the work:

1. A function ⁠$score()⁠ which quantifies the performance by comparing the truth and predictions.

2. A function ⁠$aggregator()⁠ which combines multiple performance scores returned by ⁠$score()⁠ to a single numeric value.

In addition to these two functions, meta-information about the performance measure is stored.

Predefined measures are stored in the dictionary mlr_measures, e.g. classif.auc or time_train. Many of the measures in mlr3 are implemented in mlr3measures as ordinary functions.

A guide on how to extend mlr3 with custom measures can be found in the mlr3book.

Inheriting

For some measures (such as confidence intervals from mlr3inferr) it is necessary that a measure returns more than one value. In such cases it is necessary to overwrite the public methods ⁠$aggregate()⁠ and/or ⁠$score()⁠ to return a named numeric() where at least one of its names corresponds to the id of the measure itself.

Public fields

Active bindings

Methods

Public methods

• Measure$new()

• Measure$format()

• Measure$print()

• Measure$help()

• Measure$score()

• Measure$aggregate()

• Measure$clone()

Method new()

Creates a new instance of this R6 class.

Note that this object is typically constructed via a derived classes, e.g. MeasureClassif or MeasureRegr.

Usage

Measure$new(
  id,
  task_type = NA,
  param_set = ps(),
  range = c(-Inf, Inf),
  minimize = NA,
  average = "macro",
  aggregator = NULL,
  obs_loss = NULL,
  properties = character(),
  predict_type = "response",
  predict_sets = "test",
  task_properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_,
  trafo = NULL
)

Arguments

Method format()

Helper for print outputs.

Usage

Measure$format(...)

Arguments

Method print()

Printer.

Usage

Measure$print(...)

Arguments

Method help()

Opens the corresponding help page referenced by field ⁠$man⁠.

Usage

Measure$help()

Method score()

Takes a Prediction (or a list of Prediction objects named with valid predict_sets) and calculates a numeric score. If the measure if flagged with the properties "requires_task", "requires_learner", "requires_model" or "requires_train_set", you must additionally pass the respective Task, the (trained) Learner or the training set indices. This is handled internally during resample()/benchmark().

Usage

Measure$score(prediction, task = NULL, learner = NULL, train_set = NULL)

Arguments

Returns

numeric(1).

Method aggregate()

Aggregates multiple performance scores into a single score, e.g. by using the aggregator function of the measure.

Usage

Measure$aggregate(rr)

Arguments

Returns

numeric(1).

Method clone()

The objects of this class are cloneable with this method.

Usage

Measure$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-eval

• Package mlr3measures for the scoring functions. Dictionary of Measures: mlr_measures as.data.table(mlr_measures) for a table of available Measures in the running session (depending on the loaded packages).

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

Other Measure: MeasureClassif, MeasureRegr, MeasureSimilarity, mlr_measures, mlr_measures_aic, mlr_measures_bic, mlr_measures_classif.costs, mlr_measures_debug_classif, mlr_measures_elapsed_time, mlr_measures_internal_valid_score, mlr_measures_oob_error, mlr_measures_regr.rsq, mlr_measures_selected_features

Description

This measure specializes Measure for classification problems:

• task_type is set to "classif".

• Possible values for predict_type are "response" and "prob".

Predefined measures can be found in the dictionary mlr_measures. The default measure for classification is classif.ce.

Super class

mlr3::Measure -> MeasureClassif

Methods

Public methods

• MeasureClassif$new()

• MeasureClassif$clone()

Method new()

Creates a new instance of this R6 class.

Usage

MeasureClassif$new(
  id,
  param_set = ps(),
  range,
  minimize = NA,
  average = "macro",
  aggregator = NULL,
  properties = character(),
  predict_type = "response",
  predict_sets = "test",
  task_properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

MeasureClassif$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-eval

• Package mlr3measures for the scoring functions. Dictionary of Measures: mlr_measures as.data.table(mlr_measures) for a table of available Measures in the running session (depending on the loaded packages).

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

Other Measure: Measure, MeasureRegr, MeasureSimilarity, mlr_measures, mlr_measures_aic, mlr_measures_bic, mlr_measures_classif.costs, mlr_measures_debug_classif, mlr_measures_elapsed_time, mlr_measures_internal_valid_score, mlr_measures_oob_error, mlr_measures_regr.rsq, mlr_measures_selected_features

Description

This measure specializes Measure for regression problems:

• task_type is set to "regr".

• Possible values for predict_type are "response", "se" and "distr".

Predefined measures can be found in the dictionary mlr_measures. The default measure for regression is regr.mse.

Super class

mlr3::Measure -> MeasureRegr

Methods

Public methods

• MeasureRegr$new()

• MeasureRegr$clone()

Method new()

Creates a new instance of this R6 class.

Usage

MeasureRegr$new(
  id,
  param_set = ps(),
  range,
  minimize = NA,
  average = "macro",
  aggregator = NULL,
  properties = character(),
  predict_type = "response",
  predict_sets = "test",
  task_properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments