| Title: | Measuring Contributions of Component Models to Ensemble Forecast Accuracy |
|---|---|
| Description: | Provides metrics for quantifying the contribution of individual component models to the predictive accuracy of ensemble forecasts. The package implements the Leave-One-Model-Out (LOMO) and Leave-All-Subset-of-One-Model-Out (LASOMO) model importance metrics, enabling users to assess the relative importance of component models and better understand the performance of ensemble forecasting systems. Methods are described in Kim et al. (2026) <doi:10.1016/j.ijforecast.2025.12.006>. |
| Authors: | Minsu Kim [aut, cre, cph] (ORCID: <https://orcid.org/0009-0008-4637-3589>), Li Shandross [aut, ctb] (ORCID: <https://orcid.org/0009-0008-1348-1954>), Zhian Kamvar [ctb] (ORCID: <https://orcid.org/0000-0003-1458-7108>), Nicholas Reich [aut] (ORCID: <https://orcid.org/0000-0003-3503-9899>), Evan Ray [aut] |
| Maintainer: | Minsu Kim <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-07-16 17:07:09 UTC |
| Source: | https://github.com/cran/modelimportance |
Aggregate model importance scores across tasks to compute overall importance for each model
## S3 method for class 'model_imp_tbl' aggregate( x, by = "model_id", na_action = c("drop", "worst", "average"), fun = mean, ... )## S3 method for class 'model_imp_tbl' aggregate( x, by = "model_id", na_action = c("drop", "worst", "average"), fun = mean, ... )
x |
An object of class |
by |
A character vector with column names specifying the grouping
variable(s) for summarization. Default is |
na_action |
A character string specifying how to handle
|
fun |
A function used to summarize importance scores.
Default is |
... |
Additional arguments passed to the summary function |
This method extends stats::aggregate for objects of class model_imp_tbl.
A data frame with columns model_id and importance_score_<fun>,
where <fun> is the name of the summary function
used (e.g., importance_score_mean when fun = mean).
The output is sorted in descending order of the summary importance scores.
A subset of hubExamples::forecast_outputs filtered to quantile output type
and location "25" (Massachusetts), covering the 2022/23 influenza season
(from "2022-11-19" to "2023-01-07").
forecast_data_exampleforecast_data_example
A data frame (model_out_tbl) with 168 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_outputs filtered to median output type,
location "25" (Massachusetts), horizon 1, and target end date 2022-12-24.
forecast_data_ma_h1forecast_data_ma_h1
A data frame (model_out_tbl) with 3 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_outputs filtered to median output type
and two target end dates (2022-11-26, 2022-12-10).
forecast_data_rawforecast_data_raw
A data frame (model_out_tbl) with 12 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
We measure each ensemble component model's contribution to the ensemble prediction accuracy for each model task.
This function requires that one column represent the forecast date (or
a date from which each forecast originates or is made in reference to) and
that column be named one of forecast_date, origin_date, and
reference_date.
For each output_type, the corresponding scoring rule applied to calculate
the importance is as follows.
| Output Type | Scoring Rule | Description |
| mean | se_point | evaluate using the squared error |
| median | ae_point | evaluate using the absolute error |
| quantile | wis | evaluate using the weighted interval score |
| pmf | log_score | evaluate using the logarithm of the probability assigned to the true outcome |
model_importance( forecast_data, oracle_output_data, ensemble_fun = c("simple_ensemble", "linear_pool"), importance_algorithm = c("lomo", "lasomo"), subset_wt = c("equal", "perm_based"), min_log_score = -10, ... )model_importance( forecast_data, oracle_output_data, ensemble_fun = c("simple_ensemble", "linear_pool"), importance_algorithm = c("lomo", "lasomo"), subset_wt = c("equal", "perm_based"), min_log_score = -10, ... )
forecast_data |
A data.frame with the predictions that is or can be
coerced to a |
oracle_output_data |
Ground truth data for the variables that are used to define modeling targets. This data must follow the oracle output format. See 'Details'. |
ensemble_fun |
A character string specifying a ensemble method, either
"simple_ensemble" or "linear_pool";
|
importance_algorithm |
A character string specifying algorithm for model
importance calculation; |
subset_wt |
A character string specifying method for assigning weight
to subsets when using
|
min_log_score |
A numeric value specifying a minimum threshold for log
scores for the |
... |
Optional arguments passed to |
The oracle_output_data is a data frame that contains the ground truth
values for the variables used to define modeling targets. It is referred to
as "oracle" because it is formatted as if an oracle made a perfect point
prediction equal to the truth. This data must follow the oracle output format
defined in the hubverse standard, which includes independent task ID columns
(e.g., location, target_date), the output_type column specifying the
output type of the predictions and an oracle_value column for the observed
values. As in the forecast data, if the output_type is either "quantile"
or "pmf", the output_type_id column is often required to provide further
identifying information.
The model_out_tbl and oracle_output_data must have the same task ID
columns and output_type, including output_type_id if necessary, which are
used to match the predictions with the ground truth data.
Additional argument in ... is agg_fun, which is a character string name
for a function specifying aggregation method of component model outputs.
Default is mean, indicating that equally weighted mean is calculated
across all component model outputs for each unique output_type_id.
This can be median or a custom function (e.g., geometric_mean. Details
can be found in
https://hubverse-org.github.io/hubEnsembles/articles/hubEnsembles.html).
This function uses the furrr and future for parallelization.
To enable parallel execution, please set a parallel backend, e.g., via
future::plan().
A model_imp_tbl S3 class object with columns model_id,
reference_date, output_type, and importance, along with any task ID
columns (e.g., location, horizon, and target_end_date) present in the
input forecast_data.
Note that reference_date is used as the name for the forecast date column,
regardless of its original name in the input forecast_data.
Optional progress bars are displayed via the progressr package when it is
installed and the session is interactive.
If progressr is not installed, the function will run without progress bars.
To enable progress bars,
progressr::handlers(global = TRUE)
model importance scores calculated for individual prediction tasks can be
aggregated across multiple tasks to obtain an overall importance score for
each model via the aggregate() method for model_imp_tbl objects.
Users can summarize the task-level scores with specified summary statistics
of interest (e.g., mean, median, quantiles) for a more comprehensive view of
model importance.
See aggregate.model_imp_tbl() for more details on how to use this method.
library(dplyr) forecast_data <- forecast_data_raw |> dplyr::filter( location == "25", horizon == 1 ) target_data <- target_data_raw |> dplyr::filter( target_end_date %in% unique(forecast_data$target_end_date), location == "25" ) # Example with the default arguments. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal" ) # Example with the additional argument in `...`. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal", agg_fun = median )library(dplyr) forecast_data <- forecast_data_raw |> dplyr::filter( location == "25", horizon == 1 ) target_data <- target_data_raw |> dplyr::filter( target_end_date %in% unique(forecast_data$target_end_date), location == "25" ) # Example with the default arguments. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal" ) # Example with the additional argument in `...`. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal", agg_fun = median )
Print method for model importance score table
## S3 method for class 'model_imp_tbl' print(x, ...)## S3 method for class 'model_imp_tbl' print(x, ...)
x |
An object of class |
... |
Additional arguments passed to the print method. |
x, returned invisibly. Called for its side effect of printing
the model importance score table (a model_imp_tbl object, i.e., a
data frame of per-task importance scores by model) to the console.
Print method for summary of model importance score table
## S3 method for class 'summary.model_imp_tbl' print(x, ...)## S3 method for class 'summary.model_imp_tbl' print(x, ...)
x |
An object of class |
... |
Additional arguments passed to the print method. |
x, returned invisibly. Called for its side effect of printing
a human-readable summary (number of models and tasks, and the top-scoring
model for a subset of tasks) to the console.
Summary method for model importance score table
## S3 method for class 'model_imp_tbl' summary(object, ...)## S3 method for class 'model_imp_tbl' summary(object, ...)
object |
An object of class |
... |
Additional arguments passed to the print method. |
A list of class summary.model_imp_tbl with four elements:
A character vector of the unique model IDs present in
object.
A data frame with one row per unique prediction task
(one column per task ID, e.g. location, horizon,
target_end_date).
A data frame with one row per model and columns
model_id, n_tasks (number of tasks the model was scored on),
min_importance, max_importance, and n_NA (number of tasks with a
missing importance score).
A data frame with one row per prediction task and
columns for the task ID(s), top_model (the model with the highest
importance score for that task), and max_score (that model's
importance score).
This object is printed via print.summary.model_imp_tbl.
A subset of hubExamples::forecast_target_ts matching the locations and
target end dates in forecast_data_example.
target_data_exampletarget_data_example
A data frame with 8 rows and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_target_ts for location "25"
(Massachusetts) and target end date 2022-12-24.
target_data_matarget_data_ma
A data frame with 1 row and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_target_ts matching the locations and
target end dates in forecast_data_raw, with the observed values column
renamed to oracle_value.
target_data_rawtarget_data_raw
A data frame with 4 rows and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).