Package 'modelcardr'

Audit predictions from a completed model

Description

Evaluates predictions without retraining or requiring the fitted model. For binary classification it calculates bootstrap confidence intervals, threshold trade-offs, ROC and precision-recall curves, probability calibration, subgroup performance, and automatic warning rules.

Usage

.audit_binary_predictions(
  truth,
  probability = NULL,
  estimate = NULL,
  positive = NULL,
  threshold = 0.5,
  thresholds = seq(0.05, 0.95, by = 0.01),
  groups = NULL,
  bootstrap = 500L,
  conf_level = 0.95,
  seed = NULL,
  threshold_objective = c("balanced_accuracy", "f1", "youden", "cost"),
  false_positive_cost = 1,
  false_negative_cost = 1,
  calibration_bins = 10L,
  reference_probability = NULL,
  drift_bins = 10L,
  subgroup_min_n = 30L,
  title = "Prediction Audit",
  requirements = NULL
)

Arguments

truth	Observed outcomes.
probability	Numeric probability of the positive class.
estimate	Optional predicted classes. When omitted, classes are created from probability and threshold.
positive	Label of the positive class.
threshold	Threshold used for the main confusion matrix and metrics.
thresholds	Candidate thresholds to evaluate.
groups	Optional subgroup data frame, named list, or vector.
bootstrap	Number of stratified bootstrap samples for confidence intervals. Use zero to disable.
conf_level	Confidence level for bootstrap percentile intervals.
seed	Optional random seed used for bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.
threshold_objective	Criterion for the recommended threshold: "balanced_accuracy", "f1", "youden", or "cost".
false_positive_cost	Relative cost of a false positive.
false_negative_cost	Relative cost of a false negative.
calibration_bins	Number of probability bins used in the calibration table.
reference_probability	Optional probabilities from an earlier or reference population for prediction-distribution drift checks.
drift_bins	Number of bins used for the population stability index.
subgroup_min_n	Subgroups below this size are flagged.
title	Report title.
requirements	Optional pass/fail criteria created by audit_requirements().

Value

An object of class model_audit.

---

Add data documentation

Description

Add data documentation

Usage

add_data_description(
  card,
  source = NULL,
  collection_period = NULL,
  population = NULL,
  sampling = NULL,
  exclusions = NULL,
  preprocessing = NULL,
  license = NULL,
  notes = NULL
)

Arguments

card	A model card.
source	Data source.
collection_period	Collection period.
population	Population represented by the data.
sampling	Sampling method.
exclusions	Exclusion criteria.
preprocessing	Important preprocessing steps.
license	Data license or access restrictions.
notes	Additional notes.

Value

The updated card.

---

Add known limitations

Description

Add known limitations

Usage

add_limitations(card, ...)

Arguments

card	A model card.
...	Character vectors containing limitations.

Value

The updated card.

---

Add intended-use information

Description

Add intended-use information

Usage

add_purpose(
  card,
  intended_use,
  users = NULL,
  out_of_scope = NULL,
  decision_context = NULL,
  owner = NULL
)

Arguments

card	A model card.
intended_use	Intended use of the model.
users	Intended users.
out_of_scope	Unsupported or prohibited uses.
decision_context	How predictions enter a decision process.
owner	Responsible person or team.

Value

The updated card.

---

Run model-specific diagnostic checks

Description

Diagnostics are screening evidence, not certification that assumptions hold.

Usage

audit_assumptions(card)

Arguments

card	A model card.

Value

The updated card.

---

Audit subgroup performance and fairness

Description

This function reports subgroup metrics and disparities. It does not declare a model to be fair or unfair; interpretation depends on the use case.

Usage

audit_fairness(card, groups = NULL, min_n = 30L)

Arguments

card	A model card.
groups	Optional character vector naming stored sensitive features. Defaults to all sensitive features supplied to model_card().
min_n	Minimum subgroup size before a subgroup is marked as small.

Value

The updated card.

---

Calculate model performance

Description

Calculate model performance

Usage

audit_performance(card)

Arguments

card	A model card.

Value

The updated card.

---

Audit Predictions from a Completed Model

Description

Audits completed binary-classification, multiclass-classification, or regression predictions without requiring the fitted model.

Usage

audit_predictions(
  truth,
  probability = NULL,
  estimate = NULL,
  task = c("auto", "binary_classification", "multiclass_classification", "regression"),
  positive = NULL,
  class_names = NULL,
  threshold = 0.5,
  thresholds = seq(0.05, 0.95, by = 0.01),
  groups = NULL,
  bootstrap = 500L,
  conf_level = 0.95,
  seed = NULL,
  threshold_objective = c("balanced_accuracy", "f1", "youden", "cost"),
  false_positive_cost = 1,
  false_negative_cost = 1,
  calibration_bins = 10L,
  reference_probability = NULL,
  drift_bins = 10L,
  subgroup_min_n = 30L,
  title = "Prediction Audit",
  requirements = NULL
)

Arguments

truth	Observed outcomes.
probability	A binary positive-class probability vector or multiclass probability matrix.
estimate	Predicted classes or numeric regression estimates.
task	Task type, or "auto".
positive	Positive class for binary classification.
class_names	Optional multiclass probability-column names.
threshold	Binary decision threshold.
thresholds	Candidate binary thresholds.
groups	Optional subgroup variables.
bootstrap	Number of bootstrap samples.
conf_level	Bootstrap confidence level.
seed	Optional random seed used for bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.
threshold_objective	Binary threshold objective.
false_positive_cost	Relative binary false-positive cost.
false_negative_cost	Relative binary false-negative cost.
calibration_bins	Number of calibration bins.
reference_probability	Optional binary reference probabilities for drift.
drift_bins	Number of drift bins.
subgroup_min_n	Minimum subgroup size before a warning.
title	Audit title.
requirements	Optional criteria from audit_requirements().

Value

An object of class model_audit.

Examples

audit_predictions(c(1, 2, 3), estimate = c(1.1, 1.9, 3.2), task = "regression", bootstrap = 0)

---

Define prediction-audit requirements

Description

Creates optional acceptance criteria for binary classification, multiclass classification, and regression audits. Requirements that cannot be evaluated from the supplied data are reported as INCONCLUSIVE.

Usage

audit_requirements(
  min_n = NULL,
  min_accuracy = NULL,
  min_balanced_accuracy = NULL,
  min_sensitivity = NULL,
  min_specificity = NULL,
  min_precision = NULL,
  min_f1 = NULL,
  min_roc_auc = NULL,
  min_average_precision = NULL,
  min_macro_f1 = NULL,
  min_weighted_f1 = NULL,
  min_macro_roc_auc = NULL,
  min_minimum_class_recall = NULL,
  min_r_squared = NULL,
  max_false_positive_rate = NULL,
  max_false_negative_rate = NULL,
  max_brier_score = NULL,
  max_log_loss = NULL,
  max_multiclass_log_loss = NULL,
  max_multiclass_brier_score = NULL,
  max_rmse = NULL,
  max_mae = NULL,
  max_absolute_bias = NULL,
  max_mape = NULL,
  max_smape = NULL,
  max_expected_calibration_error = NULL,
  calibration_slope_range = NULL,
  max_threshold_difference = NULL,
  max_subgroup_sensitivity_gap = NULL,
  max_subgroup_specificity_gap = NULL,
  max_subgroup_selection_rate_gap = NULL,
  max_subgroup_accuracy_gap = NULL,
  max_subgroup_macro_f1_gap = NULL,
  max_subgroup_rmse_gap = NULL,
  max_subgroup_mae_gap = NULL,
  max_population_stability_index = NULL,
  use_confidence_bounds = TRUE
)

Arguments

min_n	Minimum required number of evaluation observations. Use NULL to disable this requirement.
min_accuracy	Minimum acceptable classification accuracy. Use NULL to disable this requirement.
min_balanced_accuracy	Minimum acceptable balanced accuracy. Use NULL to disable this requirement.
min_sensitivity	Minimum acceptable binary-classification sensitivity. Use NULL to disable this requirement.
min_specificity	Minimum acceptable binary-classification specificity. Use NULL to disable this requirement.
min_precision	Minimum acceptable binary-classification precision. Use NULL to disable this requirement.
min_f1	Minimum acceptable binary-classification F1 score. Use NULL to disable this requirement.
min_roc_auc	Minimum acceptable binary-classification ROC AUC. Use NULL to disable this requirement.
min_average_precision	Minimum acceptable average precision for binary classification. Use NULL to disable this requirement.
min_macro_f1	Minimum acceptable macro F1 score for multiclass classification. Use NULL to disable this requirement.
min_weighted_f1	Minimum acceptable weighted F1 score for multiclass classification. Use NULL to disable this requirement.
min_macro_roc_auc	Minimum acceptable macro ROC AUC for multiclass classification. Use NULL to disable this requirement.
min_minimum_class_recall	Minimum acceptable recall for the worst-performing class in multiclass classification. Use NULL to disable this requirement.
min_r_squared	Minimum acceptable R-squared value for regression. Use NULL to disable this requirement.
max_false_positive_rate	Maximum acceptable false-positive rate for binary classification. Use NULL to disable this requirement.
max_false_negative_rate	Maximum acceptable false-negative rate for binary classification. Use NULL to disable this requirement.
max_brier_score	Maximum acceptable Brier score for binary classification. Use NULL to disable this requirement.
max_log_loss	Maximum acceptable log loss for binary classification. Use NULL to disable this requirement.
max_multiclass_log_loss	Maximum acceptable log loss for multiclass classification. Use NULL to disable this requirement.
max_multiclass_brier_score	Maximum acceptable Brier score for multiclass classification. Use NULL to disable this requirement.
max_rmse	Maximum acceptable root mean squared error for regression. Use NULL to disable this requirement.
max_mae	Maximum acceptable mean absolute error for regression. Use NULL to disable this requirement.
max_absolute_bias	Maximum acceptable absolute mean bias for regression. Use NULL to disable this requirement.
max_mape	Maximum acceptable mean absolute percentage error for regression. Use NULL to disable this requirement.
max_smape	Maximum acceptable symmetric mean absolute percentage error for regression. Use NULL to disable this requirement.
max_expected_calibration_error	Maximum acceptable expected calibration error for classification. Use NULL to disable this requirement.
calibration_slope_range	Numeric vector of length two containing the minimum and maximum acceptable binary calibration slope. Use NULL to disable this requirement.
max_threshold_difference	Maximum acceptable absolute difference between the selected binary threshold and the package-recommended threshold. Use NULL to disable this requirement.
max_subgroup_sensitivity_gap	Maximum acceptable difference between the highest and lowest subgroup sensitivities. Use NULL to disable this requirement.
max_subgroup_specificity_gap	Maximum acceptable difference between the highest and lowest subgroup specificities. Use NULL to disable this requirement.
max_subgroup_selection_rate_gap	Maximum acceptable difference between the highest and lowest subgroup selection rates. Use NULL to disable this requirement.
max_subgroup_accuracy_gap	Maximum acceptable difference between the highest and lowest subgroup accuracies. Use NULL to disable this requirement.
max_subgroup_macro_f1_gap	Maximum acceptable difference between the highest and lowest subgroup macro F1 scores. Use NULL to disable this requirement.
max_subgroup_rmse_gap	Maximum acceptable difference between the highest and lowest subgroup RMSE values. Use NULL to disable this requirement.
max_subgroup_mae_gap	Maximum acceptable difference between the highest and lowest subgroup MAE values. Use NULL to disable this requirement.
max_population_stability_index	Maximum acceptable population stability index for prediction drift. Use NULL to disable this requirement.
use_confidence_bounds	Logical. When TRUE, minimum requirements use lower confidence bounds and maximum requirements use upper confidence bounds when bootstrap intervals are available. When FALSE, point estimates are used.

Value

An object of class audit_requirements containing the configured audit criteria.

Examples

rules <- audit_requirements(
  min_n = 100,
  min_accuracy = 0.75,
  min_roc_auc = 0.80,
  max_log_loss = 0.60
)

print(rules)

---

Apply pass/fail requirements to a prediction audit

Description

Evaluates configured acceptance criteria and adds a rule table plus an overall PASS, FAIL, or INCONCLUSIVE result to a model audit. Missing evidence is marked INCONCLUSIVE rather than passing automatically.

Usage

check_requirements(audit, requirements)

Arguments

audit	An object of class model_audit.
requirements	An object created by audit_requirements().

Value

The updated model_audit object.

Examples

truth <- c("no", "no", "yes", "yes")
probability <- c(0.1, 0.2, 0.8, 0.9)
audit <- audit_predictions(truth, probability, positive = "yes", bootstrap = 0)
rules <- audit_requirements(min_roc_auc = 0.80, min_sensitivity = 0.80)
audit <- check_requirements(audit, rules)
audit$overall_status
audit$rule_results

---

Compare two model cards

Description

Compare two model cards

Usage

compare_cards(old, new)

Arguments

old	Older model card.
new	Newer model card.

Value

A list containing metadata and metric comparisons.

---

Compare two sets of binary-classification predictions

Description

Uses paired stratified bootstrap resampling so each model is evaluated on the same resampled observations.

Usage

compare_predictions(
  truth,
  probability_a,
  probability_b,
  positive = NULL,
  threshold = 0.5,
  bootstrap = 1000L,
  conf_level = 0.95,
  seed = NULL,
  model_names = c("model_a", "model_b")
)

Arguments

truth	Observed outcomes.
probability_a	Positive-class probabilities from model A.
probability_b	Positive-class probabilities from model B.
positive	Positive-class label.
threshold	Threshold applied to both models.
bootstrap	Number of paired bootstrap samples.
conf_level	Confidence level for differences.
seed	Optional random seed used for paired bootstrap resampling. The default NULL uses the current random-number generator state without resetting it.
model_names	Two model names.

Value

An object of class prediction_comparison.

---

Create a model card

Description

Creates a structured model-card object for a regression or binary classification model. Predictions may be generated from model, or supplied directly through estimate and probability.

Usage

model_card(
  model = NULL,
  evaluation_data = NULL,
  truth,
  estimate = NULL,
  probability = NULL,
  task = c("auto", "regression", "binary_classification"),
  positive = NULL,
  threshold = 0.5,
  title = NULL,
  version = "0.1.0",
  sensitive_features = NULL,
  metadata = list()
)

Arguments

model	A fitted model object, or NULL for prediction-only cards.
evaluation_data	A data frame used for evaluation.
truth	Outcome vector or the name of its column in evaluation_data.
estimate	Predicted values/classes or a column name.
probability	Positive-class probabilities or a column name.
task	One of "auto", "regression", "binary_classification", or "multiclass_classification".
positive	Positive outcome label for classification.
threshold	Probability threshold used to create class predictions.
title	Card title.
version	Model/card version.
sensitive_features	Character vector of columns to retain for subgroup evaluation. Raw predictor columns are not stored.
metadata	Named list of extra metadata.

Value

An object of class model_card.

---

Compatibility Interface for Prediction Audits

Description

Calls audit_predictions() for all supported prediction tasks.

Usage

model_card_from_predictions(
  truth,
  estimate = NULL,
  probability = NULL,
  task = c("auto", "regression", "binary_classification", "multiclass_classification"),
  positive = NULL,
  class_names = NULL,
  threshold = 0.5,
  title = "Prediction Audit",
  groups = NULL,
  bootstrap = 500L,
  ...
)

Arguments

truth	Observed outcomes.
estimate	Predicted classes or numeric estimates.
probability	Binary probability vector or multiclass probability matrix.
task	Task type.
positive	Binary positive class.
class_names	Optional multiclass probability-column names.
threshold	Binary classification threshold.
title	Audit title.
groups	Optional subgroup variables.
bootstrap	Bootstrap samples.
...	Additional arguments passed to audit_predictions().

Value

A model_audit object.

---

Plot a prediction audit

Description

Plot a prediction audit

Usage

## S3 method for class 'model_audit'
plot(x, type = NULL, metric = "balanced_accuracy", ...)

Arguments

x	A model_audit object.
type	Plot type. Binary audits support "roc", "pr", "calibration", and "threshold"; multiclass audits support "calibration"; regression audits support "actual_predicted", "residuals", and "residual_histogram".
metric	Metric shown for a binary threshold plot.
...	Additional graphical arguments.

Value

The input model_audit object, returned invisibly. The function is called for its side effect of drawing the selected diagnostic plot.

---

Print audit requirements

Description

Print audit requirements

Usage

## S3 method for class 'audit_requirements'
print(x, ...)

Arguments

x	An audit_requirements object.
...	Unused.

Value

The input audit_requirements object, returned invisibly. The function is called for its side effect of printing the configured audit criteria and whether confidence bounds are used.

---

Print a prediction audit

Description

Print a prediction audit

Usage

## S3 method for class 'model_audit'
print(x, ...)

Arguments

x	A model_audit object.
...	Unused.

Value

The input model_audit object, returned invisibly. The function is called for its side effect of printing a concise overview of the audit, including its task, sample size, selected metrics, status, and number of findings.

---

Print a model card

Description

Print a model card

Usage

## S3 method for class 'model_card'
print(x, ...)

Arguments

x	A model_card object.
...	Unused.

Value

The input model_card object, returned invisibly. The function is called for its side effect of printing the card title, version, task, model class, evaluation sample size, and number of performance metrics.

---

Print a prediction comparison

Description

Print a prediction comparison

Usage

## S3 method for class 'prediction_comparison'
print(x, ...)

Arguments

x	A prediction_comparison object.
...	Unused.

Value

The input prediction_comparison object, returned invisibly. The function is called for its side effect of printing the compared model names, evaluation sample size, and selected comparison metrics.

---

Render a Prediction Audit

Description

Renders task-specific evidence and omits sections that do not apply.

Usage

render_audit(audit, output_file, format = NULL, detailed = FALSE)

Arguments

audit	A model_audit object.
output_file	Destination HTML, Markdown, or JSON file.
format	Optional explicit output format.
detailed	Whether to include detailed diagnostic tables.

Value

The output path, invisibly.

---

Render a model card

Description

Render a model card

Usage

render_card(card, output_file, format = NULL)

Arguments

card	A model card.
output_file	Destination file ending in .md, .html, or .json.
format	Optional explicit format: "markdown", "html", or "json".

Value

The normalized output path, invisibly.

---

Summarize a prediction audit

Description

Summarize a prediction audit

Usage

## S3 method for class 'model_audit'
summary(object, ...)

Arguments

object	A model_audit object.
...	Unused.

Value

A named list summarizing the audit. It contains the audit title, task, evaluation-data information, overall and class-level metrics, recommended threshold, calibration summary, subgroup disparities, diagnostics, requirement-checking results, findings, and risk warnings. Components that are not applicable to the audit task may be NULL.

---

Summarize a model card

Description

Summarize a model card

Usage

## S3 method for class 'model_card'
summary(object, ...)

Arguments

object	A model_card object.
...	Unused.

Value

A named list containing the model-card title, version, task, model metadata, intended purpose, performance results, fairness summary, assumption checks, limitations, and completeness-validation results.

---

Validate model-card completeness

Description

Validate model-card completeness

Usage

validate_card(card)

Arguments

card	A model card.

Value

A data frame of validation checks.

---