The CRAN pre-test for 0.5.0 asked for three things, and profiling to fix the third one found a real performance defect.
.workflow_shap() spent 87% of a run_workflow() call inside
kernelshap(), with the sizes fixed at explain_shap()'s defaults (100 rows
explained against a 100-row background). A complete report therefore took about
a minute on a 800-patient cohort, almost all of it in one explanation.
n_explain and bg_n are now part of the specification:
uniml_spec()$explain defaults to 40 and 40, and
set_explain(spec, n_explain = , bg_n = ) changes them. The same workflow now
takes about 30 seconds instead of 90, and a paper-quality explanation is one
argument away. set_explain(spec, model = "none") skips SHAP entirely.
Measured on the cardiovascular example (2 learners, 3 data sets): 87 s → 29 s.
inst/WORDLIST declares "SHAP", "interpretability", and "multiclass", which the
incoming spell check flagged in DESCRIPTION.
The bilingual link at the top of README.md pointed at README.zh.md, which is
excluded from the built package, so the relative URI was broken inside the
tarball. Both READMEs now link to the GitHub copy, which resolves everywhere.
Vignettes run smaller bootstrap and SHAP samples (ci_R 100/200 → 25/40, and
20 to 30 explained rows against a 14 to 20 row background), so R CMD check
finishes inside CRAN's ten-minute budget. Every vignette still runs the whole
workflow; none of them is eval = FALSE and none of the numbers is typed in by
hand.
The end-to-end run on the largest shipped cohort (test-cvd-workflow.R, by far
the slowest test) is marked skip_on_cran(). It covers assembly, ranking,
selection, and prediction, all of which test-workflow.R covers on small
synthetic data that does run on CRAN, and it runs on three platforms in CI on
every commit.
Recalibration for multiclass models, which 0.4.0 could measure but not fix.
calibrate_probabilities() gains a multiclass method. It fits the single scalar
T that makes a model's reported probabilities match how often it is right, and
returns softmax(log(p) / T): T > 1 softens an over-confident model, T < 1
sharpens an under-confident one (Guo et al. 2017).
Because the division is monotone and applied to every class in the same way, temperature scaling cannot change which class the model ranks first. The confusion matrix, the accuracy, and every threshold metric come out identical; only the probabilities attached to them move.
T is fitted by minimising the log loss, and T = 1 is kept whenever it is at
least as good, so the fitted scalar never degrades the data it was fitted on. It
is a bounded one-dimensional search, not a matrix fit, so a few hundred patients
is enough.
calibrated is cross-fitted (each fold's scalar comes from the other folds), so
metrics_before / metrics_after are an honest before/after. That honesty has a
cost, stated in the documentation: on a model that was already calibrated the
cross-fitted log loss can come out a hair worse, because each fold's scalar is a
noisy estimate of 1.
new_probs / new_labels apply a scalar fitted on the validation set to the
test set, with the before/after table reported on that set. method = "none"
returns the raw probabilities, as in the binary path.
A probability matrix can be passed directly; a matrix with fewer than three class columns is refused with a pointer to the binary calibrators, and a hard 0/1 prediction is clamped rather than breaking the fit.
set_evaluation(spec, calibration = ) is now task-type aware: a multiclass
specification accepts "temperature" or "none", a binary one accepts
"platt", "platt_segment", "isotonic", or "none". The two sets are
disjoint, so a specification cannot ask for a method its task type does not have.
Deliberately not implemented: Dirichlet calibration (Kull et al. 2019), the
stronger method, fits K^2 + K parameters, can change the predicted class, and
is not identifiable on the sample size this package targets. The documentation
says so where the method would be expected.
Multiclass calibration, which the package previously declined to compute because "calibration" means more than one thing once a model outputs a probability for every class. Both meanings are implemented and named.
multiclass_calibration() reports calibration under two conventions:
top-label (the largest probability of each row against whether the
predicted class was right, Guo et al. 2017) and class-wise (a one-vs-rest
comparison for every class, averaged without weighting, Kull et al. 2019). It
returns the summary, a per-class table, and the long reliability table the
figure is drawn from.calc_metrics() on a probability matrix gains ece (expected calibration
error, top-label), mce (maximum calibration error), and ece_classwise, plus
per-class ece/mce in per_class. calc_ci() bootstraps all three.plot_calibration() method for uniml_multiclass: scope = "top_label",
"class", or "both". The caption carries ECE, MCE, the level bias and the
class-wise ECE, and annotate = FALSE removes it.mean confidence - accuracy.calibration_metrics() now refuses a probability matrix with a message that
names the two functions to use instead, rather than failing inside the binary
input checks.calc_metrics() gains ece and mce for binary outcomes too, so the two
paths expose the same numbers. The binary ECE bins the probability of the
event, which is not the same estimand as the multiclass top-label ECE; the
positive class of multiclass_calibration()$per_class is the bridge, and a test
pins the equality..binned_calibration(), so the
calibration table used by the workflow and calc_metrics() cannot drift apart.export_results() writes calibration/summary.csv whenever the task type
reports a per-time-point calibration summary. unimlsurv uses it for the ICI
and the recalibration coefficients; without this the numbers existed only
inside R.The last two gaps from the reporting audit: a precision-recall figure, and the macro/micro split for multiclass discrimination.
plot_pr_curve() draws the precision-recall curve with the prevalence as a
reference line, for the imbalanced outcomes where the ROC curve flatters a model.
The curve and the label come from PRROC::pr.curve(), the same code
mlr3measures::prauc() calls, so the figure and calc_metrics()$pr_auc cannot
disagree.auc was documented as a "macro one-vs-rest AUC". It is not:
mlr3measures::mauc_au1u is the Hand and Till multiclass AUC, the mean over
ordered class pairs computed on class i's own score column. The documentation,
the vignette table, and the tests said the wrong thing; they now say the right
one.auc_ovr: the macro one-vs-rest AUC, the unweighted mean of the per-class
one-vs-rest AUCs. This is the number a paper means by "macro-average AUC".micro_auc: all (patient, class) pairs pooled into one binary problem.
Micro high with macro low means the model is carried by the common classes.per_class$auc: one-vs-rest AUC for each class, so a class that the model
cannot separate is visible instead of averaged away.micro_specificity. There is deliberately no micro precision, recall, or F1:
for a single-label multiclass outcome all three are identically the accuracy, and
a test pins that identity.plot_roc() for multiclass predictions adds the micro-average curve beside the
per-class curves and puts all three AUCs in the subtitle; micro = FALSE draws
only the per-class curves.set_feature_selection(methods = "none") is now a documented choice: it keeps
every candidate instead of running a selector. A task type whose outcome is not
a class label (a survival time and status, say) needs this, because every other
method screens the outcome column. select_features() and the workflow treat
"none" and an empty method vector the same way.select_features() treats an empty methods vector as "no selection" and keeps
every candidate instead of failing in combine_selections(). That is what a task
type with no selection method of its own needs, and it makes the behaviour the
same whether the specification is read by the workflow or by hand.The five reporting domains a clinical prediction model is judged on are now all
reachable from calc_metrics(), and a fitted result can be handed to someone else
without handing them the fitting code.
calibration_metrics() returns the calibration intercept, the calibration
slope, and calibration-in-the-large, with percentile bootstrap intervals. They
come from one logistic recalibration, logit(observed) = a + b * logit(predicted),
which is the convention the calibration literature uses, not the slope of the
binned reliability line the package computed internally before.calc_metrics() now reports calib_slope, calib_intercept, and calib_large
alongside the other binary metrics, and the workflow's bootstrap covers them.plot_calibration() adds the three numbers to the caption by default
(annotate = FALSE turns it off), computed on the predictions rather than on
the binned points.NA when it cannot be estimated: fewer than 20
patients, fewer than 5 events or 5 non-events, or separation. A slope that ran
to infinity because of separation is not a number anyone should read.calc_ci() accepts a probability matrix and bootstraps the multiclass metric
set, so macro one-vs-rest AUC, accuracy, balanced accuracy, kappa, log loss, the
multiclass Brier score, and the macro rates all get intervals.save_result() and read_result() write and read a fitted result with a
version stamp and validation on both ends.calc_metrics.data.frame() scores the prediction table that
predict_uniml() returns, which is how subgroup or other post-hoc analyses are
meant to be done: this package does not define subgroups, but it hands you a
model and a scorer.calc_metrics.matrix() scores a multiclass probability matrix directly.features(data, spec), and set_preprocess(vars = )
is honoured when a task type declares no feature selection of its own: the
workflow keeps the columns the task type calls features instead of borrowing a
classification selector.Two additions that a task type with more than one outcome column needs.
set_preprocess() gains vars, naming the columns the preprocessor may
transform. Columns outside it are passed through untouched, so a task type can
keep its own outcome columns out of scaling and encoding. workflow_engine()
forwards it to fit_preprocessor().register_workflow_adapter() accepts two optional hooks, train() and
tune(), which default to train_model() and tune_model(). A task type whose
learners are not TaskClassif learners supplies both, and the engine fits and
tunes them without knowing what they are.The workflow is now split into a task-type-neutral engine and a per-task-type report, so a new outcome shape plugs in through a registered adapter instead of editing the pipeline.
workflow_engine() performs the half of a workflow that does not depend on
the outcome: preparing the data, fitting imputation, preprocessing and feature
selection, tuning, training one model per learner and imputation, and predicting
the training, validation, test, and external sets. It returns predictions and
never metrics.register_workflow_adapter() and workflow_adapter() declare how a task
type prepares data, builds its mlr3 task, extracts a prediction, pools
predictions across imputations, names its metrics, and turns a prediction into
metrics. The binary adapter ships with the package.run_workflow() is now the binary report on top of that engine; its arguments,
return value, and numerical behaviour are unchanged.run_workflow() still rejects non-binary outcomes, and the message now says
what is genuinely binary about it (thresholds, calibration, DeLong and
Hosmer-Lemeshow tests, NRI/IDI, decision curves) and points at
workflow_engine() instead of leaving the reader with a dead end.Multiclass outcomes are a supported task type. They are modelled at the layer level; the full workflow is still binary-only and says so.
uniml_spec(task_type = "multiclass", outcome = "y") no longer needs a
positive class, and passing one is an error: a positive class only exists
when the outcome has two levels.as_uniml_data() accepts three or more classes, keeps all of them, and sorts
the levels so the class order is the same on every machine.make_task() builds a TaskClassif with no positive class.calc_metrics() gains a multiclass method (and switches to it automatically
for an mlr3 prediction with more than two probability columns). It reports
macro one-vs-rest AUC (mlr3measures::mauc_au1u), accuracy, balanced accuracy,
Cohen's kappa, log loss, the multi-class Brier score, macro one-vs-rest
sensitivity/specificity/PPV/NPV/F1, the k-by-k confusion table, and a per-class
table. Each of those conventions is pinned by a test against mlr3measures
and a direct recomputation.as_uniml_multiclass() converts an mlr3 PredictionClassif or a
probability matrix into the object the multiclass helpers accept.plot_roc() draws one-vs-rest ROC curves, one per class, and
plot_confusion() draws a k-by-k table with the accuracy as the subtitle.tune_model() chooses a measure that exists for the task: auc maps to
classif.mauc_au1u and brier to classif.mbrier on a multiclass task, and
two-class-only measures such as pr_auc are refused instead of silently
mistranslated.plot_calibration() and plot_dca() say why when handed a
multiclass prediction.run_workflow() stops with a message pointing at vignette("uniml-multiclass")
instead of half-running: its prediction table, thresholds, calibrators, and
post-model tests all assume a single probability column.These are the two places where a task type used to be hard-coded to the binary case. Both are now declared by the task type itself, so the next task type can be added without editing the callers.
task_type_spec() gains composite_metrics, the metric names that
rank_models() averages into its default composite score. Binary declares
AUC, sensitivity, specificity, F2, balanced accuracy, and Youden's index;
multiclass declares AUC, accuracy, balanced accuracy, and F1.rank_models() gains task_type = "auto". It reads the task type from a
uniml_result and otherwise assumes "binary", so ranking a multiclass metric
table is now correct instead of quietly averaging two-class metrics. A task
type that declares no composite set returns an error asking for an explicit
best_model_metric rather than falling back to the binary one..workflow_metric_values() is the single point that turns a prediction into a
metric row, and it goes through the task type's adapter, so the next task type
has one obvious place to plug in instead of editing the fold loop.First release. The binary-classification stack is implemented end to end;
multiclass and survival are registered as placeholders and raise a
"not implemented" error.
tripod_ai_check() audits a result against the TRIPOD+AI domains and
reports which information is present, absent, or out of scope. The checklist
is written to reporting/tripod_ai.csv by export_results().as_uniml_binary() converts an mlr3 PredictionClassif into the
uniml_binary object used by the evaluation and plotting helpers;
plot_roc(), plot_calibration(), and plot_confusion() also accept an
mlr3 prediction directly.models = "best" and models = "all", matching the
vocabulary of the SHAP plots.explain_shap() on a uniml_result gains a model argument and falls back
to the stored training features when data is omitted.inst/REFERENCES.md maps every layer to the literature it implements.model and set in separate columns instead of
a combined "model:set" label.uniml_spec() declares the task type, outcome, positive class, identifier and
excluded columns, leakage policy, defaults, and output options.as_uniml_data() validates the binary outcome and records column roles.make_split() builds two- or three-way splits, optionally stratified.median_mode, mice, missForest, missRanger, or none.zscore or minmax.fit_preprocessor() / apply_preprocessor() keep fitted state separate from
the data it is applied to.combine_selections() intersects or unions methods; remove_collinear()
prunes by correlation or VIF."evals" or "seconds".repeats > 1
instantiates repeated cross-validation and the summary averages folds within
a repeat before summarising across repeats.reference_model (default: the
model with the best training AUC) with bootstrap confidence intervals, and
exported as tests/nri_idi.csv.set_explain(thresholds = list(targets = ...)) extracts SHAP zero-crossing
thresholds for the requested features and exports thresholds/zero_crossings.csv
and thresholds/paper_ready_thresholds.csv.sv_interaction() support is
limited to xgboost/lightgbm). A single-observation waterfall plot
(plot_shap_waterfall()) is added instead, and the default plots are
importance, beeswarm, dependence, and waterfall.export_results() now writes predictions/predictions.csv and
performance/thresholds.csv, so figures can be rebuilt from CSVs without
loading the RDS.run_workflow() gains an optional external data set for external
validation. test and external are reporting-only; model selection uses
validation by default (select_on = "validation", "auto" now means the
same). Selecting on test is still possible but warns about optimism.set argument and default to the comparison set, so train,
validation, test, and external rows are no longer pooled into one curve.performance/metrics_external.csv is written when an external set is given."none". Platt, segment Platt, and isotonic
calibration are fitted on out-of-fold training predictions, and the raw and
calibrated probabilities are both kept in the result.strict_in.id and exclude columns are now actually dropped before
preprocessing (the previous setdiff() removed nothing, so identifiers could
enter feature selection and models).apply_imputer() honours the imputation index stored on a step, and
cross-validation for multiple imputations splits the completed data set
instead of always transforming folds with the first imputation.calc_ci() keeps the
bootstrap standard error, pool_mi() combines the per-imputation estimates
(T = U + (1 + 1/m) B, Barnard-Rubin degrees of freedom, t interval), and
cross-validation runs once per completed imputation before pooling. Pooled
intervals are clamped to each metric's range (for example AUC in [0, 1]).classif.log_reg and records the downgrade in meta$failures.impute_args, scale_args,
encode_args, balance_args, selection args per method, terminator /
seconds, and calibration_args. Unknown argument names are rejected
instead of being silently ignored.assemble_result() and rank_models() produce the comparison table and the
composite ranking; export_results() writes the layout to disk.
ROC, calibration, DCA, and confusion-matrix figures.
Plot helpers accept theme, title, subtitle, and caption; palette
takes a ggsci name, a single colour, or a colour vector.
export_results(plot_args = list(...)) controls figure size, DPI, and device
(for example device = "png", dpi = 600).
explain_shap() plus importance, beeswarm, dependence, waterfall, and force
plots, and extract_thresholds() for candidate cut points.
explain_shap() now computes model-agnostic SHAP values with kernelshap
(CRAN) instead of the archived fastshap. The new n_explain (rows to
explain, default 100) and bg_n (background rows, default 100) arguments keep
the run time bounded, and any other kernelshap() argument can be passed
through ....
The lgb, randomForest, and gbm shortcuts are retired: their mlr3
learners live in mlr3extralearners, which is not on CRAN, so the package can
now pass CRAN's dependency checks. The shortcuts fail with a message that
names the mlr3 learner key to use instead; install and load
mlr3extralearners and pass "classif.lightgbm" (or "classif.randomForest",
"classif.gbm") directly.
Bug fix: calc_metrics(), as_uniml_binary(), and every plot_*() method
that accepts an mlr3 PredictionClassif read the positive class from the last
probability column. mlr3 puts the positive class in the first column, so the
confusion-matrix metrics (sensitivity, specificity, PPV, NPV, F1, F2, PR-AUC)
were computed for the negative class while the ranking-based metrics looked
correct. The class is now taken from the mlr3 task when the prediction carries
it and from the first probability column otherwise; an explicit positive
still overrides both.
Bug fix: the internal Hosmer-Lemeshow fallback grouped probabilities with
right = FALSE while ResourceSelection::hoslem.test() uses right = TRUE,
so hl_test() returned a different statistic when ResourceSelection was not
installed. The fallback now reproduces the package result exactly.
hl_test() no longer returns NaN. When a group's probabilities are all 0 or
all 1 the expected cell is empty and the statistic is not defined; the result
is now NA with method = "not_estimable" and a message, on both paths.
tests/testthat/test-consistency.R, which pins the package to
independent implementations: ROC AUC to mlr3measures, pROC, and the
Mann-Whitney U statistic; PR-AUC to PRROC; the 2x2 metrics to a direct
recomputation; Platt calibration to stats::glm() on the logit scale;
isotonic calibration to stats::isoreg(); net benefit to the decision-curve
formula; DeLong AUCs to calc_metrics(); and the Hosmer-Lemeshow fallback to
ResourceSelection.vignette("uniml-realdata"), an end-to-end run on pROC::aSAH
(113 patients, 12-month Glasgow outcome) rather than on the simulated cohort,
including the gos6 leakage guard and the reporting behaviour when a statistic
cannot be estimated.run_workflow() chains the layers with strict_in, strict_out, or custom
leakage handling, and predict_uniml() scores new data from the fitted state.Added a 2000-row cardiovascular in-hospital mortality development cohort and an independent 2000-row new-admission cohort for external validation examples.
Added an end-to-end workflow regression test that trains, ranks, and scores the synthetic cardiovascular example cohorts.