The mlr3 stack moved from Imports to Suggests. mlr3, mlr3pipelines,
mlr3learners, mlr3tuning, paradox, R6 and future back vimpute()
and nothing else -- 6 of 58 R files -- yet as hard dependencies they tied
VIM, and every package importing VIM, to the fate of the mlr3 chain: CRAN
archives reverse dependencies recursively, so one archived link would have
taken VIM down and sdcMicro, simPop, robCompositions, riskutility, deepImp
and 15 further packages with it. None of those packages executes a line of
mlr3-backed code; they use kNN(), hotdeck() and gowerD(), which are
mlr3-free. VIM now installs, loads and runs its visualisation, donor and
IRMI machinery without the stack present.
vimpute() -- and regressionImp(), rangerImpute(), xgboostImpute()
and overimpute(), which delegate to it -- stop with an actionable message
naming every missing package when the stack is absent. No behavioural change
when it is installed.
vimpute() no longer calls lgr::get_logger() unconditionally; lgr is a
suggested package and is now checked before use.
CRAN check-time reductions for the r-devel-windows 10-minute budget: the
vimpute and vimpute-mi vignettes are precomputed like the simulation
vignettes (code runs in vignettes/precompute.R, the check renders text
only), and thirteen further long-running regression-test files run only
with NOT_CRAN=true (CI and devtools::check()). No user-facing changes.
with() on a vimmi object returns a mice-compatible mira (elements call, call1, nmis, analyses) instead of an anonymous list, so mice::pool(), summary(pool(fits)) and mice::getfit() run unchanged. Code that indexed the old list directly (fits[[i]], lapply(fits, ...)) should use mice::getfit(fits) or fits$analyses.vimpute() returns are type-stable: the result is always the imputed data, classed like the input (data.frame in, data.frame out; data.table in, data.table out). With tune = TRUE or pred_history = TRUE the diagnostics are attached as attr(result, "tuning_log") / attr(result, "pred_history") instead of switching the return to a bare list.vimpute()'s default uncert is now "pmm" (random draw among the 5 nearest donors): default numeric imputations are observed donor values with an honest distribution, instead of deterministic conditional means. Set uncert = "none" for the previous behaviour. Default m > 1 runs are now stochastic between imputations. rangerImpute(), xgboostImpute() and regressionImp() keep their deterministic behaviour (uncert = "none" pinned internally).vimpute(m > 1) now bootstraps by default: boot defaults to TRUE for multiple imputation -- each of the m imputations refits its models on a bootstrap sample, so parameter uncertainty is propagated into the draws (approximately proper multiple imputation in combination with the default PMM) -- and to FALSE for single imputation. Set boot = FALSE explicitly for the previous single-fit behaviour.uncert = "pmm" and "midastouch" now perform true predictive mean matching (Little 1988): donors are matched on their predicted values -- scored by the trained model on the observed rows, out-of-bag for a no-bootstrap ranger fit -- instead of on their observed values. Matching on observed values selected donors whose values happened to lie near the prediction (including gross outliers from other covariate regions) and its donor spread shrank with n; predicted-value matching carries residual-scale variability, as in mice and Hmisc::aregImpute().uncert != "none": the imputed category is sampled from the predicted class probabilities on every sweep (the convergence criterion tracks the noise-free most-probable class), so early stopping no longer strips factor variables of between-imputation variability; under uncert = "none" factor imputation is now fully deterministic (previously sequential = FALSE drew stochastically regardless of uncert).diabetes is now a synthetic data set, generated by Matthias Templ from mlbench::PimaIndiansDiabetes2 with the synvey package: the team that collected the original Pima Indians Diabetes data has asked for redistribution to stop. Column names, types, dimensions and the Outcome levels are unchanged, so code runs as before, but the values -- and any results computed from them -- differ from earlier VIM versions; 0 pregnancies is now a valid value rather than NA.vimpute()'s imputation methods are resolved through a package-level registry, and register_vimpute_method() adds a user-defined method backed by any mlr3 learner pair in one call -- e.g. register_vimpute_method("cart", learner = list(regr = "regr.rpart", classif = "classif.rpart"), packages = "rpart"), or lightgbm via mlr3extralearners -- usable everywhere the built-in names work (global method, per-variable lists, method-keyed learner_params, tune = TRUE via an optional search_space hook). vimpute_methods() lists the registered methods, unregister_vimpute_method() removes user-added ones. The six built-ins are seeded through the same contract; unsupported-method errors now name the offending method and list what is registered.vimpute(data, spec = list(Sleep = vs_ranger(num.trees = 300, tune = TRUE), NonD = vs_robust(donorcond = ">= 0"), .default = vs_ranger())) bundles a variable's method, learner parameters, formula/predictors, tune, PMM settings, makeNA and donorcond into one object instead of nine parallel arguments. The vs_*() constructors (and vimpute_spec() for registered methods) validate learner parameters eagerly against the method's parameter set, so a typo fails at the constructor call, not mid-imputation. The flat per-variable arguments keep working unchanged; specs compile to exactly them.vimpute(data, Sleep ~ Dream + Span | ranger(tune = TRUE), NonD ~ . | robust(donorcond = ">= 0"), .default = vs_ranger(), m = 20, seed = 1). A plain-column right-hand side restricts the predictors (works for every method, including ranger/xgboost); a right-hand side with transformations (s(x), log(x), I(x^2)) becomes a model formula (formula-capable methods only). Grammar formulas compile to specs. Note: vimpute()'s signature now has ... in second position for the grammar -- arguments after data must be passed by name (they always were in all documented usage).vimpute() gains seed for whole-run reproducibility (applied once at entry, as in mice; the m imputations still differ from each other).vimpute() gains predictors: per-variable predictor control -- the equivalent of mice's predictorMatrix (named list or 0/1 matrix), working for every method including ranger and xgboost (where formula is rejected). A variable's formula takes precedence over its predictors entry.vimpute() gains visit_sequence: "asis", "increasing.na", "decreasing.na", or an explicit permutation of the NA-variables.vimpute() gains tuned_params to apply (and reuse) tuned hyperparameters without running the tuner; each tuning_log entry now carries its chosen parameters in $params.vimpute() gains tune_control = vimpute_tune_control(budget = , folds = , tuner = , batch_size = ): the tuning evaluation budget, resampling folds, tuner and batch size are user-controllable (defaults keep the built-in data-size heuristics); tuning_log entries record the budget and folds used. vimpute_search_space() exports the built-in per-learner search spaces (the starting point for register_vimpute_method() search_space hooks). Nested resampling is intentionally not offered -- the goal is good imputations, not unbiased generalisation estimates.m > 1 and tune = TRUE, tuning now runs once (in the first imputation) and the chosen parameters are shared by all m imputations -- previously every imputation re-tuned independently, conflating tuner noise with missing-data uncertainty. The vimmi object carries the tuning report in $tuning_log.vimpute() warns (naming the variable) and falls back to a featureless learner for that variable.vimmi objects store per-iteration chain statistics (mean/variance of the imputed values per variable, iteration and imputation) plus the seed, and plot(vimmi) draws mice-style convergence trace plots. Single runs expose the same trace data as attr(result, "chain").vim_as_mids(): the documented name for the vimmi-to-mids conversion (as.mids.vimmi() is kept as the historical alias; despite the dotted name it was never an S3 method -- mice::as.mids() is not a generic).makeMissing(): amputation generator for simulation studies (the variable-wise mice::ampute() counterpart) -- MCAR/MAR/MNAR mechanisms with exact per-variable proportions, driver weights, and an attr(., "where") indicator that plugs directly into evaluation()/nrmse()/pfc().overimpute(): model-agnostic calibration diagnostic (the Amelia::overimpute() analogue) -- the observed cells of a variable are overimputed fold by fold with multiple draws; print() reports the empirical interval coverage and plot() draws the observed-vs-imputed calibration plot. Works with any method, spec, or grammar configuration.OOBerror analogue): every vimpute() run reports NRMSE (numeric) / PFC (factor) per variable as attr(result, "model_error") -- out-of-bag for ranger, honestly labelled in-sample for the other learners -- and print(vimmi) shows it.plot(vimmi, "density") and plot(vimmi, "strip"): observed-vs-imputed distribution diagnostics (the mice::densityplot()/stripplot() analogues), complementing the "chains" convergence traces.makeMissing() -> vimpute(m = 5) -> chain/density diagnostics -> Rubin pooling via with()/mice::pool() and vim_as_mids() -> tuning with tune_control -> overimpute() calibration -> evaluation() against the simulated truth.makeMissing() scenarios, NRMSE on the amputed cells, runtime) comparing vimpute ranger/robust and kNN() with mice and missRanger (both guarded Suggests); precomputed at demo scale (vignettes/precompute.R), paper-scale via one constant.boot + uncert = "pmm"), the boot-free PMM variant, the textbook-proper (boot + normalerror), and a deliberately improper (bootstrap without residual noise) configuration -- demonstrating the anti-conservative pooled SEs the improper setting produces and the warning that guards against it.?vimpute states the missingness assumptions (MAR incl. MCAR; MNAR caveat with pointers to makeMissing() sensitivity simulation).imputeCellM() gains a chained-equations data.frame interface: imputeCellM(data) now imputes every variable with missing values -- each is regressed on all remaining variables, sweeps run with deterministic predictions until the imputed values stabilise (maxit, eps), and the requested uncert step is applied once after convergence -- returning the same data_imputed/cellweights/converged/iterations list as imputeCellIRMI(). imputeCellwise(data, method = "cellM") dispatches to it instead of erroring ("cellM requires a formula"), and the formula interface gains uncert = "none" (deterministic predictions; most-probable category for factors).method = "restricted" no longer has uncertainty draws layered over its constrained solutions: the 7.3.0 default uncert = "pmm" replaced the solver's rule-satisfying value with a donor draw that knows nothing about the rules -- under a rule y >= 4 with observed donors 1 and 2, the "restricted" imputation was 2 (CI-red on all platforms since the restrictionRegression merge, which predated the new default). Registry entries can now pin their variables' uncertainty mechanism (uncert_override): restricted pins "none", silently for the default and with a warning when a draw mechanism was requested explicitly. The per-variable model-quality report is also skipped for restricted (model_error = FALSE in the registry entry) -- its in-sample predict re-ran the conic solver once more per variable for a metric of marginal value, so save_optimization_problem = TRUE again records exactly one problem per imputed variable.uncert = "normalerror" / "resid" no longer draw from an in-sample residual scale: for learners that expose no scale of their own (ranger, xgboost), sigma_hat and the residual pool were derived from the model's predictions on its own training rows. A forest's in-sample predictions are near-interpolating, so the estimated scale came out at roughly half the true predictive spread (0.57 against a held-out 1.16 on a linear DGP with residual sigma 1): "normalerror" injected half the noise it should, "resid" sampled from a residual pool that was far too tight, and the pooled intervals of the textbook-proper boot + normalerror configuration under-covered badly (0.83 / 0.73 against a nominal 0.95 at 30% / 50% missingness). Where the fitted model exposes out-of-bag predictions -- ranger stores them for free, in training-row order -- both are now derived from those, which track the predictive spread to ~1%; every other learner keeps the in-sample fallback. Learners reporting a proper scale of their own (lm, lmrob, glmrob, gam, robgam) never took this path and are unchanged, as is the default uncert = "pmm", which matches donors on predicted values and needs no scale estimate at all. The same residual pool feeds robustboot = "stratified" (the default) and "residual", whose good/bad split was previously computed on the same over-tight residuals. A regression test pins both the scale and the residual pool against a held-out predictive SD.imputeCellIRMI() no longer errors with "subscript out of bounds" when DDC refuses columns: cellWise::DDC()'s checkDataSet() drops constant, too-discrete and NA-heavy columns (on the colic data 11 of 19 continuous columns), so stdResid covers only the retained submatrix while the weight-initialisation loop indexed all continuous columns. The DDC weights are now mapped back through colInAnalysis/rowInAnalysis; refused rows and columns keep univariate weights, and DDC's console chatter ("The final data set we will analyze has ...") is captured instead of leaking to the console.imputeCellEM(), imputeCellIRMI(), imputeCellMCD(), imputeCellMM(), imputeCellReg()) built per-variable model formulas by pasting column names, so with duplicated names -- e.g. after colnames(x) <- substr(colnames(x), 1, 15), which maps mucous_membranes_col and mucous_membranes_group to the same name -- the response resolved to the first column of that name: the model was fitted on the wrong variable, factor imputations drew from the wrong level set and were silently turned into NA by the levels-mismatch, with repeated "response appeared on the RHS" multinom warnings. The functions now fit under internal positional names and restore the user's names on exit.imputeCellM()'s formula interface never received its weights: model.frame() evaluates the weights argument in data and then in environment(formula) -- the caller's environment, where the internal row-weight vector does not exist -- so the cell-weighted fit always errored and silently fell back to the unweighted model (with a "Multinomial model failed" warning). The row weights are now passed as a column of the fitting data.classif.glm_rob no longer inverts binary classifications: the robust logistic learner behind vimpute(method = "robust") attached the fitted binomial probability -- P of the second factor level, the glm/glmrob convention -- to the first level, so binary factor imputations drew from inverted class probabilities and the deterministic argmax picked the less likely class: worse-than-chance imputation exactly where the signal is strong (PFC 0.70 instead of 0.13 on the diabetes data). The multiclass one-vs-rest path was oriented correctly and is unchanged; a direction regression test now pins both paths."robust"/"robgam" provide).evaluation() no longer returns NaN when one variable type has no missing cells (0/0 guarded).evaluation() gains where, the documented name for the amputed-cell mask (matching makeMissing()'s "where" attribute); the historical m keeps working, supplying both errors.eps now bounds the per-variable relative change (numeric: mean squared change of the imputed values divided by the variance of the observed values; factors: share of changed categories), and the run stops when the largest per-variable change stays below eps -- previously the raw changes were summed, so eps was meaningless across data scales (a variable measured in thousands could block convergence forever, and one variable could mask another). The full iterations-by-variables change matrix is returned as attr(result, "convergence").batch_size = 1 for the random search).cellWeightsMCD() keeps exact-constant columns at weight 1 instead of feeding them to covMcd() (every subsample is singular there, which produced repeated warnings on each outer iteration), and cellIRWLS() skips the lmrob.S initialisation for rank-deficient designs (e.g. a constant column aliased with the intercept), using the ridge-regularised cell-weighted OLS init instead.future plan active at entry is restored instead of being forced to "sequential".kNN()/gowerD() used every core, which CRAN's incoming checks flag ("examples with CPU time > 2.5 times elapsed time"). options(VIM.ncores = ) now sets the number of threads; without the option VIM uses at most 2 threads under R CMD check (CRAN's two-core policy) and OpenMP's default otherwise. Results do not depend on the thread count. The xgboostImpute() example was reduced to a single two-target call (CRAN's 5-second example limit).R CMD check within CRAN's 10-minute budget (the incoming pre-test on Windows took 14--17 minutes, 330 s of it tests and 280 s vignette rebuilds). The two simulation vignettes (Benchmarking imputation methods, Validating multiple-imputation properness) are now precomputed -- their code is executed by vignettes/precompute.R in the source repository and the output is embedded, so CRAN only renders text; the comparison chunks of the vimpute vignette use single-sweep imputations (sequential = FALSE) and the overimpute() demo three folds. Long-running tests -- hyperparameter tuning, the multiple-imputation and vimpute-default regression tests, and the two vimpute integration scripts under tests/ -- now run only when NOT_CRAN=true (set by devtools::check()/test() and by the GitHub Actions workflow); CRAN runs the fast core of the suite.vimpute() gains keep_all_columns (default TRUE): the full dataset is returned, with columns excluded via considered_variables passed through unchanged (matching kNN()/hotdeck()/irmi()); set FALSE for the previous considered-only shape.vimpute() warns when m > 1 cannot produce between-imputation variability (no boot, uncert, or stochastic pmm), so improper multiple imputation is no longer silent.vimpute() returns ordered-factor columns as ordered factors (previously flattened to plain factors; the m > 1 path also lost the level order).vimpute(tune = TRUE) now runs with sequential = FALSE (was a silent no-op).vimpute() handles per-variable method lists correctly: a named length-1 list validates the variable name, and an unnamed per-column list maps by column position.rangerImpute() and xgboostImpute() forward their hyperparameters to the backend learner.regressionImp() uses lm/glm as documented, falling back to regularized regression only when needed.vim_complete(). VIM exports no complete() generic of its own: mice and tidyr both export one, so an exported VIM generic would mask theirs (and be masked by them) and would make packages importing VIM and mice/tidyr wholesale emit "replacing previous import" at load time. The same function is registered as a method on mice::complete() and tidyr::complete(), so complete(obj, 1) keeps working whenever either package is attached.evaluation() supports vartypes = "guess".irmi(mi > 1) returns a list of imputations again (was a single mangled data.frame under the default imp_var = TRUE).imputeRobust(): method = "gamRob" and uncert = "wresid" now work (previously crashed); uncert is validated with a clear error; the PMM donor pool no longer includes initialised values.imputeRobustChain() repaired (previously imputed zeros or crashed on most paths).imputeCellIRMI(), imputeCellM(), imputeCellMCD()) no longer scale the design matrix by cell weights, which had made them impute worse than the median on clean data; the default init_weights is now "ddc".imputeCellReg() cell-weight computation fixed (could produce negative weights).imputeCellMCD(): documented that boot = TRUE does not yet propagate parameter uncertainty.kNN()/gowerD(): semi-continuous (mixed) distance variables are now range-scaled like numeric ones, so a large-scale mixed variable no longer dominates the neighbour search.kNN(weightDist = TRUE) no longer produces NaN imputations when distances exceed 1 (e.g. with methodStand = "iqr").?kNN and ?gowerD document the distance standardisation and the NA-sentinel convention.car moved from Imports to Suggests (Box-Cox implemented natively).vimpute() compatibility and validationrangerImpute(), xgboostImpute(), and regressionImp() delegate to vimpute()vimpute()mlr3 backend for a flexible imputation method.hotdeck() when ord_var and variable overlapirmi(),
kNN(), hotdeck() and regressionImp()). Thanks @wolfgangrannetbauer
(#44, #45)tableMiss(): A table that highlights missing and imputed values via
colors (#47).ranger::ranger() (#35)irmi() with logical and integer columns (#42)gowerD()maxCat() and sampleCat()collision addedimp_var and imp_suffix to irmi(), so it is more consistent with the other functions (#27)kNN(), to use random forest in combination with kNN()matchImpute() for imputing randomly within groupsimpNA in hotdeck()irmi() with factors (#13). Thanks @Deleetdkirmi() (#23)kNN() and data.tablehotdeck()data.frame is passed to irmi() (#6)kNN(): weightDist to use the distances for the k nearest neighbours as weightswhich.minN() is not used anymore, instead there is a C++ function, kNN() is now about 1.6 times faster on a replication (100x) of the sleep datasethotdeck()hotdeck() if no ord_var is definedhotdeck() with makeNAkNN() useImputedDist if the imputed values of a variable should be used in subsequent imputation of another variable.irmi() with newer version of nnet (multinom) and if residual scale can not be computed (noise)modelFormula in irmi()irmi()hotdeck() based on data.table -> faster and quite stablekNN()makeNA in kNN()imp_var now updates existing TF imp_vars (with warning)regressionImp()