First release of the package after its archival on CRAN, and the first to include the spatially-clustered STEM (SC-STEM) model family. The classical STEM workflow of version 1.0 is unchanged in its statistical content; what changes is its numerical robustness, the whole SC-STEM layer, and the packaging.
A (it was K): STEM_Model(..., A = ),
skeleton$A, STEM_Kriging(..., A.newlocations = ). Scripts written for
version 1.0 must replace K = by A =.K, with index k = 1, ..., K, as in the paper and
as in the SCDA package: STEM_Fit(K = ), SCSTEM_Estimation(K = ),
SCSTEM_Infocrit(K_grid = ), SCSTEM_Select()$K_selected, and the columns
K, K_eff of the grid. The transition matrix stays G.df: info_crit["df"] (it was "k") and
the column df of the grid (it was npar).mink and maxk of SCSTEM_Infocrit() are removed.k, which R would otherwise complete silently to knn.STEM_Fit() is a single entry point for the pooled and the clustered model,
with or without a penalty on the regression coefficients. K = 1 fits the
pooled STEM model and K > 1 the spatially-clustered one; lambda = 0, the
default, gives the ordinary maximum likelihood estimator and reproduces
STEM_Estimation() exactly.alpha and lambda follow the glmnet parameterization: alpha = 0 is
ridge, alpha = 1 the lasso, anything between the elastic net. Only the
regression coefficients are penalized; the intercept is excluded by default.
In the clustered model the penalty acts within each regime, with one
(alpha, lambda) shared by all of them.lambda is dimensionless. Its L1 part is measured against the largest partial
gradient, so lambda in (0, 1] traverses the whole lasso path and means the
same thing at any error variance and in any regime; its L2 part is left alone,
being already unit-free. lambda_scale = "absolute" recovers the raw
convention.lambda_by = "size" spreads the penalty over the regimes in proportion to
1/n_g instead of equally, shrinking the smaller regimes more. It remains one
hyperparameter.latent = FALSE switches the latent process off and spatial = FALSE
replaces the spatial correlation by the identity. Together with
regularization = 0 they reduce the model exactly to penalized linear
regression, and with K > 1 to clusterwise penalized regression.lambda means the same thing for every covariate and every regime.dev/regularization/stem-elastic-net.tex in the
source repository.SCSTEM_Estimation() fits an SC-STEM model: the monitoring locations are
partitioned into K latent spatial regimes and a separate STEM model is
estimated within each of them, so that regression coefficients, variance
components and latent temporal dynamics are all cluster-specific. Labels and
parameters are estimated jointly by maximizing a Potts-penalized
log-likelihood.SCSTEM_Infocrit() now fits a full (K, phi) grid and returns exact
log-likelihoods, AIC, BIC and KIC, an admissibility flag, the cluster sizes
and the estimated partitions.SCSTEM_Select() implements a two-step rule for choosing the
hyperparameters: (S1) the modal BIC-minimizing K inside a
moderate-penalty band, ties resolved towards the smaller K; (S2) the
phi with the smallest BIC at that K over the whole grid, ties resolved
towards the smaller phi. Only admissible configurations enter the rule.
The pooled K = 1 model competes in (S1) and is returned when no partition
improves on it.SCSTEM_Bootstrap() is now a refit-with-clustering parametric bootstrap:
data are generated cluster by cluster from the fitted model and the entire
procedure, endogenous partitioning included, is re-estimated on every draw, so
that the uncertainty of the partition is propagated. It replaces the previous
bootstrap, which conditioned on the estimated partition and therefore
understated the uncertainty. The refits use the same alpha, lambda,
penalize, lambda_scale, lambda_by, latent and spatial as the
original fit, so a fit with a ridge on the coefficients is resampled with it.
Its draws can be computed in parallel: cores (default 1) shares them among
that many R processes. Every draw runs under its own seed, drawn from seed,
so the draws are the same whatever the number of processes.SCSTEM_BootInference() aligns every refit onto the original clusters by the
majority rule and returns bootstrap standard errors; normal, basic,
percentile and bias-corrected confidence intervals; pairwise percentile tests
for the differences between clusters; the co-clustering matrix; and the ARI of
each refit against the original partition.SCSTEM_CV() is an optional validation of predictive accuracy, separate
from the choice of (K, phi): blocked cross-validation leaving out
locations ("LKLO"), time blocks ("LKTO"), both ("LKLHTO") or random
cells, of one fit or of a named list of fits on the same data (the pooled
model, the selected one, models with different covariates), all refitted on
the same folds and ranked by root mean squared error. A removed location
inherits the regime of its nearest retained location and is kriged within it.SCSTEM_Estimation, SCSTEM_Infocrit, SCSTEM_Select,
SCSTEM_Bootstrap, SCSTEM_BootInference and SCSTEM_CV, each with a
print() method.NA), so the panel may be
unbalanced. The Kalman filter restricts each time point to the locations
observed there (Durbin and Koopman 2012), and the M-step completes the
sufficient statistics with the conditional moments of the missing values,
which the spatial covariance couples to the observed ones. Kriging and both
bootstraps follow the same rule. Every location must keep at least one
observation; covariates and coordinates cannot be missing.STEM_Complete() and SCSTEM_Complete() return the conditional expectation
of the response given the observed data: the observed value where there is
one, the model-based prediction where there is not.STEM_Signal() and SCSTEM_Signal() return the fitted values, the
regression surface plus the smoothed latent process, without the
measurement error.label_update = "simultaneous".Q = sum_i l_{i,k_i} + phi * c * #{concordant neighbor pairs} is now
computed explicitly and traced along the iterations (obj_trace).
Convergence is declared on label stability, on the improvement of Q, on
cycle detection (the best visited partition is returned) or at max_iter.enforce_min_size = TRUE). Within-cluster
homogeneity is what the assignment step seeks, so the cluster-wise variance
components shrink and, left unconstrained, the cluster with the smallest
residual variance attracts every location. On the pm10 example the
unconstrained sweep collapses to a single cluster even at phi = 0. A
location may now leave its cluster only if that cluster stays at or above
min_cluster_size.swap_pass = TRUE). Since the constraint above
freezes any location sitting in a minimum-size cluster, each sweep is followed
by a pass that exchanges the labels of two locations whenever this strictly
increases Q. Swaps leave cluster sizes unchanged, so feasibility and
monotonicity both hold. An exact pruning bound keeps the scan affordable.knn graph is symmetrized: the Potts penalty is defined on an
undirected graph, whereas spdep::knearneigh() returns an asymmetric one.knn graph is built with the same metric as the covariance. The
nearest neighbors used to be taken on raw coordinates, so on longitude and
latitude they were the neighbors of a planar metric in which a degree of
longitude and a degree of latitude count the same; at 45 degrees of latitude
the first is about 78 km and the second about 111 km, so the graph preferred
north-south neighbors while Sigma_e was measured on the sphere. The
distance argument now governs both.phi_scale argument. Each location contributes T observations to the
likelihood, so a penalty calibrated for cross-sectional models is not
transferable. The default "auto" normalizes the penalty by the median spread
of the location-wise log-likelihood contributions; "per-observation" and
"raw" are also available. The penalty actually applied is reported in
phi_effective.init_method = "departures"): the mean of its residual from the pooled
signal, the slopes of that residual on its covariates, and the lag-one
autocorrelation and log-variance of what the slopes leave. The covariate
means, the previous default, carry no information on the regimes when the
covariates are exogenous to them; they remain available as
init_method = "kmeans". SCSTEM_Infocrit() computes the departures once,
from its fit at K = 1.phi_penalty > 0 starts from the solution of the unpenalized
fit at the same K (SCSTEM_Estimation() fits phi = 0 first;
SCSTEM_Infocrit() passes on the partition of the phi = 0 fit it already
has). A penalty that is strong from the first sweep froze whatever partition
it was given, so a start unrelated to the regimes stayed where it was. The
automatic scale, computed at the first sweep, is then the one of the
unpenalized fit and no longer depends on the initialization. Measured there
it is several times larger than before, so the defaults are recalibrated:
phi_penalty = 0.05, phi_grid = c(0, 0.025, 0.05, 0.1, 0.2, 0.5, 1) and
the band c(0.025, 0.2) of SCSTEM_Select(); the strong values 0.5 and 1
lie outside the band and compete only in the choice of phi.The score used to allocate locations to clusters before 2.0.0 contained four defects, all fixed:
m0 instead of
Sigmaeta, which flips its sign whenever m0 < 0;sigma2omega and sigma2eps were taken from the pooled fit rather than from
the candidate cluster, which made the corresponding term constant across
clusters and turned the nugget_var switch into a no-op;y_{t-1} instead of the
contemporaneous K_i y_t of the measurement equation;phi without a
comparable scale.The contribution is now an explicit conditional pseudo-likelihood, documented as such, and used only to rank clusters: coefficients, variance components and information criteria all come from the exact cluster-wise likelihoods.
K_eff * (ncov + 3 + 3p) free parameters and, in the BIC, the
number of observed values of the response as the sample size (d * T when
the panel is complete). The previous implementation used the number of
clusters as the number of parameters and the number of locations as the
sample size.STEM_Estimation()estimates$loglik is now the log-likelihood. Every earlier version,
1.0 included, returned four times it: the EM loop stored -2 times the
log-likelihood and the value returned multiplied that by -2 again. The
iterations were not affected (the stopping rule used differences of the
stored value, halved), but everything computed from the returned value was:
the AIC, BIC and KIC of SC-STEM weighted the likelihood four times against
the penalty, which favored more regimes. On the 951 replications of an
earlier run of the simulation study, the BIC on the true log-likelihood
selects fewer regimes in 40, mostly in the weakly separated scenarios. The
log-likelihood, the smoothed states and the maximum are now tested against
dense computations on the full covariance of the data
(tests/testthat/test-exact.R).STEM_control(): the computational settings of every estimation
function in one object, in the spirit of optim(control = ). All the
estimation functions take control; options(Stem.control = list(...)) sets
them for a whole session.STEM_control(algorithm = ): the EM
algorithm; the ECME algorithm (Liu and Rubin 1994), in which the regression
coefficients and m0 are updated on the observed likelihood by a
generalized least-squares step computed with the Kalman filter; and both
accelerated by SQUAREM (Varadhan and Roland 2008). They reach the same
maximum. The default is "SQUAREM": in the simulation study it fitted the
regimes in about a third of the time of the EM algorithm, closer to the maximum,
and ran the whole grid of SCSTEM_Infocrit() in about half the time, on
small and large cells alike, with the same selections. The
algorithm applies to the pooled fit and to the final refits; inside the
alternation of SC-STEM, whose fits stop after a few iterations, the
iterations are not accelerated. With a ridge on the coefficients
(lambda > 0, alpha = 0) the safeguard of SQUAREM compares the penalized
log-likelihood, the scale of the penalty measured at each point: the
accelerated iterations reach the fixed point of the plain ones, in about
half the time. Lasso and elastic-net fits (alpha > 0) run the plain
iterations.regularization defaults to 0 (it was 0.01) in STEM_Estimation(),
SCSTEM_Estimation() and STEM_Bootstrap(). The constant was added to every
matrix the EM algorithm inverts but not to the filter that computes the
likelihood, so the iterations converged to a point that is not the maximum:
the variance parameters were off by 3 to 13% on the regimes of the
simulation study, and the variance of the latent innovations of a pooled fit
by 84%, whatever the tolerance.convergence.par$theta.bound of STEM_Estimation()
and theta_bound of SCSTEM_Estimation() report a range at a limit. The
Newton-Raphson step of the spatial parameters accepts a step only if it does
not worsen the objective, and allows one grid search where it used to
allow 30; nr_hess_maxit is removed from STEM_control().STEM_Estimation() refer to the same parameters: the log-likelihood and the
smoothed states used to be those of the last-but-one iteration.kalman(), R/kalman.R), the E-step (R/estep.R), the
conditional updates of the M-step (R/mstep-updates.R), the M-steps of the
EM and ECME algorithms (R/mstep-em.R, R/mstep-ecme.R), SQUAREM
(R/squarem.R) and the wrapper that runs the algorithm chosen
(R/em-fit.R). kalman() used to run a whole EM iteration; it now runs the
filter and the smoothers, which the E-step and the fit at the estimates use. With
algorithm = "EM" and regularization = 0.01 the iterations reproduce those
of the previous code to 1e-13. The smoother gain of the initial state now
uses the transpose of G, which matters only for a non-symmetric G.1e-3 on both by
default. With em_stop = "any" (the default) the algorithm
stops when either criterion is met, with em_stop = "all" only when both are;
in either case after em_maxit iterations (default 500). The previous
rule stopped when the relative change of the whole parameter vector and the
relative change of the log-likelihood fell below precision. That vector
contained the fixed loadings, so the rule loosened with the number of
locations, and with log-likelihoods of the order of 1e4 a relative tolerance
stopped the algorithm while several units could still be gained.SCSTEM_Infocrit()).K = 1, so that the log-likelihoods compared across K are computed to the
same accuracy; the fit reports in em_converged whether the EM algorithm of
each regime met its stopping rule.SCSTEM_Infocrit() shares the refits across its grid, and a
penalized fit that ends at the partition of the unpenalized fit it starts
from reuses that refit. The refit depends on the partition and the settings,
not on phi_penalty; two refits of one partition from different starts
stopped at different points of the likelihood, and that difference decided
between penalties with the same partition. The fit reports
refit_reused; the argument refit_cache carries the shared refits and is
not a setting to tune.precision, max.iter, precision_full_dataset,
max_iter, abs_tol, rel_tol) still work, as overrides of control.control (nr_maxit).These were pre-existing defects, harmless for a single pooled fit but fatal for
a clusterwise algorithm, which refits the model on hundreds of different subsets
of locations. Both aborted the fit with
missing value where TRUE/FALSE needed:
kalman(): the Newton-Raphson step is validated before use; a singular or
non-finite Hessian, or a non-finite step, now exits the loop keeping the last
valid iterate instead of propagating NaN into the convergence test. The
relative criterion has a floor on its denominator, the Hessian condition is
evaluated through isTRUE(), and the fallback grid search is guarded against
an underflowed scale that produced log(0).STEM_Estimation(): a non-finite EM iterate is discarded with a warning and
the last valid one is returned; both relative convergence criteria have a
floor on the denominator and are wrapped in isTRUE().On the pm10 example these fixes take the parametric bootstrap from 4 usable
draws out of 12 to 12 out of 12.
STEM_Simulation() called mvrnorm() unqualified, relying on a NAMESPACE
import; it now calls MASS::mvrnorm().STEM_Kriging(): the checks on the number of columns of
coord.newlocations and covariates.newlocations never fired; they do now.
The documented predictor had the wrong sign on the latent term of the
residual (the code was right), and the documentation now states what
se.pred measures: the standard error of the spatially structured
component, without the nugget and the uncertainty of the smoothed state.d x d predictive
covariance. Since the measurement covariance does not depend on time and the
state contribution has rank p, the Woodbury identity and the matrix
determinant lemma reduce each step to p x p algebra against a Cholesky
factor computed once per pass, so a forward pass costs O(d^3 + T d^2)
instead of O(T d^3). On 200 stations over 365 days an EM iteration is about
ten times faster; the estimates are unchanged to floating point, the two forms
being algebraic identities.d observations is of order
exp(-d), so past roughly 300 locations it underflowed to zero and the fit
aborted with a non-finite log-likelihood. Networks of that size can now be
fitted at all.T matrices of size d x d at once. The derivative helpers
receive the inverse of the scaled covariance from the caller instead of
recomputing it up to seven times each, and every trace of a matrix product is
evaluated without forming the product.log b is
carried as the multiplier of the identity that it is rather than as a dense
matrix, the products against the inverse covariance are formed once instead
of three times each, and the log determinant and the inverse now come from a
single Cholesky factorization. The d x d matrix products per inner
iteration go from twelve to five.mvtnorm moves from Imports to Suggests: no function in the package uses
it any more, though the test suite still does, as an independent reference
implementation of the likelihood.Addresses the review comments received on the previous submission:
DESCRIPTION now carry <doi:...>, <https:...> and
ISBN links in the required format;T and F have been replaced by TRUE and FALSE throughout the code and
the documentation;print()/cat() diagnostics have been replaced by message() behind a
verbose = FALSE argument, threaded through kalman(),
STEM_Estimation(), STEM_Bootstrap() and the SC-STEM routines. The only
remaining cat() calls are inside print() methods, where they belong;.GlobalEnv. Where a seed is needed for
reproducibility, the RNG stream is saved and restored on exit.Further packaging work required before submission:
pm10 moved from a 250 KB source file duplicated in R/ and data/ to a
proper data/pm10.rda (25 KB), with R/pm10.R reduced to documentation;init_partition argument;dplyr and sf were likewise dropped, as nothing in the package needs them
any more; every external call is written as package::function() and
NAMESPACE no longer carries importFrom directives;URL and BugReports fields added, along with NEWS.md and a testthat
suite.STEM_Model() to the selection rule and the bootstrap), a map of
the package, computational notes, and numerical experiments on the
estimation algorithms.inst/extdata/, showing how the STEM
and SC-STEM functions call one another, and reachable from R with
system.file("extdata", package = "Stem").system.file("extdata", "Stem-computational-supplement.pdf", package = "Stem").
The source repository adds documentation pages on the models, the
computations and the functions (docs/), and the data and code that
reproduce the paper (papers/sc-stem-local-leader).