--- title: "3. Map of the package" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{3. Map of the package} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` This vignette is the conceptual map of `Stem`: what each function is for, and how they call one another. It is meant to be read once, before diving into the manual pages. ```{r diagram, echo = FALSE, out.width = "100%", fig.alt = "Call graph of the Stem package, organized in four layers: data and model object, estimation engine, simulation and prediction, spatially-clustered STEM"} knitr::include_graphics("function-map.svg") ``` The same diagram is installed with the package and can be opened from R: ```{r file, eval = FALSE} system.file("extdata", "STEM_function_map.svg", package = "Stem") ``` ## The four layers ### 1. Data and the model object Everything starts from a `STEM_Model` object, built by `STEM_Model()` from four ingredients: the observation matrix `z` ($T \times d$), the stacked covariate matrix (`(T \cdot d) \times r`, blocks ordered by location), the coordinates ($d \times 2$) and a list `phi` of starting values. Two internal constructors do the validation: `STEM_Skeleton()` checks the parameter list and the loading matrix `A`, `STEM_Data()` checks the data and derives the dimensions. Two datasets ship with the package: `pm10`, the small 22-station example, and `povalley`, a real network of 36 background stations observed daily over 2019--2023. ### 2. The estimation engine `STEM_Estimation()` is the entry point. It hands the fit to `stem_em_fit()` (`R/em-fit.R`), which runs the algorithm chosen in `STEM_control()`: the iterations of the EM or of the ECME algorithm, plain or accelerated by SQUAREM (`R/squarem.R`). One iteration is an E-step, `stem_estep()` (`R/estep.R`), followed by an M-step, `stem_mstep_em()` (`R/mstep-em.R`) or `stem_mstep_ecme()` (`R/mstep-ecme.R`), both built from the conditional updates of `R/mstep-updates.R`. The E-step rests on `kalman()` (`R/kalman.R`), the state-space core of the model: the Kalman filter, the fixed-interval smoother and the lag-one covariance smoother at the current parameters. Underneath: * `filtering()` / `filterstep()` - the forward Kalman recursions, which also accumulate the log-likelihood. They never form the $d \times d$ predictive covariance $Q_t$: the Woodbury identity turns each step into $p \times p$ algebra against a Cholesky factor of $\Sigma_\varepsilon$ that is computed once per pass, and the log-density is evaluated directly rather than as the logarithm of a density that would underflow on a large network; * `stem_obs_index()`, `stem_blocks_cache()`, `stem_missing_blocks()` - the missing-data bookkeeping: which locations are observed at each time point, and the Gaussian conditioning blocks that complete the sufficient statistics; * `smoothing()` / `smootherstep()` / `smootherstep.uni()` - the backward recursions producing the smoothed latent states; * `stem_gls_mean()` - the augmented Kalman filter of the ECME step, which updates $\beta$ and $m_0$ on the observed likelihood; * `Q_function_addendo1()` - the part of the expected complete-data log-likelihood that depends on the spatial parameters; * `d1_Q()`, `d2_Q()`, `d12_Q()` and the `d1/d2_Sigmastar_*()` derivatives - the gradient and Hessian of the Newton-Raphson step that updates $\theta$ and $\log b = \log(\sigma^2_\varepsilon / \sigma^2_\omega)$; * `Sigmastar.exp()` and `changedimension_covariates()` - matrix utilities. Closed-form M-steps update $\beta$, $\sigma^2_\omega$, $G$, $\Sigma_\eta$ and $m_0$; the Newton-Raphson step handles $\theta$ and $\sigma^2_\varepsilon$. **How the iterations are organized.** Four algorithms are available, chosen by `STEM_control(algorithm = )`: `"SQUAREM"` (the default), `"EM"`, `"ECME"` and `"SQUAREM-ECME"`, which extrapolates the ECME iterations. One iteration is a *map*, the parameters in and the parameters out together with the log-likelihood at the parameters in, and the map is run either plainly or by SQUAREM until the stopping rule. `STEM_Estimation()` builds the data of the fit once (distances, covariates, the index of the observed rows), runs the algorithm, and calls `kalman()` once more at the estimates for the log-likelihood and the smoothed states it reports, so that estimates, log-likelihood and states always refer to the same parameters. The settings are those of `STEM_control()`: `em_tol_par`, `em_tol_loglik`, `em_maxit` and `em_stop` for the pooled fit and the final refits; `alt_em_tol_par`, `alt_em_tol_loglik` and `alt_em_maxit` inside the alternation of SC-STEM; `nr_maxit` for the Newton-Raphson step. Each fit reports in `convergence.par` the number of iterations, the Newton-Raphson iterations of each, the algorithm, whether the stopping rule was met, the limits of $\theta$ and whether the estimate sits at one of them (`convergence.par$theta.bound` in `STEM_Estimation()`, `theta_bound` in `SCSTEM_Estimation()`). The algorithms, the stopping rule and the limits of $\theta$ are described in Section A1 of the Computational and Algorithmic Supplement of the SC-STEM paper; the experiments behind the defaults are in the vignette "5. Numerical experiments on the estimation algorithms". ### 3. Simulation, prediction and uncertainty * `STEM_Simulation()` draws a new dataset from a `STEM_Model` at the parameter values held in its skeleton. It is the generator behind both bootstraps. * `STEM_Kriging()` predicts at unobserved locations through `spatial.pred()`. * `STEM_Complete()` returns $E[z \mid z_{obs}]$ at the observed locations, so a gap in the response is filled by the model rather than left as `NA`; it is the object a cross-validation exercise scores. `SCSTEM_Complete()` does the same regime by regime and reassembles the network. * `STEM_Bootstrap()` repeats *simulate, then re-estimate* through `STEM_Bootstrap.fn()`, giving the sampling distribution of the parameters of a single pooled model. ### 4. Spatially-clustered STEM This layer sits on top of the previous ones: every cluster-wise fit is an ordinary `STEM_Model` estimated by `STEM_Estimation()`. * `SCSTEM_Estimation()` alternates between estimating one STEM model per regime and reallocating locations by an ICM sweep on a Potts-penalized objective. Its helpers are `scstem_neighbors()` (symmetrized k-nearest-neighbor graph), `scstem_init()` and `scstem_repair_partition()` (initialization), `scstem_loglike_i()` (the per-location contribution), `scstem_potts_pairs()` and `scstem_pen_local()` (the penalty), `scstem_swap_pass()` (the size-preserving swap step) and `scstem_npar()` (the parameter count behind the information criteria). * `SCSTEM_Infocrit()` loops `SCSTEM_Estimation()` over a grid of $(K, \phi)$ and collects the criteria and the admissibility flags. * `SCSTEM_Select()` applies the two-step tuning rule to that grid, and uses `scstem_ari()` to compare every partition on it with the selected one. * `SCSTEM_Bootstrap()` regenerates the data regime by regime with `STEM_Simulation()` and re-runs the whole of `SCSTEM_Estimation()` on every draw, so the uncertainty of the partition is propagated. * `SCSTEM_BootInference()` aligns the refits with `scstem_align_labels()` and turns the draws into standard errors, confidence intervals and between-regime tests. * `SCSTEM_CV()` is an optional validation, separate from the choice of $(K, \phi)$: it refits one or more models on blocked folds (locations, time blocks, both, or random cells), predicts the removed cells with `SCSTEM_Complete()` and, at removed locations, with `scstem_predict_new()`, which kriges within the regime of the nearest retained location, and ranks the models by their predictive error on the same folds. ## A typical session ```{r session, eval = FALSE} library(Stem) data(povalley) # 1. build the model object mod <- STEM_Model(z = ..., covariates = ..., coordinates = ..., phi = ..., A = ...) # 2. the pooled benchmark pooled <- STEM_Estimation(mod, distance = "geo") # 3. explore the (K, phi) grid and select ic <- SCSTEM_Infocrit(mod, K_grid = 1:4, distance = "geo") sel <- SCSTEM_Select(ic) # 4. quantify the uncertainty, partition included boot <- SCSTEM_Bootstrap(sel$fit, B = 200, seed = 1) inf <- SCSTEM_BootInference(boot) ``` ## Regenerating the map The edge list underlying the diagram is obtained by parsing `R/` and recording which package functions each function calls; it is stored, in text form, in ```{r edges, eval = FALSE} system.file("extdata", "STEM_function_map.md", package = "Stem") ``` ## Is the implementation faithful to the papers? The estimation code has been checked equation by equation against the three companion works of Fasso and Cameletti that define the STEM model - the heterogeneous-network paper of 2007, the technical report that introduced this package, and the 2010 synthesis whose Equations (12)-(21) the code implements. The result, including the one defect the check turned up, is documented in: ```{r verification, eval = FALSE} browseURL(system.file("extdata", "STEM_model_verification.html", package = "Stem")) ```