The trees an E-step returns under the thinning proposal carry their lineage
ids and covariate columns again. The copy rcpp_mcem.cpp made of them had
only brts, n and t_ext, so under stop_rule = "mc_error" every batch
M-step of a D, ED or K model failed to build its covariate table, the
Monte Carlo error of the step was never computed (mc_z was NA on every
iteration), the draws never grew and no thinning fit of those models met
the rule: it ran to max_iter whatever the estimate did. The surrogate
proposal was not affected.
estimate_rates(cond = "surface") conditions a fit on survival for any
model and link: an unconditioned fit, then MCEM continued on local
quadratic surfaces of a direct Monte Carlo estimate of log P(S) around the
iterate, rebuilt whenever the iterate leaves the inner half of a surface's
box, with a second run from the nested start when the unconditioned
extinction intercept is on an edge of the box. The reported log-likelihood
subtracts a direct estimate of log P(S) at the estimate, and the fit's
conditioning records the surfaces, clades, seconds and the start kept.
It is the recommended conditioning for the lineage-level models, whose
survival GAM is not accurate enough (E43). The settings are control
entries named surface_*.
survival_mc() estimates log P(S) by direct simulation for any model, link
and sampling fraction, with common random numbers across parameter values;
survival_surface() fits the local surface, with predict() and print()
methods. predict_survival() and estimate_rates(cond = ) accept a
surface.
First release on CRAN. The package was rewritten after version 0.4
(2021--2024). The functions of that version (emphasis(), emphasis_de(),
sim_tree_pd_cpp(), augmentPD(), loglikelihood() and the rest) are gone;
the interface is the one described below.
estimate_rates(), simulate_tree(), auto_bounds()
and emphasis_pipeline() is now "exponential", under which the
intercepts are log rates; it was "linear". The model ladder, the
real-tree analyses, the exact comparison on real trees and the conditioned
estimability study use the exponential link; the earlier simulation studies
ran on the linear link and remain valid for it. Scripts relying on the linear default
must pass link = "linear", which stays supported.D likelihood on a bare branching-time vector counted one lineage too
many after an observed split, so a d model with zero D coefficients did
not reduce to cr; the alive lineages now follow the forward sweep by
lineage id. Trees given as phylo objects were not affected.simulate_tree(), replaces the separate forward simulators
and augmentation functions. It simulates trees forward, in batches over a
parameter matrix, or conditionally on an observed tree (augmentation).estimate_rates(), replaces emphasis() and
emphasis_de(), with the method as an argument.rho enters the
simulator, the likelihood and both augmentation samplers.cond argument of estimate_rates(): the
likelihood is divided by the survival probability of the clade, given as a
function of the parameters, a GAM or an nnet object. train_GAM() now
fits a binomial GAM of survival on the parameters; in version 0.4 it fitted
a GAM to simulated log-likelihoods. That GAM is not accurate enough to
condition on: in E43 (https://franciscorichter.github.io/emphasis/experiments/E43.html) its
90th-percentile error of log P(S) was 0.44 to 2.56 for every lineage-level
model at 2000, 5000 and 20 000 training clades, against 0.05 required.
Under cr and dd condition on the exact survival probability, passed as
a function (?estimate_rates gives the cr formula); under lineage-level
models on a direct Monte Carlo estimate, or not at all.D, evolutionary
distinctiveness ED (raw or centred) and the number of ancestral splits K
join diversity N. Version 0.4 read diversity and the clade's phylogenetic
diversity.compare_models(), lineage_table() and
covariate_path().num_threads is set.N, age-imbalance D, evolutionary
distinctiveness ED (raw or centred, EDc) and the number of ancestral
splits K. Models are named ("cr", "dd", "d", "nd", "ed",
"ned", "edc", "nedc", "k", "nk") or given as a formula such as
~ N + ED, on an exponential (the default) or linear link.estimate_rates() fits a model by Monte Carlo expectation-maximization
("mcem"), the cross-entropy method ("cem") or a GAM fitted to a
likelihood surface ("gam"). model = "all" fits the nested ladder of
models and ranks it.auto_bounds() builds a parameter box from the observed tree by forward
simulation.compare_models() compares fits by AIC and reports the Monte Carlo error of
each log-likelihood and the downward bias of a log of a Monte Carlo mean.lineage_table() returns the per-lineage design the likelihood is built on,
and covariate_path() traces a selection path over the covariates.cond takes is a function, a GAM from
train_GAM() or an nnet object, evaluated by predict_survival().diagnose_mcem(), diagnose_cem() and diagnose_gam() report convergence
and importance-sampling diagnostics.message() and can be
silenced with suppressMessages() or verbose = FALSE.emphasis_pipeline() runs on one thread unless control$num_threads is
set.simulate_tree()'s max_tries is the number of forward draws in all, the
first included: the default 1 is a single draw, unconditioned on
survival, and an extinct clade is redrawn until one survives or max_tries
draws have been made. max_tries = 0, which asked for a single draw when
the argument counted the draws after the first, is read as 1 with a
warning. auto_bounds() passes surv_tries (default 20) as max_tries,
and the survival GAM is trained on single draws.control$max_time binds inside the E-step and the M-step of an MCEM fit,
and the fit ends with stop_reason = "time_budget" and a clock note
naming the step it cut. Both samplers read the clock between draws and
inside a draw and stop drawing when it runs out; the iteration in progress
is discarded. The M-step's optimiser (NLopt's sbplx, under both samplers)
reads the same deadline before every evaluation of its objective and stops
once it has passed; the iteration then keeps the best point the optimiser
had evaluated when that point improves on the current iterate, and is
discarded otherwise. The E-step that evaluates the returned parameters gets
max_time again and, when it runs out, estimates the log-likelihood from
the draws it made, so a fit returns within about twice max_time.notes, the named notes estimate_rates() states about it.
When the surrogate proposal's mean-field iteration was relaxed below the
full step, ran under a damping other than "auto", or did not converge
within its sweep budget, the mean_field note gives the setting, the sweeps
taken, the relaxation and the final residual at the last E-step and whether
it converged, and print() shows it. Every E-step's mean field is in the
trace (mf_converged, mf_delta, mf_sweeps, mf_omega, mf_budget).control$damping defaults to "default", which is "auto" at complete
sampling (rho = 1) and a fixed relaxation of 1/8 when rho < 1. In
experiment E10c a fixed 1/8 converged 0 of 192 units within the 20 sweeps
at rho = 1 against 131 for "auto", at 1.4--3.3 times the E-step time
and no gain in effective sample, and below rho = 1 it converged more
(142 against 131 at rho = 0.2). An explicit "auto" or number is used as
given, and the mean_field note names the resolved setting and whether it
is the default.compare_models() chooses the model with the fewest parameters within
margin of the minimum AIC and, among those, the one with the lower AIC,
whatever order the fits are passed in.