| Title: | Direct Estimation and Inference for Spectral Differential Networks |
|---|---|
| Description: | Estimates and performs inference on the difference between two high-dimensional inverse spectral densities, which describes how the frequency-domain partial correlation network of a multivariate time series changes across two conditions. The difference is estimated directly with the lasso penalized D-trace loss of Yuan et al. (2017) <doi:10.1093/biomet/asx049>, and de-biased confidence intervals and p-values for individual edges are computed following Hellstern, Kim and Shojaie (2026) <doi:10.48550/arXiv.2609.13609>. Also provides multivariate spectral density estimates based on Welch's method and simulated example data. |
| Authors: | Michael Hellstern [aut, cre, cph], Huili Yuan [ctb, cph] (Author of the D-trace loss C code in src/dtl.c, ported from the archived 'Difdtl' package) |
| Maintainer: | Michael Hellstern <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 0.1.0 |
| Built: | 2026-11-20 13:28:12 UTC |
| Source: | https://github.com/cran/sddinf |
Direct estimation and inference for the difference between two high-dimensional inverse spectral densities. The inverse spectral density describes the frequency-domain partial correlation network of a multivariate time series, so its difference describes how that network changes between two conditions (e.g. brain connectivity before and after stimulation).
Start from an n x p time series for each condition, or simulate
example data with sim_example_data.
Estimate the spectral density matrix of each condition with
mvspec_welch.
Estimate the difference of the inverse spectral densities at a chosen
frequency with sdd, which uses the lasso penalized D-trace
loss and selects the penalty by eBIC.
Choose the entries to test, e.g. the non-zero entries of the estimate
with gen_nonzero_idxs.
Compute de-biased estimates, confidence intervals and p-values with
sdd_inference.
Maintainer: Michael Hellstern [email protected] [copyright holder]
Other contributors:
Huili Yuan (Author of the D-trace loss C code in src/dtl.c, ported from the archived 'Difdtl' package) [contributor, copyright holder]
Hellstern, M., Kim, B. and Shojaie, A. (2026). Assumption-lean inference for spectral differential network analysis of high-dimensional time series. arXiv preprint. doi:10.48550/arXiv.2609.13609
Yuan, H., Xi, R., Chen, C. and Deng, M. (2017). Differential network analysis via lasso penalized D-trace loss. Biometrika, 104(4), 755–770. doi:10.1093/biomet/asx049
set.seed(1) n <- 2000 p <- 10 exdata <- sim_example_data(n, p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n)))) mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n)))) # difference of inverse spectral densities at the second frequency sdd_est <- sdd(mspecsY$specs[[2]], mspecsX$specs[[2]]) # test the first few non-zero entries of the estimate idxs <- head(gen_nonzero_idxs(sdd_est$sdd), 5) inf <- sdd_inference(mspecsY$specs[[2]], mspecsX$specs[[2]], sdd_est, idxs, verbose = FALSE) inf[, c("infI", "infJ", "part", "theta_tilde", "pval")]set.seed(1) n <- 2000 p <- 10 exdata <- sim_example_data(n, p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n)))) mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n)))) # difference of inverse spectral densities at the second frequency sdd_est <- sdd(mspecsY$specs[[2]], mspecsX$specs[[2]]) # test the first few non-zero entries of the estimate idxs <- head(gen_nonzero_idxs(sdd_est$sdd), 5) inf <- sdd_inference(mspecsY$specs[[2]], mspecsX$specs[[2]], sdd_est, idxs, verbose = FALSE) inf[, c("infI", "infJ", "part", "theta_tilde", "pval")]
Opposite of expand_to_real.
contract_to_imaginary(spec)contract_to_imaginary(spec)
spec |
|
i x j complex-valued matrix
set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole") realmspecs1 <- expand_to_real(mspecs$specs[[1]]$spec) immspecs1 <- contract_to_imaginary(realmspecs1)set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole") realmspecs1 <- expand_to_real(mspecs$specs[[1]]$spec) immspecs1 <- contract_to_imaginary(realmspecs1)
Expansion occurs by computing rbind(cbind(Re(spec), -Im(spec)), cbind(Im(spec), Re(spec)))).
expand_to_real(spec)expand_to_real(spec)
spec |
|
2i x 2j real-valued matrix
set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole") realmspecs1 <- expand_to_real(mspecs$specs[[1]]$spec)set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole") realmspecs1 <- expand_to_real(mspecs$specs[[1]]$spec)
This function generates a data.frame of the non-zero entries of the input matrix est_mat that is suitable to
pass to sdd_inference.
gen_nonzero_idxs(est_mat, lower_tri = TRUE)gen_nonzero_idxs(est_mat, lower_tri = TRUE)
est_mat |
Matrix. Typically output from sdd. See examples. |
lower_tri |
Boolean (default is |
Data.frame where each row represents a non-zero entry of est_mat and the columns are the i indices (infI), j indices (infJ), and part = "b".
Part = "b" performs inference on both the real and imaginary components of an entry simultaneously. See sdd_inference.
set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") sdd_est <- sdd(mspecsX$specs[[2]], mspecsY$specs[[2]]) idxs = gen_nonzero_idxs(sdd_est$sdd)set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") sdd_est <- sdd(mspecsX$specs[[2]], mspecsY$specs[[2]]) idxs = gen_nonzero_idxs(sdd_est$sdd)
This function uses pwelch in the gsignal to form the cross-spectral density matrix from the components of pwelch.
Some of the default arguments from pwelch are changed to be suitable for SDD inference.
It is also important to note that compared to pwelch this function scales the final spectral density by 1/(2pi).
That is, the multivariate spectral densities returned by this function will be 1/(2pi) times those in pwelch.
mvspec_welch( x, window, overlap = 0, nfft = if (length(window) == 1L) window else length(window), fs = 1, detrend = "none", range = "whole" )mvspec_welch( x, window, overlap = 0, nfft = if (length(window) == 1L) window else length(window), fs = 1, detrend = "none", range = "whole" )
x |
|
window |
Window function to use. See the same argument in pwelch. |
overlap |
Numeric scalar (default is |
nfft |
Integer scalar (default is |
fs |
Integer scalar (default is |
detrend |
Character string (default is |
range |
Character string (default is |
List containing:
specs |
List. Elements are (1) spec: estimated spectral density matrix. See pwelch. (2) freq: frequency at which spec is estimated (3) orig.n: number of samples used in computing spec, (4) smoothing span of kernel used. Output is structured this way to easily pass to upstream functions like sdd. |
window |
Value of window argument. |
overlap |
Value of overlap argument. |
nfft |
Value of nfft argument. |
fs |
Value of fs argument. |
detrend |
Value of detrend argument. |
range |
Value of range argument. |
set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) # get data and specify frequency of 10 Hz (i.e. 10 samples per second) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole")set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) # get data and specify frequency of 10 Hz (i.e. 10 samples per second) Xdata <- ts(exdata$X$data, frequency = 10) mspecs <- mvspec_welch(Xdata, window = rep(1, floor(sqrt(n))), overlap = 0, fs = frequency(Xdata), detrend = "none", range = "whole")
Generates SDD estimate of spec1-1 - spec2-1 for variety of penalty values and chooses the optimal one based on eBIC. Can handle real or complex valued inputs. If inputs are complex valued they will first be expanded to the real space before estimating the difference. After estimation, the difference will be reformed in the complex space. While the intended use is to estimate the difference at the same frequency, estimating differences across frequencies is also supported.
sdd(spec1, spec2, nlambda = 20, lamseq = NULL, return_all = FALSE)sdd(spec1, spec2, nlambda = 20, lamseq = NULL, return_all = FALSE)
spec1 |
List. Typically this is output from mvspec_welch. See examples. However, it can also be any list with entries:
|
spec2 |
List. Same as spec1 but for condition 2. |
nlambda |
Integer (default is |
lamseq |
Numeric vector (default is |
return_all |
Boolean (default is |
List of SDD estimates. When return_all = TRUE, list is of length nlambda or length(lamseq).
When return_all=FALSE and there are multiple estimates with the same eBIC value, then all estimates are returned. Otherwise, only the optimal estimate is returned.
Each list of SDD estimates contains:
sdd |
Matrix representing sdd estimate. Matrix is either real or complex-valued depending on input type |
eBIC |
eBIC value of sdd. |
lam |
Lambda value at which sdd is estimated. |
freq |
Frequency at which sdd is estimated. If both |
lam_num |
Which number out of |
set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") sdd_est <- sdd(mspecsX$specs[[2]], mspecsY$specs[[2]])set.seed(1) n <- 1000 p <- 20 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") sdd_est <- sdd(mspecsX$specs[[2]], mspecsY$specs[[2]])
Generates confidence intervals and p-values for a difference in inverse spectral densities ( i.e. spec1-1 - spec2-1 ) using one of several available methods
(see ee_type). Computational efficient implementations are available by specifying n_restrict.
sdd_inference( spec1, spec2, sdd_est, idxs, nlambda = 20, ee_type = "s1Left", v_hat_method = "approximate_clime", alpha = 0.05, lambda_min_ratio = NULL, n_restrict = 0.99, n_max = 500, useMtopN = floor((2 * dim(spec1[["spec"]])[1])^(1/2)), verbose = TRUE, warn = TRUE )sdd_inference( spec1, spec2, sdd_est, idxs, nlambda = 20, ee_type = "s1Left", v_hat_method = "approximate_clime", alpha = 0.05, lambda_min_ratio = NULL, n_restrict = 0.99, n_max = 500, useMtopN = floor((2 * dim(spec1[["spec"]])[1])^(1/2)), verbose = TRUE, warn = TRUE )
spec1 |
List. Typically this is output from mvspec_welch. However, it can also be any list with entries:
|
spec2 |
List. same as spec1 but for condition 2. |
sdd_est |
List. Typically this is output from sdd. It can also be any list with entries:
|
idxs |
Data.frame. Should contains 3 columns:
|
nlambda |
Integer (default is 20). Number of tuning parameter values used to estimate the sparse projections |
ee_type |
Character (default is |
v_hat_method |
Character (default is |
alpha |
Numeric (default is |
lambda_min_ratio |
Numeric (default is |
n_restrict |
Numeric (default is |
n_max |
Numeric (default is |
useMtopN |
Numeric (default is |
verbose |
Boolean (default is |
warn |
Boolean (default is |
Data.table. Columns are
infI |
i index of the entry of sdd that we are performing inference on. |
infJ |
j index of the entry of sdd that we are performing inference on. |
part |
Options are one of |
theta_tilde |
Estimates of (infI, infJ) entry. These differ from |
ci_lower |
Lower part of the CI. For |
ci_upper |
Upper part of the CI. For |
pval |
p-value. |
test_stat |
Test statistic |
asympVars |
Asymptotic variance estimate. For |
asympNscale |
Asymptotic scaling in n. This is |
n_var_est_used |
Number of components used to estimate asymptotic variance. |
freq |
Frequency for which we are performing inference. |
p |
Number of variables. |
n |
Number of observations. |
ee_type |
Same as |
v_hat_method |
Same as |
alpha |
Same as |
set.seed(1) n <- 2000 p <- 10 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") spec1 <- mspecsY$specs[[2]] spec2 <- mspecsX$specs[[2]] sdd_est <- sdd(spec1, spec2) idxs_use <- data.frame(infI = c(2, 8,6,8,9, 8,8,9), infJ = c(1, 9, 2, 10, 10, 9, 10, 10), part = c("r","b", "b","r","r", "i","i","i")) # s1Right - restricted variance est inf_s1Right <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Right") # s1Left - restricted variance est inf_s1Left <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Left") # s1Left_Right - restricted variance est inf_s1Left_Right <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Left_Right") # symm - restricted variance est inf_symm <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "symm", v_hat_method = "exact") # unrestricted variance est - can be very slow if components used to estimate variances is large inf_symm_unr <- sdd_inference(spec1, spec2, sdd_est, idxs_use, nlambda=20, ee_type = "symm", v_hat_method = "exact", n_restrict = NULL, n_max = NULL, useMtopN = NULL)set.seed(1) n <- 2000 p <- 10 exdata <- sim_example_data(n,p) mspecsX <- mvspec_welch(exdata$X$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") mspecsY <- mvspec_welch(exdata$Y$data, window = rep(1, floor(sqrt(n))), overlap = 0, fs = 1, detrend = "none", range = "whole") spec1 <- mspecsY$specs[[2]] spec2 <- mspecsX$specs[[2]] sdd_est <- sdd(spec1, spec2) idxs_use <- data.frame(infI = c(2, 8,6,8,9, 8,8,9), infJ = c(1, 9, 2, 10, 10, 9, 10, 10), part = c("r","b", "b","r","r", "i","i","i")) # s1Right - restricted variance est inf_s1Right <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Right") # s1Left - restricted variance est inf_s1Left <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Left") # s1Left_Right - restricted variance est inf_s1Left_Right <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "s1Left_Right") # symm - restricted variance est inf_symm <- sdd_inference(spec1, spec2, sdd_est, nlambda=20, idxs = idxs_use, ee_type = "symm", v_hat_method = "exact") # unrestricted variance est - can be very slow if components used to estimate variances is large inf_symm_unr <- sdd_inference(spec1, spec2, sdd_est, idxs_use, nlambda=20, ee_type = "symm", v_hat_method = "exact", n_restrict = NULL, n_max = NULL, useMtopN = NULL)
Simulates data for two conditions (X and Y) from a VAR1 process.
sim_example_data( n, p, minA = 0.4, maxA = 0.8, propNonZero_full = 0.05, propNonZero_add = 0.6, addMatDim = 3 )sim_example_data( n, p, minA = 0.4, maxA = 0.8, propNonZero_full = 0.05, propNonZero_add = 0.6, addMatDim = 3 )
n |
Sample size. |
p |
Dimension. |
minA |
Minimum value of transition matrix. |
maxA |
Maximum value of transition matrix. |
propNonZero_full |
Proportion of entries of the |
propNonZero_add |
Proportion of entries of the |
addMatDim |
Size of additional block. |
Data is simulated from e.g.
where is are independent standard normal errors.
The transition matrices, A, in both conditions will be block diagonal with two blocks. The first is size p-addMatDim and the second is of size addMatDim. The transition matrix in condition Y will be the same as conditions X except the last addMatDim block
is multiplied by -1. Entries are generated from either a Uniform(minA,maxA) or Uniform(-maxA, -minA) each with
equal probability. Density of the p-addMatDim and addMatDim blocks are controlled with propNonZero_full and propNonZero_add respectively.
List containing:
X |
List of transition matrix (A) and data. |
Y |
List of transition matrix (A) and data. |
set.seed(1) exdata <- sim_example_data(1000,20) Xdata <- exdata$X$data Ydata <- exdata$Y$dataset.seed(1) exdata <- sim_example_data(1000,20) Xdata <- exdata$X$data Ydata <- exdata$Y$data