Families of tests and pairwise comparisons

This vignette is precomputed: the results shown were obtained with the code displayed, which takes a few minutes to run.

Analyses of physiological signals rarely end with a single test. Several features are often extracted from the same trials (mean, peak, late response), and factors often have more than two levels, which calls for pairwise comparisons. This vignette shows how lmmresample controls the family-wise error rate in both situations while keeping the power that corrections such as Bonferroni or Holm give away.

library(lmmresample)
library(lme4)

Several features from the same trials

We simulate time series and extract three features per trial: the mean over the whole trial, the peak, and the mean of the late window (from 1 s).

ts <- sim_blocks("within", level = "timeseries", n_participants = 20,
                 n_trials = 30, n_time = 50, sampling_rate = 25,
                 effect_condition = 0.3, effect_onset = 0.5, sd_slope = 0.3,
                 ar1 = 0.9, seed = 3)
feat <- aggregate(y ~ participant + trial + condition, ts, mean)
names(feat)[names(feat) == "y"] <- "mean"
feat$peak <- aggregate(y ~ participant + trial + condition, ts, max)$y
feat$late <- aggregate(y ~ participant + trial + condition,
                       ts[ts$time >= 1, ], mean)$y
head(feat)
#>   participant   trial condition       mean        peak       late
#> 1         p01 p01_t01         A  0.1159974  1.79451310  0.2730235
#> 2         p01 p01_t02         A -1.0140783 -0.04281802 -1.2378820
#> 3         p01 p01_t03         A -1.9399087 -0.07689758 -1.9605965
#> 4         p01 p01_t08         A -0.3624909  1.35299691 -0.2773798
#> 5         p01 p01_t09         A  0.6104789  2.97043862  0.3173673
#> 6         p01 p01_t10         A  1.5867119  2.66694798  1.7880359

One model per feature:

m_mean <- lmer(mean ~ condition + (1 | participant), data = feat)
m_peak <- lmer(peak ~ condition + (1 | participant), data = feat)
m_late <- lmer(late ~ condition + (1 | participant), data = feat)

The features are computed from the same trials, so they are correlated. perm_maxt() permutes all models together, with the same permutation of trials for every feature, and compares each observed statistic with the distribution of the largest statistic across the family. This controls the family-wise error rate and, unlike Bonferroni or Holm, takes the correlation between features into account: the more similar the features, the smaller the penalty.

fam <- perm_maxt(mean = perm_spec(m_mean, "condition"),
                 peak = perm_spec(m_peak, "condition"),
                 late = perm_spec(m_late, "condition"),
                 unit = "trial", exchange = exch_signflip("participant"),
                 B = 1999, seed = 1)
fam
#> 
#> Family-wise permutation tests for `condition` (max-t, step-down)
#> 
#> Method:    relabelling of units
#> Exchange:  labels of `trial` swapped (sign-flipped) by `participant` (600 units in 20 blocks)
#> Resamples: 1999 of 1999 permutations used
#> 
#>  Test Statistic   p (unadjusted) p (family-wise)
#>  mean |t| = 3.955 0.002          0.002          
#>  peak |t| = 4.453 0.001          0.002          
#>  late |t| = 4.959 0.002          0.002
plot(fam)
plot of chunk maxt-plot
plot of chunk maxt-plot

The step-down version (the default) is uniformly more powerful than the single-step one and controls the same error rate. combine = "min-p" uses p-values instead of statistics, which is preferable when the statistics are on different scales, for instance when combining tests with different numbers of coefficients. The same function combines terms of one model, such as the terms of a growth curve analysis:

perm_maxt(level = perm_spec(m_gca, "condition"),
          linear = perm_spec(m_gca, "ot1:condition"),
          quadratic = perm_spec(m_gca, "ot2:condition"),
          unit = "trial", exchange = exch_signflip("participant"))

Pairwise comparisons between levels of a factor

With a factor of more than two levels, a significant omnibus test is usually followed by pairwise comparisons. perm_pairwise() tests every pair by permutation, adjusts the p-values across pairs and, optionally, gives bootstrap intervals for each difference.

Here, stimuli of three types are presented, and only the negative stimuli increase the response. Trial order is included as a covariate.

d <- sim_blocks("within", level = "trial", n_participants = 24, n_trials = 48,
                seed = 4)
d$condition <- NULL
set.seed(4)
d$stimulus <- factor(sample(c("neutral", "positive", "negative"), nrow(d),
                            replace = TRUE),
                     levels = c("neutral", "positive", "negative"))
d$y <- d$y + 0.3 * (d$stimulus == "negative")
d$order <- as.numeric(scale(d$trial_index))
m <- lmer(y ~ stimulus + order + (1 | participant), data = d)
pw <- perm_pairwise(m, "stimulus", unit = "trial",
                    exchange = exch_signflip("participant"),
                    cluster = "participant", B = 1999, seed = 1)
pw
#> 
#> Pairwise permutation tests of `stimulus` between levels of `stimulus`
#> 
#> Method:     Freedman-Lane (residuals of the reduced model)
#> Exchange:   labels of `trial` swapped (sign-flipped) by `participant` (1152 units in 24 blocks)
#> Adjustment: Westfall-Young max-t, step-down
#> Resamples:  1999 of 1999 permutations used
#> 
#>  Comparison          Estimate 95% CI          Statistic   p (unadjusted)
#>  positive - neutral  -0.047   [-0.147, 0.041] |t| = 1.034 0.291         
#>  negative - neutral  0.256    [0.159, 0.349]  |t| = 5.629 < 0.001       
#>  negative - positive 0.303    [0.215, 0.382]  |t| = 6.583 < 0.001       
#>  p (adjusted)
#>  0.291       
#>  < 0.001     
#>  < 0.001     
#> 
#> Intervals: expanded percentile bootstrap, not adjusted for multiplicity.

Each pair is tested with the other levels and the covariates kept in the model (a Freedman-Lane test in a reparametrised model), so a difference between two levels does not leak into the comparison of the other two. This keeps the error rate under control even when some levels differ and others do not. The default adjustment, Westfall-Young, is the max-t procedure across pairs; adjust = "holm" and adjust = "none" are also available.

cluster = "participant" adds bootstrap intervals of each difference, which take the participant structure into account. tidy() returns everything as a data frame:

tidy(pw)
#>            comparison reference    level    estimate stat statistic p.value
#> 1  positive - neutral   neutral positive -0.04738957  |t|  1.033848  0.2910
#> 2  negative - neutral   neutral negative  0.25581385  |t|  5.629209  0.0005
#> 3 negative - positive  positive negative  0.30320342  |t|  6.582779  0.0005
#>   p.adjusted   conf.low  conf.high         adjust
#> 1     0.2910 -0.1466948 0.04072638 westfall-young
#> 2     0.0005  0.1593071 0.34932389 westfall-young
#> 3     0.0005  0.2149402 0.38227683 westfall-young

Pairwise comparisons of time courses

For time-series models, term names the interaction to compare, and time identifies the samples within trials, exactly as in perm_test():

library(splines)
mt <- lmer(y ~ stimulus * ns(time, 4) + (1 | participant), data = d_timeseries)
perm_pairwise(mt, "stimulus", term = "stimulus:ns(time, 4)",
              unit = "trial", exchange = exch_signflip("participant"),
              time = "time")

In simulations, pairwise Wald tests of time-course models with Holm’s correction declared a false difference in about 80% of data sets, whereas perm_pairwise() kept the family-wise error rate at 5%.