Introduction

This vignette provides a comprehensive guide for performing permutation tests to compare quantiles of two groups in R. A permutation test is a non-parametric statistical method that involves rearranging the data points to generate a distribution of a test statistic under the null hypothesis. This technique is particularly useful when the assumptions required for traditional parametric tests are not met, providing a robust alternative for statistical inference.

In experiments involving two groups, it is essential to assess whether there are significant differences between their distributions. Traditional methods might focus on comparing means or medians, but quantile comparison offers a deeper insight into the distributional characteristics of the groups. Quantiles, such as the median (50th percentile) or quartiles (25th and 75th percentiles), divide the data into equal parts and highlight specific points in the dataset that are critical for understanding the distribution’s shape and spread.

This vignette demonstrates how to implement permutation tests for quantile comparison using the permtest function in the package groupcompare. By following the steps outlined, users will be able to conduct thorough statistical analyses and draw meaningful conclusions about their data.

Required Packages and Data Sets

Install and load the package gpcomp

The recent version of the package from CRAN is installed with the following command:

install.packages("groupcompare", dep=TRUE)

If you have already installed groupcompare, you can load it into R working environment by using the following command:

library("groupcompare")

Data sets

The dataset to be analyzed can be in wide format where the values for Group 1 and Group 2 are written in two separate columns or in long format where the values are entered in the first column and the group names are entered in the second column. In the following code chunk, a dataset named ds1 is created using the ghdist function to simulate the G&H distribution. The generated dataset contains data for two groups named A and B, each consisting of 25 observations, with a mean of 50 and a standard deviation of 2. In the example, by assigning zeros to the skewness (g) and kurtosis (h) arguments, the simulated data is intended to have a normal distribution. As expected, the means and variances of groups A and B are done equal. In the example, the generated dataset is in wide format, and immediately after, it is converted to long format using the wide2long function to create the dataset ds2. This provides an idea of the long data format, and as can be seen, in the long data format, the first column contains the observation values, while the second column contains the group names or codes. As understood from the example, different groups can be created by changing the means, variances, skewness, and kurtosis parameters.

 set.seed(12) # For reproducibility purpose
 grp1 <- ghdist(50, 50, 2, g=0, h=0)
 grp2 <- ghdist(50, 45, 4, g=0.8, h=0)
 ds1 <- data.frame(grp1=grp1, grp2=grp2)
 head(ds1)

##       grp1     grp2
## 1 47.03886 44.83214
## 2 53.15434 44.56903
## 3 48.08651 47.20590
## 4 48.15999 65.17133
## 5 46.00472 42.15702
## 6 49.45541 48.99942

 # Data in long format
 ds2 <- wide2long(ds1)
 head(ds2)

##        obs group
## 1 47.03886  grp1
## 2 53.15434  grp1
## 3 48.08651  grp1
## 4 48.15999  grp1
## 5 46.00472  grp1
## 6 49.45541  grp1

Data Visualization

For statistical tests, data visualization is performed before the analysis to provide insights about the structure or distribution of the data. In the comparison of two groups using parametric tests such as the t-test, visualization provides preliminary information on whether the assumptions of the test are met. The bivarplot function in the following code chunk facilitates the examination and comparison of group data using various plots.

bivarplot(ds2)

Resampling Statistics with Permutations

The permtest function of the package groupcompare, given an example of its usage in the following code chunk, compares the groups in the dataset using permutations and returns the results. In the result object, pval contains the p-values of for the differences of each statictic related to two compared groups. (Please note that the value of R should be higher (e.g. 3000) in real-world applications.)

results <- permtest(ds2, statistic=calcquantdif, alternative="two.sided", R=500)

In the code chunk above, ds2 is the name of dataset in long data format in which the group names locate in the second column. As a mandatory argument, the statistic is the name of the function that calculates and returns the quantile differences of the compared groups. In the example, calcquantdif is a function that calculates and returns the differences between group quantiles. Among its arguments, alternative shows the type of null hyphothesis. The default is two.sided but it can be set to less or greater for a single-tail alternative hyphothesis as well. R shows the number of repetitions for bootstrap. The default number of permutations is 5000 but it is recommended to increase up to a number as high as 20000.

Results and Interpretation

Below, the code chunk displays the structure of results object and the results stored obtained above. Here, results$tstar stores the statistics obtained in permutations and can be used for further visualization and analysis. As it is seen, the p-values are 2.20e-16 for the difference between all of the quantiles, meaning all the quantiles are siginificantly different.

str(results)

## List of 7
##  $ t0         : Named num [1:9] 5.62 4.95 4.86 4.98 4.5 ...
##   ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
##  $ tstar      : num [1:500, 1:9] -1.325 0.523 -0.974 -0.109 -0.641 ...
##   ..- attr(*, "dimnames")=List of 2
##   .. ..$ : NULL
##   .. ..$ : chr [1:9] "P10" "P20" "P30" "P40" ...
##  $ pval       : Named num [1:9] 2.2e-16 2.2e-16 2.2e-16 2.2e-16 2.2e-16 ...
##   ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
##  $ alternative: chr "two.sided"
##  $ R          : num 500
##  $ pvalsk     : logi [1:2] NA NA
##  $ call       : language permtest(x = ds2, statistic = calcquantdif, alternative = "two.sided",      R = 500)

results$t0

##       P10       P20       P30       P40       P50       P60       P70       P80 
## 5.6160059 4.9501201 4.8594475 4.9845448 4.4981128 3.5381490 2.8770037 0.9401363 
##       P90 
## 0.1316084

head(results$tstar)

##             P10        P20        P30        P40         P50          P60
## [1,] -1.3245136 -1.7530604 -2.6483186 -1.0072093 -1.23359354 -0.229271264
## [2,]  0.5231460 -0.3013630 -0.3761942 -0.6354308 -1.42081451 -0.866892293
## [3,] -0.9744594  0.1857390 -0.3761942  0.4736984 -0.01297126  0.259526066
## [4,] -0.1085207  0.6999764  1.2457520  0.1484691 -0.53111609  0.013482962
## [5,] -0.6413481  0.2435761  1.3410715  0.6504253 -0.06140080  0.009553991
## [6,] -0.2829107  0.5266653  1.1272236  0.9468752  0.61473382 -0.139181597
##             P70         P80         P90
## [1,] -0.5960080 -0.64748245 -1.03102517
## [2,] -0.8817546 -0.46472389 -0.78346152
## [3,]  0.5389690  0.51634212  0.10362627
## [4,]  0.3276260  0.03798994  1.15707616
## [5,]  0.4385084  0.14315489 -0.08459354
## [6,] -0.4630248 -0.63551890 -1.18610636

results$pval

##      P10      P20      P30      P40      P50      P60      P70      P80 
## 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 6.20e-02 
##      P90 
## 9.46e-01

Other Tests

In addition to permutation test, performing bootstrap can be useful for validation of the results when comparing the quantiles of two groups. For this purpose, the bootstrap function in the groupcompare package can be run. For details, you can refer to the usage documentation of the package as well as the vignette titled Quantiles Comparison of Two Groups with Bootstrap.

While p-values for the difference of percentiles of two groups have been calculated here, significance differences for other group statistics can be determined by passing different function names to the statistic argument. For example, when the calcstatdif function is assigned to the statistic argument in the example above, p-values can be calculated for the differences between the means, medians, IQRs, and variances of the two groups.

Comparison of Quantiles of Two Groups with Permutation Tests