Package 'DataSimilarity'

Quantifying Similarity of Datasets and Multivariate Two- And k-Sample Testing

Description

A collection of methods for quantifying the similarity of two or more datasets, many of which can be used for two- or k-sample testing. It provides newly implemented methods as well as wrapper functions for existing methods that enable calling many different methods in a unified framework. The methods were selected from the review and comparison of Stolte et al. (2024) <doi:10.1214/24-SS149>.

Details

The DESCRIPTION file:

Package:	DataSimilarity
Type:	Package
Title:	Quantifying Similarity of Datasets and Multivariate Two- And k-Sample Testing
Version:	0.1.1
Date:	2025-03-18
Authors@R:	c(person(given = "Marieke", family = "Stolte", email = "stolte@statistik.tu-dortmund.de", role = c("aut", "cre", "cph"), comment = c(ORCID = "0009-0002-0711-6789")), person(given = "Luca", family = "Sauer", role = c("aut")), person(given = "David", family = "Alvarez-Melis", role = c("ctb"), comment = "Original python implementation of OTDD, <https://github.com/microsoft/otdd.git>"), person(given = "Nabarun", family = "Deb", role = c("ctb"), comment = "Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>"), person(given = "Bodhisattva", family = "Sen", role = c("ctb"), comment = "Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>"))
Depends:	R (>= 3.5.0)
Imports:	boot, stats
Suggests:	ade4, approxOT, Ball, caret, clue, cramer, crossmatch, dbscan, densratio, DWDLargeR, e1071, Ecume, energy, expm, FNN, gTests, gTestsMulti, HDLSSkST, hypoRF, kernlab, kerTests, KMD, knitr, LPKsample, Matrix, mvtnorm, nbpMatching, pROC, purrr, randtoolbox, rlemon, rpart, rpart.plot, testthat, RSNNS
Description:	A collection of methods for quantifying the similarity of two or more datasets, many of which can be used for two- or k-sample testing. It provides newly implemented methods as well as wrapper functions for existing methods that enable calling many different methods in a unified framework. The methods were selected from the review and comparison of Stolte et al. (2024) <doi:10.1214/24-SS149>.
License:	GPL (>= 3)
NeedsCompilation:	no
Packaged:	2025-03-18 16:10:24 UTC; stolte
Author:	Marieke Stolte [aut, cre, cph] (<https://orcid.org/0009-0002-0711-6789>), Luca Sauer [aut], David Alvarez-Melis [ctb] (Original python implementation of OTDD, <https://github.com/microsoft/otdd.git>), Nabarun Deb [ctb] (Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>), Bodhisattva Sen [ctb] (Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>)
Maintainer:	Marieke Stolte <stolte@statistik.tu-dortmund.de>
Repository:	CRAN
Date/Publication:	2025-03-18 21:10:06 UTC

Index of help topics:

BF                      Baringhaus and Franz (2010) rigid motion
                        invariant multivariate two-sample test
BG                      Biau and Gyorfi (2005) two-sample homogeneity
                        test
BG2                     Biswas and Ghosh (2014) Two-Sample Test
BMG                     Biswas et al. (2014) two-sample run test
BQS                     Barakat et al. (1996) Two-Sample Test
Bahr                    Bahr (1996) multivariate two-sample test
BallDivergence          Ball Divergence based two- or k-sample test
C2ST                    Classifier Two-Sample Test
CCS                     Weighted Edge-Count Two-Sample Test
CCS_cat                 Weighted Edge-Count Two-Sample Test for
                        Discrete Data
CF                      Generalized Edge-Count Test
CF_cat                  Generalized Edge-Count Test for Discrete Data
CMDistance              Constrained Minimum Distance
Cramer                  Cramér Two-Sample Test
DISCOB                  Distance Components (DISCO) Tests
DISCOF                  Distance Components (DISCO) Tests
DS                      Rank-Based Energy Test (Deb and Sen, 2021)
DataSimilarity-package
                        Quantifying Similarity of Datasets and
                        Multivariate Two- And k-Sample Testing
DiProPerm               Direction-Projection-Permutation (DiProPerm)
                        Test
Energy                  Energy Statistic and Test
FR                      Friedman-Rafsky Test
FR_cat                  Friedman-Rafsky Test for Discrete Data
FStest                  Multisample FS Test
GGRL                    Decision-Tree Based Measure of Dataset Distance
                        and Two-Sample Test
GPK                     Generalized Permutation-Based Kernel (GPK)
                        Two-Sample Test
HMN                     Random Forest Based Two-Sample Test
HamiltonPath            Shortest Hamilton path
Jeffreys                Jeffreys divergence
KMD                     Kernel Measure of Multi-Sample Dissimilarity
                        (KMD)
LHZ                     Li et al. (2022) empirical characteristic
                        distance
LHZStatistic            Calculation of the Li et al. (2022) empirical
                        characteristic distance
MMCM                    Multisample Mahalanobis Crossmatch (MMCM) Test
MMD                     Maximum Mean Discrepancy (MMD) Test
MST                     Minimum Spanning Tree (MST)
MW                      Nonparametric Graph-Based LP (GLP) Test
NKT                     Decision-Tree Based Measure of Dataset
                        Similarity (Ntoutsi et al., 2008)
OTDD                    Optimal Transport Dataset Distance
Petrie                  Multisample Crossmatch (MCM) Test
RItest                  Multisample RI Test
Rosenbaum               Rosenbaum Crossmatch Test
SC                      Graph-Based Multi-Sample Test
SH                      Schilling-Henze Nearest Neighbor Test
Wasserstein             Wasserstein Distance based Test
YMRZL                   Yu et al. (2007) Two-Sample Test
ZC                      Maxtype Edge-Count Test
ZC_cat                  Maxtype Edge-Count Test for Discrete Data
dipro.fun               Direction-Projection Functions for DiProPerm
                        Test
engineerMetric          Engineer Metric
gTests                  Graph-Based Tests
gTestsMulti             Graph-Based Multi-Sample Test
gTests_cat              Graph-Based Tests for Discrete Data
kerTests                Generalized Permutation-Based Kernel (GPK)
                        Two-Sample Test
knn                     K-Nearest Neighbor Graph
rectPartition           Calculate a rectangular partition
stat.fun                Univariate Two-Sample Statistics for DiProPerm
                        Test

Further information is available in the following vignettes:

vignette	Using DataSimilarity (source, pdf)

The package provides various methods for comparing two or more datasets or their underlying distributions. Often, a permutation or asymptotic test for the null hypothesis of equal distributions $H_0: F_1 = F_2$ or $H_0: F_1 = \dots = F_k$ is performed.

Author(s)

Marieke Stolte [aut, cre, cph] (<https://orcid.org/0009-0002-0711-6789>), Luca Sauer [aut], David Alvarez-Melis [ctb] (Original python implementation of OTDD, <https://github.com/microsoft/otdd.git>), Nabarun Deb [ctb] (Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>), Bodhisattva Sen [ctb] (Original implementation of rank-based Energy test (DS), <https://github.com/NabarunD/MultiDistFree.git>)

Maintainer: Marieke Stolte <stolte@statistik.tu-dortmund.de>

References

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

Stolte, M., Kappenberg, F., Rahnenführer, J. & Bommert, A. (2024). A Comparison of Methods for Quantifying Dataset Similarity. https://shiny.statistik.tu-dortmund.de/data-similarity/

---

Bahr (1996) multivariate two-sample test

Description

The function implements the Bahr (1996) multivariate two-sample test. This test is a special case of the rigid-motion invariant multivariate two-sample test of Baringhaus and Franz (2010). The implementation here uses the cramer.test implementation from the cramer package.

Usage

Bahr(X1, X2, n.perm = 0, just.statistic = n.perm <= 0, 
      sim = "ordinary", maxM = 2^14, K = 160, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permutation or Bootstrap test, respectively (default: 0, no permutation test performed)
just.statistic	Should only the test statistic be calculated without performing any test (default: TRUE if number of permutations is set to 0 and FALSE if number of permutations is set to any positive number)
sim	Type of Bootstrap or eigenvalue method for testing. Possible options are "ordinary" (default) for ordinary Boostrap, "permutation" for permutation testing, or "eigenvalue" for bootstrapping the limit distribution (especially good for datasets too large for performing Bootstrapping). For more details see cramer.test
maxM	Maximum number of points used for fast Fourier transform involved in eigenvalue method for approximating the null distribution (default: 2^14). Ignored if sim is either "ordinary" or "permutation". For more details see cramer.test.
K	Upper value up to which the integral for calculating the distribution function from the characteristic function is evaluated (default: 160). Note: when K is increased, it is necessary to also increase maxM. Ignored if sim is either "ordinary" or "permutation". For more details see cramer.test.
seed	Random seed (default: 42)

Details

The Bahr (1996) test is a specialcase of the test of Bahrinhaus and Franz (2010)

$T_{n_1, n_2} = \frac{n_1 n_2}{n_1+n_2}\left(\frac{2}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2} \phi(||X_{1i} - X_{2j}||^2) - \frac{1}{n_1^2}\sum_{i,j=1}^{n_1} \phi(||X_{1i} - X_{1j}||^2) - \frac{1}{n_2^2}\sum_{i,j=1}^{n_2} \phi(||X_{2i} - X_{2j}||^2)\right)$

where the kernel function $\phi$ is set to

$\phi_{\text{Bahr}}(x) = 1 - \exp(-x/2).$

The theoretical statistic underlying this test statistic is zero if and only if the distributions coincide. Therefore, low values of the test statistic incidate similarity of the datasets while high values indicate differences between the datasets.

This implementation is a wrapper function around the function cramer.test that modifies the in- and output of that function to match the other functions provided in this package. For more details see the cramer.test.

Value

An object of class htest with the following components:

method	Description of the test
d	Number of variables in each dataset
m	Sample size of first dataset
n	Sample size of second dataset
statistic	Observed value of the test statistic
p.value	Boostrap/ permutation p value (only if n.perm > 0)
sim	Type of Boostrap or eigenvalue method (only if n.perm > 0)
n.perm	Number of permutations for permutation or Boostrap test
hypdist	Distribution function under the null hypothesis reconstructed via fast Fourier transform. $x contains the x-values, $Fx contains the corresponding distribution function values. (only if n.perm > 0)
ev	Eigenvalues and eigenfunctions when using the eigenvalue method (only if n.perm > 0)
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Baringhaus, L. and Franz, C. (2010). Rigid motion invariant two-sample tests, Statistica Sinica 20, 1333-1361

Bahr, R. (1996). Ein neuer Test fuer das mehrdimensionale Zwei-Stichproben-Problem bei allgemeiner Alternative, German, Ph.D. thesis, University of Hanover

Franz, C. (2024). cramer: Multivariate Nonparametric Cramer-Test for the Two-Sample-Problem. R package version 0.9-4, https://CRAN.R-project.org/package=cramer.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

BF, Cramer, Energy

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Bahr test 
if(requireNamespace("cramer", quietly = TRUE)) {
  Bahr(X1, X2, n.perm = 100)
}

---

Ball Divergence based two- or $k$-sample test

Description

The function implements the Pan et al. (2018) multivariate two- or $k$-sample test based on the Ball Divergence. The implementation here uses the bd.test implementation from the Ball package.

Usage

BallDivergence(X1, X2, ..., n.perm = 0, seed = 42, num.threads = 0, 
                kbd.type = "sum", weight = c("constant", "variance"), 
                args.bd.test = NULL)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
...	Optionally more datasets as matrices or data.frames
n.perm	Number of permutations for permutation test (default: 0, no permutation test performed). Note that for more than two samples, no test is performed.
seed	Random seed (default: 42)
num.threads	Number of threads (default: 0, all available cores are used)
kbd.type	Character specifying which k-sample test statistic will be used. Must be one of "sum" (default), "maxsum", or "max".
weight	Character specifying the weight form of the Ball Divergence test statistic. Must be one of "constant" (default) or "variance".
args.bd.test	Further arguments passed to bd.test as a named list.

Details

For n.perm = 0, the asymptotic test is performed. For n.perm > 0, a permutation test is performed.

The Ball Divergence is defined as the square of the measure difference over a given closed ball collection. The empirical test performed here is based on the difference between averages of metric ranks. It is robust to outliers and heavy-tailed data and suitable for imbalanced sample sizes.

The Ball Divergence of two distributions is zero if and only if the distributions coincide. Therefore, low values of the test statistic indicate similarity and the test rejects for large values of the test statistic.

For the $k$-sample problem the pairwise Ball divergences can be summarized in different ways. First, one can simply sum up all pairwise Ball divergences (kbd.type = "sum"). Next, one can find the sample with the largest difference to the other, i.e. take the maximum of the sums of all Ball divergences for each sample with all other samples (kbd.type = "maxsum"). Last, one can sum up the largest $k-1$ pairwise Ball divergences (kbd.type = "max").

This implementation is a wrapper function around the function bd.test that modifies the in- and output of that function to match the other functions provided in this package. For more details see bd.test and bd.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Permutation p value (only if n.perm > 0 and for two datasets)
n.perm	Number of permutations for permutation test
size	Number of observations for each dataset
method	Description of the test
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	Yes

References

Pan, W., T. Y. Tian, X. Wang, H. Zhang (2018). Ball Divergence: Nonparametric two sample test, Annals of Statistics 46(3), 1109-1137, doi:10.1214/17-AOS1579.

J. Zhu, W. Pan, W. Zheng, and X. Wang (2021). Ball: An R Package for Detecting Distribution Difference and Association in Metric Spaces, Journal of Statistical Software, 97(6), doi:10.18637/jss.v097.i06

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Calculate Ball Divergence and perform test 
if(requireNamespace("Ball", quietly = TRUE)) {
  BallDivergence(X1, X2, n.perm = 100)
}

---

Baringhaus and Franz (2010) rigid motion invariant multivariate two-sample test

Description

The function implements the Baringhaus and Franz (2010) multivariate two-sample test. The implementation here uses the cramer.test implementation from the cramer package.

Usage

BF(X1, X2, n.perm = 0, just.statistic = n.perm <= 0, kernel = "phiLog", 
    sim = "ordinary", maxM = 2^14, K = 160, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permutation or Bootstrap test, respectively (default: 0, no permutation test performed)
just.statistic	Should only the test statistic be calculated without performing any test (default: TRUE if number of permutations is set to 0 and FALSE if number of permutations is set to any positive number)
kernel	Name of the kernel function as character. Possible options are "phiLog" (default), "phiFracA", and "phiFracB". Alternatively, a user-defined function can be supplied. The function should allow a matrix as input and fulfill the following properties. The output should be non-negative, the value of 0 should be mapped to 0, and the first derivative should be non-constant completely monotone.
sim	Type of Bootstrap or eigenvalue method for testing. Possible options are "ordinary" (default) for ordinary Boostrap, "permutation" for permutation testing, or "eigenvalue" for bootstrapping the limit distribution (especially good for datasets too large for performing Bootstrapping). For more details see cramer.test
maxM	Maximum number of points used for fast Fourier transform involved in eigenvalue method for approximating the null distribution (default: 2^14). Ignored if sim is either "ordinary" or "permutation". For more details see cramer.test.
K	Upper value up to which the integral for calculating the distribution function from the characteristic function is evaluated (default: 160). Note: when K is increased, it is necessary to also increase maxM. Ignored if sim is either "ordinary" or "permutation". For more details see cramer.test.
seed	Random seed (default: 42)

Details

The Bahrinhaus and Franz (2010) test statistic

$T_{n_1, n_2} = \frac{n_1 n_2}{n_1+n_2}\left(\frac{2}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2} \phi(||X_{1i} - X_{2j}||^2) - \frac{1}{n_1^2}\sum_{i,j=1}^{n_1} \phi(||X_{1i} - X_{1j}||^2) - \frac{1}{n_2^2}\sum_{i,j=1}^{n_2} \phi(||X_{2i} - X_{2j}||^2)\right)$

is defined using a kernel function $\phi$. A choice recommended preferably for location alternatives is

$\phi_{\text{log}}(x) = \log(1 + x),$

two choices recommended preferably for dispersion alternatives are

$\phi_{\text{FracA}}(x) = 1 - \frac{1}{1+x}$

and

$\phi_{\text{FracB}}(x) = 1 - \frac{1}{(1+x)^2}.$

The theoretical statistic underlying this test statistic is zero if and only if the distributions coincide. Therefore, low values of the test statistic incidate similarity of the datasets while high values indicate differences between the datasets.

This implementation is a wrapper function around the function cramer.test that modifies the in- and output of that function to match the other functions provided in this package. For more details see cramer.test.

Value

An object of class htest with the following components:

method	Description of the test
d	Number of variables in each dataset
m	Sample size of first dataset
n	Sample size of second dataset
statistic	Observed value of the test statistic
p.value	Boostrap/ permutation p value (only if n.perm > 0)
sim	Type of Boostrap or eigenvalue method (only if n.perm > 0)
n.perm	Number of permutations for permutation or Boostrap test
hypdist	Distribution function under the null hypothesis reconstructed via fast Fourier transform. $x contains the x-values, $Fx contains the corresponding distribution function values. (only if n.perm > 0)
ev	Eigenvalues and eigenfunctions when using the eigenvalue method (only if n.perm > 0)
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Baringhaus, L. and Franz, C. (2010). Rigid motion invariant two-sample tests, Statistica Sinica 20, 1333-1361

Franz, C. (2024). cramer: Multivariate Nonparametric Cramer-Test for the Two-Sample-Problem. R package version 0.9-4, https://CRAN.R-project.org/package=cramer.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Bahr, Cramer, Energy

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Baringhaus and Franz test 
if(requireNamespace("cramer", quietly = TRUE)) {
  BF(X1, X2, n.perm = 100)
  BF(X1, X2, n.perm = 100, kernel = "phiFracA")
  BF(X1, X2, n.perm = 100, kernel = "phiFracB")
}

---

Biau and Gyorfi (2005) two-sample homogeneity test

Description

The function implements the Biau and Gyorfi (2005) two-sample homogeneity test. This test uses the $L_1$-distance between two empicial distribution functions restricted to a finite partition.

Usage

BG(X1, X2, partition = rectPartition, exponent = 0.8, eps = 0.01, seed = 42, ...)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame of the same sample size as X1
partition	Function that creates a finite partition for the subspace spanned by the two datasets (default: rectPartition, see Details)
exponent	Exponent used in the partition function, should be between 0 and 1 (default: 0.8)
eps	Small threshold to guarantee edge points are included (default: 0.01)
seed	Random seed (default: 42)
...	Further arguments to be passed to the partition function

Details

The Biau and Gyorfi (2005) two-sample homogeneity test is defined for two datasets of the same sample size.

By default a rectangular partition (rectPartition) is being calculated under the assumption of approximately equal cell probabilities. Use the exponent argument to choose the number of elements of the partition $m_n$ accoring to the convergence criteria in Biau and Gyorfi (2005). By default choose $m_n = n^{0.8}$. For each of the $p$ variables of the datasets, create $m_n^{1/p} + 1$ cutpoints along the range of both datasets to define the partition, and ensure at least three cutpoints exist per variable (min, max, and one point splitting the data into two bins).

The test statistic is the $L_1$-distance between the vectors of the proportions of points falling into each cell of the partition for each dataset. An asymptotic test is performed using a standardized version of the $L_1$ distance that is approximately standard normally distributed (Corollary to Theorem 2 in Biau and Gyorfi (2005)). Low values of the test statistic indicate similarity. Therefore, the test rejects for large values of the test statistic.

Value

An object of class htest with the following components:

statistic	Observed value of the (asymptotic) test statistic
p.value	p value
method	Description of the test
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Biau G. and Gyorfi, L. (2005). On the asymptotic properties of a nonparametric $L_1$-test statistic of homogeneity, IEEE Transactions on Information Theory, 51(11), 3965-3973. doi:10.1109/TIT.2005.856979

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

rectPartition

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform BG test 
BG(X1, X2)

---

Biswas and Ghosh (2014) Two-Sample Test

Description

Performs the Biswas and Ghosh (2014) two-sample test for high-dimensional data.

Usage

BG2(X1, X2, n.perm = 0, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
seed	Random seed (default: 42)

Details

The test is based on comparing the means of the distributions of the within-sample and between-sample distances of both samples. It is intended for the high dimension low sample size (HDLSS) setting and claimed to perform better in this setting than the tests of Friedman and Rafsky (1979), Schilling (1986) and Henze (1988) and the Cramér test of Baringhaus and Franz (2004).

The statistic is defined as

$T = ||\hat{\mu}_{D_F} - \hat{\mu}_{D_G}||^2_2, \text{ where}$

$\hat{\mu}_{D_F} = \left[\hat{\mu}_{FF} = \frac{2}{n_1(n_1 - 1)}\sum_{i=1}^{n_1}\sum_{j=i+1}^{n_1}||X_{1i} - X_{1j}||, \hat{\mu}_{FG} = \frac{1}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2}||X_{1i} - X_{2j}||\right],$

$\hat{\mu}_{D_G} = \left[\hat{\mu}_{FG} = \frac{1}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2}||X_{1i} - X_{2j}||, \hat{\mu}_{GG} = \frac{2}{n_2(n_2 - 1)}\sum_{i=1}^{n_2}\sum_{j=i+1}^{n_2}||X_{2i} - X_{2j}||\right].$

For testing, the scaled statistic

$T^* = \frac{N\hat{\lambda}(1 - \hat{\lambda})}{2\hat{\sigma}_0^2} T \text{ with}$

$\hat{\lambda} = \frac{n_1}{N},$

$\hat{\sigma}_0^2 = \frac{n_1S_1 + n_2S_2}{N}, \text{ where}$

$S_1 = \frac{1}{\binom{n_1}{3}} \sum_{1\le i < j < k \le n_1} ||X_{1i} - X_{1j}||\cdot ||X_{1i} - X_{1k}|| - \hat{\mu}_{FF}^2 \text{ and}$

$S_2 = \frac{1}{\binom{n_2}{3}} \sum_{1\le i < j < k \le n_2} ||X_{2i} - X_{2j}||\cdot ||X_{2i} - X_{2k}|| - \hat{\mu}_{GG}^2$

is used as it is asymptotically $\chi^2_1$-distributed.

In both cases, low values indicate similarity of the datasets. Thus, the test rejects the null hypothesis of equal distributions for large values of the test statistic.

For n.perm > 0, a permutation test is conducted. Otherwise, an asymptotic test using the asymptotic distibution of $T^*$ is performed.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Biswas, M., Ghosh, A.K. (2014). A nonparametric two-sample test applicable to high dimensional data. Journal of Multivariate Analysis, 123, 160-171, doi:10.1016/j.jmva.2013.09.004.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Energy, Cramer

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Biswas and Ghosh test
BG2(X1, X2)

---

Biswas et al. (2014) two-sample run test

Description

The function implements the Biswas, Mukhopadhyay and Gosh (2014) distribution-free two-sample run test. This test uses a heuristic approach to calculate the shortest Hamilton path between the two datasets using the HamiltonPath function. By default the asymptotic version of the test is calculated.

Usage

BMG(X1, X2, seed = 42, asymptotic = TRUE)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
seed	Random seed (default: 42)
asymptotic	Should the asymptotic version of the test be performed (default: TRUE)

Details

The test counts the number of edges in the shortest Hamilton path calculated on the pooled sample that connect points from different samples, i.e.

$T_{m,n} = 1 + \sum_{i = 1}^{N-1} U_i,$

where $U_i$ is an indicator function with $U_i = 1$ if the $i$th edge connects points from different samples and $U_i = 0$ otherwise.

For a combined sample size N smaller or equal to 1030, the exact version of the Biswas, Mukhopadhyay and Gosh (2014) test can be calculated. It uses the univariate run statistic (Wald and Wolfowitz, 1940) to calculate the test statistic. For N larger than 1030, the calculation for the exact version breaks.

If an asymptotic test is performed the asymptotic null distribution is given by

$T_{m, n}^{*} \sim \mathcal{N}(0, 4\lambda^2(1-\lambda)^2)$

where $T_{m, n}^{*}= \sqrt{N} (T_{m, n} / N - 2 \lambda (1 - \lambda))$ the asymptotic test statistic, $\lambda = m/N$ and $m$ is the sample size of the first dataset. Therefore, low absolute values of the asymptotic test statistic indicate similarity of the two datasets whereas high absolute values indicate differences between the datasets.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic (note: this is not the asymptotic test statistic)
p.value	(asymptotic) p value
method	Description of the test
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Biswas, M., Mukhopadhyay, M. and Ghosh, A. K. (2014). A distribution-free two-sample run test applicable to high-dimensional data, Biometrika 101 (4), 913-926, doi:10.1093/biomet/asu045

Wald, A. and Wolfowitz, J. (1940). On a test whether two samples are from the same distribution, Annals of Mathematical Statistic 11, 147-162

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

HamiltonPath

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform BMG test 
BMG(X1, X2)

---

Barakat et al. (1996) Two-Sample Test

Description

Performs the nearest-neighbor-based multivariate two-sample test of Barakat et al. (1996).

Usage

BQS(X1, X2, dist.fun = stats::dist, n.perm = 0, dist.args = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset (default: stats::dist, Euclidean distance).
n.perm	Number of permutations for permutation test (default: 0, no test is performed).
dist.args	Named list of further arguments passed to dist.fun (default: NULL).
seed	Random seed (default: 42)

Details

The test is an extension of the Schilling (1986) and Henze (1988) neighbor test that bypasses choosing the number of nearest neighbors to consider. The Schilling-Henze test statistic is the proportion of edges connecting points from the same dataset in a K-nearest neighbor graph calculated on the pooled sample (standardized with expectation and SD under the null). Barakat et al. (1996) take the weighted sum of the Schilling-Henze test statistics for $K = 1,\dots,N-1$, where $N$ denotes the pooled sample size.

As for the Schilling-Henze test, low values of the test statistic indicate similarity of the datasets. Thus, the null hypothesis of equal distributions is rejected for high values. A permutation test is performed if n.perm is set to a positive number.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Permutation p value (if n.perm > 0)
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Barakat, A.S., Quade, D. and Salama, I.A. (1996), Multivariate Homogeneity Testing Using an Extended Concept of Nearest Neighbors. Biom. J., 38: 605-612. doi:10.1002/bimj.4710380509

Schilling, M. F. (1986). Multivariate Two-Sample Tests Based on Nearest Neighbors. Journal of the American Statistical Association, 81(395), 799-806. doi:10.2307/2289012

Henze, N. (1988). A Multivariate Two-Sample Test Based on the Number of Nearest Neighbor Type Coincidences. The Annals of Statistics, 16(2), 772-783.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

SH, FR, CF, CCS, ZC for other graph-based tests, FR_cat, CF_cat, CCS_cat, and ZC_cat for versions of the test for categorical data

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Barakat et al. test
BQS(X1, X2)

---

Classifier Two-Sample Test

Description

The function implements the Classifier Two-Sample Test (C2ST) of Lopez-Paz & Oquab (2017). The comparison of multiple ($\ge 2$) samples is also possible. The implementation here uses the classifier_test implementation from the Ecume package.

Usage

C2ST(X1, X2, ..., split = 0.7, thresh = 0, method = "knn", control = NULL, 
      train.args = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
...	Optionally more datasets as matrices or data.frames
split	Proportion of observations used for training
thresh	Value to add to the null hypothesis value (default:0). The null hypothesis tested can be formulated as $H_0: t = p_0 +$ thresh, where $t$ denotes the test accuracy of the classifier and $p_0$ is the chance level (proportion of largest dataset in pooled sample).
method	Classifier to use during training (default: "knn"). See details for possible options.
control	Control parameters for fitting. See trainControl. Defaults to NULL in which case it is set to caret::trainControl(method = "cv").
train.args	Further arguments passed to train as a named list.
seed	Random seed (default: 42)

Details

The classifier two-sample test works by first combining the datasets into a pooled dataset and creating a target variable with the dataset membership of each observation. The pooled sample is then split into training and test set and a classifier is trained on the training data. The classification accuracy on the test data is then used as a test statistic. If the distributions of the datasets do not differ, the classifier will be unable to distinguish between the datasets and therefore the test accuracy will be close to chance level. The test rejects if the test accuracy is greater than chance level.

All methods available for classification within the caret framework can be used as methods. A list of possible models can for example be retrieved via

names(caret::getModelInfo())[sapply(caret::getModelInfo(), function(x) "Classification" %in% x$type)]

This implementation is a wrapper function around the function classifier_test that modifies the in- and output of that function to match the other functions provided in this package. For more details see the classifier_test.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Asymptotic p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names
classifier	Chosen classification method

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	Yes	Yes

References

Lopez-Paz, D., and Oquab, M. (2022). Revisiting classifier two-sample tests. ICLR 2017. https://openreview.net/forum?id=SJkXfE5xx.

Roux de Bezieux, H. (2021). Ecume: Equality of 2 (or k) Continuous Univariate and Multivariate Distributions. R package version 0.9.1, https://CRAN.R-project.org/package=Ecume.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

HMN, YMRZL

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform classifier two-sample test 
if(requireNamespace("Ecume", quietly = TRUE)) {
  C2ST(X1, X2)
}

---

Weighted Edge-Count Two-Sample Test

Description

Performs the weighted edge-count two-sample test for multivariate data proposed by Chen, Chen and Su (2018). The test is intended for comparing two samples with unequal sample sizes. The implementation here uses the g.tests implementation from the gTests package.

Usage

CCS(X1, X2, dist.fun = stats::dist, graph.fun = MST, n.perm = 0, 
    dist.args = NULL, graph.args = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset (default: stats::dist, Euclidean distance).
graph.fun	Function for calculating a similarity graph using the distance matrix on the pooled sample (default: MST, Minimum Spanning Tree).
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
dist.args	Named list of further arguments passed to dist.fun (default: NULL).
graph.args	Named list of further arguments passed to graph.fun (default: NULL).
seed	Random seed (default: 42)

Details

The test is an enhancement of the Friedman-Rafsky test (original edge-count test) that aims at improving the test's power for unequal sample sizes by weighting. The test statistic is given as

$Z_w = \frac{R_w - \text{E}_{H_0}(R_w)}{\sqrt{\text{Var}_{H_0}(R_w)}}, \text{ where}$

$R_w = \frac{n_1}{n_1+n_2} R_1 + \frac{n_2}{n_1+n_2} R_2$

and $R_1$ and $R_2$ denote the number of edges in the similarity graph connecting points within the first and second sample $X_1$ and $X_2$, respectively.

High values of the test statistic indicate dissimilarity of the datasets as the number of edges connecting points within the same sample is high meaning that points are more similar within the datasets than between the datasets.

For n.perm = 0, an asymptotic test using the asymptotic normal approximation of the null distribution is performed. For n.perm > 0, a permutation test is performed.

This implementation is a wrapper function around the function g.tests that modifies the in- and output of that function to match the other functions provided in this package. For more details see the g.tests.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Chen, H., Chen, X. and Su, Y. (2018). A Weighted Edge-Count Two-Sample Test for Multivariate and Object Data. Journal of the American Statistical Association, 113(523), 1146-1155, doi:10.1080/01621459.2017.1307757

Chen, H., and Zhang, J. (2017). gTests: Graph-Based Two-Sample Tests. R package version 0.2, https://CRAN.R-project.org/package=gTests.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

FR for the original edge-count test, CF for the generalized edge-count test, ZC for the maxtype edge-count test, gTests for performing all these edge-count tests at once, SH for performing the Schilling-Henze nearest neighbor test, CCS_cat, FR_cat, CF_cat, ZC_cat, and gTests_cat for versions of the test for categorical data

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform weighted edge-count test
if(requireNamespace("gTests", quietly = TRUE)) {
  CCS(X1, X2)
}

---

Weighted Edge-Count Two-Sample Test for Discrete Data

Description

Performs the weighted edge-count two-sample test for multivariate data proposed by Chen, Chen and Su (2018). The test is intended for comparing two samples with unequal sample sizes. The implementation here uses the g.tests implementation from the gTests package.

Usage

CCS_cat(X1, X2, dist.fun, agg.type, graph.type = "mstree", K = 1, n.perm = 0, 
        seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset.
agg.type	Character giving the method for aggregating over possible similarity graphs. Options are "u" for union of possible similarity graphs and "a" for averaging over test statistics calculated on possible similarity graphs.
graph.type	Character specifying which similarity graph to use. Possible options are "mstree" (default, Minimum Spanning Tree) and "nnlink" (Nearest Neighbor Graph).
K	Parameter for graph (default: 1). If graph.type = "mstree", a K-MST is constructed (K=1 is the classical MST). If graph.type = "nnlink", K gives the number of neighbors considered in the K-NN graph.
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
seed	Random seed (default: 42)

Details

The test is an enhancement of the Friedman-Rafsky test (original edge-count test) that aims at improving the test's power for unequal sample sizes by weighting. The test statistic is given as

$Z_w = \frac{R_w - \text{E}_{H_0}(R_w)}{\sqrt{\text{Var}_{H_0}(R_w)}}, \text{ where}$

$R_w = \frac{n_1}{n_1+n_2} R_1 + \frac{n_2}{n_1+n_2} R_2$

and $R_1$ and $R_2$ denote the number of edges in the similarity graph connecting points within the first and second sample $X_1$ and $X_2$, respectively. For discrete data, the similarity graph used in the test is not necessarily unique. This can be solved by either taking a union of all optimal similarity graphs or averaging the test statistics over all optimal similarity graphs. For details, see Zhang and Chen (2022).

For n.perm = 0, an asymptotic test using the asymptotic normal approximation of the null distribution is performed. For n.perm > 0, a permutation test is performed.

This implementation is a wrapper function around the function g.tests that modifies the in- and output of that function to match the other functions provided in this package. For more details see the g.tests.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	No	Yes	No

References

Chen, H., Chen, X. and Su, Y. (2018). A Weighted Edge-Count Two-Sample Test for Multivariate and Object Data. Journal of the American Statistical Association, 113(523), 1146 - 1155, doi:10.1080/01621459.2017.1307757

Zhang, J. and Chen, H. (2022). Graph-Based Two-Sample Tests for Data with Repeated Observations. Statistica Sinica 32, 391-415, doi:10.5705/ss.202019.0116.

Chen, H., and Zhang, J. (2017). gTests: Graph-Based Two-Sample Tests. R package version 0.2, https://CRAN.R-project.org/package=gTests.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

FR_cat for the original edge-count test, CF_cat for the generalized edge-count test, ZC_cat for the maxtype edge-count test, gTests_cat for performing all these edge-count tests at once, CCS, FR, CF, ZC, and gTests for versions of the tests for continuous data, and SH for performing the Schilling-Henze nearest neighbor test

Examples

# Draw some data
X1cat <- matrix(sample(1:4, 300, replace = TRUE), ncol = 3)
X2cat <- matrix(sample(1:4, 300, replace = TRUE, prob = 1:4), ncol = 3)
# Perform weighted edge-count test
if(requireNamespace("gTests", quietly = TRUE)) {
  CCS_cat(X1cat, X2cat, dist.fun = function(x, y) sum(x != y), agg.type = "a")
}

---

Generalized Edge-Count Test

Description

Performs the generalized edge-count two-sample test for multivariate data proposed by Chen and Friedman (2017). The implementation here uses the g.tests implementation from the gTests package.

Usage

CF(X1, X2, dist.fun = stats::dist, graph.fun = MST, n.perm = 0, 
    dist.args = NULL, graph.args = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset (default: stats::dist, Euclidean distance).
graph.fun	Function for calculating a similarity graph using the distance matrix on the pooled sample (default: MST, Minimum Spanning Tree).
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
dist.args	Named list of further arguments passed to dist.fun (default: NULL).
graph.args	Named list of further arguments passed to graph.fun (default: NULL).
seed	Random seed (default: 42)

Details

The test is an enhancement of the Friedman-Rafsky test (original edge-count test) that aims at detecting both location and scale alternatives. The test statistic is given as

$S = (R_1 - \mu_1, R_2 - \mu_2)\Sigma^{-1} \binom{R_1 - \mu_1}{R_2 - \mu_2}, \text{ where}$

$R_1$ and $R_2$ denote the number of edges in the similarity graph connecting points within the first and second sample $X_1$ and $X_2$, respectively, $\mu_1 = \text{E}_{H_0}(R_1)$, $\mu_2 = \text{E}_{H_0}(R_2)$ and $\Sigma$ is the covariance matrix of $R_1$ and $R_2$ under the null.

High values of the test statistic indicate dissimilarity of the datasets as the number of edges connecting points within the same sample is high meaning that points are more similar within the datasets than between the datasets.

For n.perm = 0, an asymptotic test using the asymptotic $\chi^2$ approximation of the null distribution is performed. For n.perm > 0, a permutation test is performed.

This implementation is a wrapper function around the function g.tests that modifies the in- and output of that function to match the other functions provided in this package. For more details see the g.tests.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
parameter	Degrees of freedom for $\chi^2$ distribution under $H_0$ (only for asymptotic test)
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Chen, H. and Friedman, J.H. (2017). A New Graph-Based Two-Sample Test for Multivariate and Object Data. Journal of the American Statistical Association, 112(517), 397-409. doi:10.1080/01621459.2016.1147356

Chen, H., and Zhang, J. (2017). gTests: Graph-Based Two-Sample Tests. R package version 0.2, https://CRAN.R-project.org/package=gTests.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

FR for the original edge-count test, CCS for the weighted edge-count test, ZC for the maxtype edge-count test, gTests for performing all these edge-count tests at once, SH for performing the Schilling-Henze nearest neighbor test, CCS_cat, FR_cat, CF_cat, ZC_cat, and gTests_cat for versions of the test for categorical data

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform generalized edge-count test
if(requireNamespace("gTests", quietly = TRUE)) {
  CF(X1, X2)
}

---

Generalized Edge-Count Test for Discrete Data

Description

Performs the generalized edge-count two-sample test for multivariate data proposed by Chen and Friedman (2017). The implementation here uses the g.tests implementation from the gTests package.

Usage

CF_cat(X1, X2, dist.fun, agg.type, graph.type = "mstree", K = 1, n.perm = 0, 
        seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset.
agg.type	Character giving the method for aggregating over possible similarity graphs. Options are "u" for union of possible similarity graphs and "a" for averaging over test statistics calculated on possible similarity graphs.
graph.type	Character specifying which similarity graph to use. Possible options are "mstree" (default, Minimum Spanning Tree) and "nnlink" (Nearest Neighbor Graph).
K	Parameter for graph (default: 1). If graph.type = "mstree", a K-MST is constructed (K=1 is the classical MST). If graph.type = "nnlink", K gives the number of neighbors considered in the K-NN graph.
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
seed	Random seed (default: 42)

Details

The test is an enhancement of the Friedman-Rafsky test (original edge-count test) that aims at detecting both location and scale alternatives. The test statistic is given as

$S = (R_1 - \mu_1, R_2 - \mu_2)\Sigma^{-1} \binom{R_1 - \mu_1}{R_2 - \mu_2}, \text{ where}$

$R_1$ and $R_2$ denote the number of edges in the similarity graph connecting points within the first and second sample $X_1$ and $X_2$, respectively, $\mu_1 = \text{E}_{H_0}(R_1)$, $\mu_2 = \text{E}_{H_0}(R_2)$ and $\Sigma$ is the covariance matrix of $R_1$ and $R_2$ under the null.

For discrete data, the similarity graph used in the test is not necessarily unique. This can be solved by either taking a union of all optimal similarity graphs or averaging the test statistics over all optimal similarity graphs. For details, see Zhang and Chen (2022).

High values of the test statistic indicate dissimilarity of the datasets as the number of edges connecting points within the same sample is high meaning that points are more similar within the datasets than between the datasets.

For n.perm = 0, an asymptotic test using the asymptotic normal approximation of the null distribution is performed. For n.perm > 0, a permutation test is performed.

This implementation is a wrapper function around the function g.tests that modifies the in- and output of that function to match the other functions provided in this package. For more details see the g.tests.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
parameter	Degrees of freedom for $\chi^2$ distribution under $H_0$ (only for asymptotic test)
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	No	Yes	No

References

Chen, H. and Friedman, J.H. (2017). A New Graph-Based Two-Sample Test for Multivariate and Object Data. Journal of the American Statistical Association, 112(517), 397-409. doi:10.1080/01621459.2016.1147356

Zhang, J. and Chen, H. (2022). Graph-Based Two-Sample Tests for Data with Repeated Observations. Statistica Sinica 32, 391-415, doi:10.5705/ss.202019.0116.

Chen, H., and Zhang, J. (2017). gTests: Graph-Based Two-Sample Tests. R package version 0.2, https://CRAN.R-project.org/package=gTests.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

FR_cat for the original edge-count test, CCS_cat for the weighted edge-count test, ZC_cat for the maxtype edge-count test, gTests_cat for performing all these edge-count tests at once, CCS, FR, CF, ZC, and gTests for versions of the tests for continuous data, and SH for performing the Schilling-Henze nearest neighbor test

Examples

# Draw some data
X1cat <- matrix(sample(1:4, 300, replace = TRUE), ncol = 3)
X2cat <- matrix(sample(1:4, 300, replace = TRUE, prob = 1:4), ncol = 3)
# Perform generalized edge-count test
if(requireNamespace("gTests", quietly = TRUE)) {
  CF_cat(X1cat, X2cat, dist.fun = function(x, y) sum(x != y), agg.type = "a")
}

---

Constrained Minimum Distance

Description

Calculates the Constrained Minimum Distance (Tatti, 2007) between two datasets.

Usage

CMDistance(X1, X2, binary = NULL, cov = FALSE,
            S.fun = function(x) as.numeric(as.character(x)), 
            cov.S = NULL, Omega = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
binary	Should the simplified form for binary data be used? (default: NULL, it is checked internally if each variable in the pooled dataset takes on exactly two distinct values)
cov	If the the binary version is used, should covariances in addition to means be used as features? (default: FALSE, corresponds to example 3 in Tatti (2007), TRUE corresponds to example 4). Ignored if binary = FALSE.
S.fun	Feature function (default: NULL). Should be supplied as a function that takes one observation vector as its input. Ignored if binary = TRUE (default: NULL).
cov.S	Covariance matix of feature function (default: NULL). Ignored if binary = TRUE.
Omega	Sample space as matrix (default: NULL, the sample space is derived from the data internally). Each row represents one value in the sample space. Used for calculating the covariance matrix if cov.S = NULL. Either cov.S or Omega must be given. Ignored if binary = TRUE.
seed	Random seed (default: 42)

Details

The constrained minimum (CM) distance is not a distance between distributions but rather a distance based on summaries. These summaries, called frequencies and denoted by $\theta$, are averages of feature functions $S$ taken over the dataset. The constrained minimum distance of two datasets $X_1$ and $X_2$ can be calculated as

$d_{CM}(X_1, X_2 |S)^2 = (\theta_1 - \theta_2)^T\text{Cov}^{-1}(S)(\theta_1 - \theta_2),$

where $\theta_i = S(X_i)$ is the frequency with respect to the $i$-th dataset, $i = 1, 2$, and

$\text{Cov}(S) = \frac{1}{|\Omega|}\sum_{\omega\in\Omega} S(\omega)S(\omega)^T - \left(\frac{1}{|\Omega|}\sum_{\omega\in\Omega} S(\omega)\right)\left(\frac{1}{|\Omega|}\sum_{\omega\in\Omega} S(\omega)\right)^T,$

where $\Omega$ denotes the sample space.

Note that the implementation can only handle limited dimensions of the sample space. The error message

"Error in rep.int(rep.int(seq_len(nx), rep.int(rep.fac, nx)), orep) : invalid 'times' value"

occurs when the sample space becomes too large to enumerate all its elements. In case of binary data and $S$ chosen as a conjunction or parity function $T_{F}$ on a family of itemsets, the calculation of the CMD simplifies to

$d_{CM}(D_1, D_2 | S_{F}) = 2 ||\theta_1 - \theta_2||_2,$

where $\theta_i = T_{F}(X_i), i = 1, 2,$ as the sample space and covariance matrix are known. In case of more than two categories, either the sample space or the covariance matrix of the feature function must be supplied.

Small values of the CM Distance indicate similarity between the datasets. No test is conducted.

Value

An object of class htest with the following components:

statistic	Observed value of the CM Distance
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names
binary, cov, S.fun, cov.S, Omega	Input parameters

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	No	Yes	No

Note

Note that there is an error in the calculation of the covariance matrix in A.4 Proof of Lemma 8 in Tatti (2007). The correct covariance matrix has the form

$\text{Cov}[T_{\mathcal{F}}] = 0.25I$

since

$\text{Var}[T_A] = \text{E}[T_A^2] - \text{E}[T_A]^2 = 0.5 - 0.5^2 = 0.25$

following from the correct statement that $\text{E}[T_A^2] = \text{E}[T_A] = 0.5$. Therefore, formula (4) changes to

$d_{CM}(D_1, D_2 | S_{\mathcal{F}}) = 2 ||\theta_1 - \theta_2||_2$

and the formula in example 3 changes to

$d_{CM}(D_1, D_2 | S_{I}) = 2 ||\theta_1 - \theta_2||_2.$

Our implementation is based on these corrected formulas. If the original formula was used, the results on the same data calculated with the formula for the binary special case and the results calculated with the general formula differ by a factor of $\sqrt{2}$.

References

Tatti, N. (2007). Distances between Data Sets Based on Summary Statistics. JMRL 8, 131-154.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

Examples

# Test example 2 in Tatti (2007)
CMDistance(X1 = data.frame(c("C", "C", "C", "A")), 
           X2 = data.frame(c("C", "A", "B", "A")),
           binary = FALSE, S.fun = function(x) as.numeric(x == "C"),
           Omega = data.frame(c("A", "B", "C")))

# Demonstration of corrected calculation
X1bin <- matrix(sample(0:1, 100 * 3, replace = TRUE), ncol = 3)
X2bin <- matrix(sample(0:1, 100 * 3, replace = TRUE, prob = 1:2), ncol = 3)
CMDistance(X1bin, X2bin, binary = TRUE, cov = FALSE)
Omega <- expand.grid(0:1, 0:1, 0:1)
S.fun <- function(x) x
CMDistance(X1bin, X2bin, binary = FALSE, S.fun = S.fun, Omega = Omega)
CMDistance(X1bin, X2bin, binary = FALSE, S.fun = S.fun, cov.S = 0.5 * diag(3))
CMDistance(X1bin, X2bin, binary = FALSE, S.fun = S.fun, 
            cov.S = 0.5 * diag(3))$statistic * sqrt(2)

# Example for non-binary data
X1cat <- matrix(sample(1:4, 300, replace = TRUE), ncol = 3)
X2cat <- matrix(sample(1:4, 300, replace = TRUE, prob = 1:4), ncol = 3)
CMDistance(X1cat, X2cat, binary = FALSE, S.fun = S.fun, 
           Omega = expand.grid(1:4, 1:4, 1:4))
CMDistance(X1cat, X2cat, binary = FALSE, S.fun = function(x) as.numeric(x == 1), 
           Omega = expand.grid(1:4, 1:4, 1:4))
CMDistance(X1cat, X2cat, binary = FALSE, S.fun = function(x){ 
           c(x, x[1] * x[2], x[1] * x[3], x[2] * x[3])}, 
           Omega = expand.grid(1:4, 1:4, 1:4))

---

Cramér Two-Sample Test

Description

Performs Two-Sample Cramér Test (Baringhaus and Franz, 2004). The implementation here uses the cramer.test implementation from the cramer package.

Usage

Cramer(X1, X2, n.perm = 0, just.statistic = (n.perm <= 0), sim = "ordinary", 
        maxM = 2^14, K = 160, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permutation or Bootstrap test, respectively (default: 0, no permutation test performed)
just.statistic	Should only the test statistic be calculated without performing any test (default: TRUE if number of permutations is set to 0 and FALSE if number of permutations is set to any positive number)
sim	Type of Bootstrap or eigenvalue method for testing. Possible options are "ordinary" (default) for ordinary Boostrap, "permutation" for permutation testing, or "eigenvalue" for bootstrapping the limit distribution (especially good for datasets too large for performing Bootstrapping). For more details see cramer.test
maxM	Maximum number of points used for fast Fourier transform involved in eigenvalue method for approximating the null distribution (default: $2^14$). Ignored if sim is either "ordinary" or "permutation". For more details see cramer::cramer.test.
K	Upper value up to which the integral for calculating the distribution function from the characteristic function is evaluated (default: 160). Note: when K is increased, it is necessary to also increase maxM. Ignored if sim is either "ordinary" or "permutation". For more details see cramer.test.
seed	Random seed (default: 42)

Details

The Cramér test (Baringhaus and Franz, 2004) is a specialcase of the test of Bahrinhaus and Franz (2010) where the kernel function $\phi$ is set to

$\phi_{\text{Cramer}}(x) = \sqrt{x} / 2$

and can be recommended for location alternatives. The test statistic simplifies to

$T_{n_1, n_2} = \frac{n_1 n_2}{n_1+n_2}\left(\frac{1}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2} ||X_{1i} - X_{2j}|| - \frac{1}{2n_1^2}\sum_{i,j=1}^{n_1} ||X_{1i} - X_{1j}|| - \frac{1}{2n_2^2}\sum_{i,j=1}^{n_2} ||X_{2i} - X_{2j}||\right).$

This is equal to the Energy statistic (Székely and Rizzo, 2004).

The theoretical statistic underlying this test statistic is zero if and only if the distributions coincide. Therefore, low values of the test statistic incidate similarity of the datasets while high values indicate differences between the datasets.

This implementation is a wrapper function around the function cramer.test that modifies the in- and output of that function to match the other functions provided in this package. For more details see the cramer.test.

Value

An object of class htest with the following components:

method	Description of the test
d	Number of variables in each dataset
m	Sample size of first dataset
n	Sample size of second dataset
statistic	Observed value of the test statistic
p.value	Boostrap/ permutation p value (only if n.perm > 0)
sim	Type of Boostrap or eigenvalue method (only if n.perm > 0)
n.perm	Number of permutations for permutation or Boostrap test
hypdist	Distribution function under the null hypothesis reconstructed via fast Fourier transform. $x contains the x-values, $Fx contains the corresponding distribution function values. (only if n.perm > 0)
ev	Eigenvalues and eigenfunctions when using the eigenvalue method (only if n.perm > 0)
data.name	The dataset names
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

Note

The Cramér test (Baringhaus and Franz, 2004) is equivalent to the test based on the Energy statistic (Székely and Rizzo, 2004).

References

Baringhaus, L. and Franz, C. (2010). Rigid motion invariant two-sample tests, Statistica Sinica 20, 1333-1361

Bahr, R. (1996). Ein neuer Test fuer das mehrdimensionale Zwei-Stichproben-Problem bei allgemeiner Alternative, German, Ph.D. thesis, University of Hanover

Franz, C. (2024). cramer: Multivariate Nonparametric Cramer-Test for the Two-Sample-Problem. R package version 0.9-4, https://CRAN.R-project.org/package=cramer.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Energy, Bahr, BF

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Cramer test 
if(requireNamespace("cramer", quietly = TRUE)) {
  Cramer(X1, X2, n.perm = 100)
}

---

Direction-Projection Functions for DiProPerm Test

Description

Helper functions performing the direction and projection step using different classifiers for the Direction-Projection-Permutation (DiProPerm) two-sample test for high-dimensional data (Wei et al., 2016)

Usage

dwdProj(X1, X2)
svmProj(X1, X2)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame

Details

The DiProPerm test works by first combining the datasets into a pooled dataset and creating a target variable with the dataset membership of each observation. A binary linear classifier is then trained on the class labels and the normal vector of the separating hyperplane is calculated. The data from both samples is projected onto this normal vector. This gives a scalar score for each observation. On these projection scores, a univariate two-sample statistic is calculated. The permutation null distribution of this statistic is calculated by permuting the dataset labels and repeating the whole procedure with the permuted labels. The functions here correspond to the direction and projection step for either the DWD or SVM classifier as proposed by Wei et al., 2016.

The DWD model implementation genDWD in the DWDLargeR package is used with the penalty parameter C calculated with penaltyParameter using the recommended default values. More details on the algorithm can be found in Lam et al. (2018).

For the SVM, the implementation svm in the e1071 package is used with default parameters.

Value

A numeric vector containing the projected values for each observation in the pooled sample

References

Lam, X. Y., Marron, J. S., Sun, D., & Toh, K.-C. (2018). Fast Algorithms for Large-Scale Generalized Distance Weighted Discrimination. Journal of Computational and Graphical Statistics, 27(2), 368-379. doi:10.1080/10618600.2017.1366915

Wei, S., Lee, C., Wichers, L., & Marron, J. S. (2016). Direction-Projection-Permutation for High-Dimensional Hypothesis Tests. Journal of Computational and Graphical Statistics, 25(2), 549-569. doi:10.1080/10618600.2015.1027773

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

DiProPerm

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)

# calculate projections separately (only for demonstration)

dwdProj(X1, X2)

svmProj(X1, X2)

# Use within DiProPerm test 
# Note: For real applications, n.perm should be set considerably higher
# No permutations chosen for demonstration due to runtime

if(requireNamespace("DWDLargeR", quietly = TRUE)) {
  DiProPerm(X1, X2, n.perm = 10, dipro.fun = dwdProj)
}
if(requireNamespace("e1071", quietly = TRUE)) {
  DiProPerm(X1, X2, n.perm = 10, dipro.fun = svmProj)
}

---

Direction-Projection-Permutation (DiProPerm) Test

Description

Performs the Direction-Projection-Permutation (DiProPerm) two-sample test for high-dimensional data (Wei et al., 2016).

Usage

DiProPerm(X1, X2, n.perm = 0, dipro.fun = dwdProj, stat.fun = MD, 
            direction = "two.sided", seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permutation test (default: 0, no permutation test performed)
dipro.fun	Function performing the direction and projection step using a linear classifier. Implemented options are dwdProj (default, distance weighted discrimination, DWD), and svmProj (support vector machine). Must take the two datasets as input and output the calculated scores for the pooled sample.
stat.fun	Function that calculates a univariate two-sample statistic from two vectors. Implemented options are MD (default, mean difference, recommended for detecting mean differendes), tStat (t test statistic) and AUC (area under the receiver operating curve). Must take the two numeric vectors as input and output the two sample statistic as a numeric scalar.
direction	Character indicating for which values of the univariate test statistic the test should reject the null hypothesis. Possible options are "two.sided" (reject both for low and high values, appropriate for MD and tStat), "greater" (reject for high values, appropriate for AUC), or "smaller" (reject for low values).
seed	Random seed (default: 42)

Details

The DiProPerm test works by first combining the datasets into a pooled dataset and creating a target variable with the dataset membership of each observation. A binary linear classifier is then trained on the class labels and the normal vector of the separating hyperplane is calculated. The data from both samples is projected onto this normal vector. This gives a scalar score for each observation. On these projection scores, a univariate two-sample statistic is calculated. The permutation null distribution of this statistic is calculated by permuting the dataset labels and repeating the whole procedure with the permuted labels.

At the moment, distance weighted discrimination (DWD), and support vector machine (SVM) are implemented as binary linear classifiers.

The DWD model implementation genDWD in the DWDLargeR package is used with the penalty parameter C calculated with penaltyParameter using the recommended default values. More details on the algorithm can be found in Lam et al. (2018).

For the SVM, the implementation svm in the e1071 package is used with default parameters.

Other classifiers can be used by supplying a suitable function for dipro.fun.

For the univariate test statistic, implemented options are the mean difference, t statistic and AUC. Other suitable statistics can be used by supplying a suitable function of stat.fun.

Whether high or low values of the test statistic correspond to similarity of the datasets depends on the chosen univariate statistic. This is reflected by the direction argument which modifies the behavior of the test to reject the null for appropriate values.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Lam, X. Y., Marron, J. S., Sun, D., & Toh, K.-C. (2018). Fast Algorithms for Large-Scale Generalized Distance Weighted Discrimination. Journal of Computational and Graphical Statistics, 27(2), 368-379. doi:10.1080/10618600.2017.1366915

Wei, S., Lee, C., Wichers, L., & Marron, J. S. (2016). Direction-Projection-Permutation for High-Dimensional Hypothesis Tests. Journal of Computational and Graphical Statistics, 25(2), 549-569. doi:10.1080/10618600.2015.1027773

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

stat.fun, dipro.fun

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform DiProPerm test 
# Note: For real applications, n.perm should be set considerably higher than 10
# Low values for n.perm chosen for demonstration due to runtime

if(requireNamespace("DWDLargeR", quietly = TRUE)) {
  DiProPerm(X1, X2, n.perm = 10)
  DiProPerm(X1, X2, n.perm = 10, stat.fun = tStat)
  if(requireNamespace("pROC", quietly = TRUE)) {
    DiProPerm(X1, X2, n.perm = 10, stat.fun = AUC, direction = "greater")
  }
}

if(requireNamespace("e1071", quietly = TRUE)) {
  DiProPerm(X1, X2, n.perm = 10, dipro.fun = svmProj)
  DiProPerm(X1, X2, n.perm = 10, dipro.fun = svmProj, stat.fun = tStat)
  if(requireNamespace("pROC", quietly = TRUE)) {
    DiProPerm(X1, X2, n.perm = 10, dipro.fun = svmProj, stat.fun = AUC, direction = "greater")
  }
}

---

Distance Components (DISCO) Tests

Description

Performs Energy statistics distance components (DISCO) multi-sample tests (Rizzo and Székely, 2010). The implementation here uses the disco implementation from the energy package.

Usage

DISCOB(X1, X2, ..., n.perm = 0, alpha = 1, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
...	Further datasets as matrices or data.frames
n.perm	Number of permutations for Bootstrap test (default: 0, no Bootstrap test performed)
alpha	Power of the distance used for generalized Energy statistic (default: 1). Has to lie in $(0,2]$. For values in $(0, 2)$, consistency of the resulting test has been shown.
seed	Random seed (default: 42)

Details

DISCO is a method for multi-sample testing based on all pairwise between-sample distances. It is analogous to the classical ANOVA. Instead of decomposing squared differences from the sample mean, the total dispersion (generalized Energy statistic) is composed into distance components (DISCO) consisting of the within-sample and between-sample measures of dispersion.

DISCOB computes the between-sample DISCO statistic which is the between-sample component.

In both cases, small values of the statistic indicate similarity of the datasets and therefore, the null hypothesis of equal distributions is rejected for large values of the statistic.

This implementation is a wrapper function around the function disco that modifies the in- and output of that function to match the other functions provided in this package. For more details see the disco.

Value

An object of class htest with the following components:

call	The function call
statistic	Observed value of the test statistic
p.value	Bootstrap p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	Yes

References

Szekely, G. J. and Rizzo, M. L. (2004) Testing for Equal Distributions in High Dimension, InterStat, November (5).

Rizzo, M. L. and Szekely, G. J. (2010). DISCO Analysis: A Nonparametric Extension of Analysis of Variance, Annals of Applied Statistics, 4(2), 1034-1055. doi:10.1214/09-AOAS245

Szekely, G. J. (2000) Technical Report 03-05: E-statistics: Energy of Statistical Samples, Department of Mathematics and Statistics, Bowling Green State University.

Rizzo, M., Szekely, G. (2022). energy: E-Statistics: Multivariate Inference via the Energy of Data. R package version 1.7-11, https://CRAN.R-project.org/package=energy.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

DISCOF, Energy

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform DISCO tests
if(requireNamespace("energy", quietly = TRUE)) {
  DISCOB(X1, X2, n.perm = 100)
}

---

Distance Components (DISCO) Tests

Description

Performs Energy statistics distance components (DISCO) multi-sample tests (Rizzo and Székely, 2010). The implementation here uses the disco implementation from the energy package.

Usage

DISCOF(X1, X2, ..., n.perm = 0, alpha = 1, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
...	Further datasets as matrices or data.frames
n.perm	Number of permutations for Bootstrap test (default: 0, no Bootstrap test performed)
alpha	Power of the distance used for generalized Energy statistic (default: 1). Has to lie in $(0,2]$. For values in $(0, 2)$, consistency of the resulting test has been shown.
seed	Random seed (default: 42)

Details

DISCO is a method for multi-sample testing based on all pairwise between-sample distances. It is analogous to the classical ANOVA. Instead of decomposing squared differences from the sample mean, the total dispersion (generalized Energy statistic) is composed into distance components (DISCO) consisting of the within-sample and between-sample measures of dispersion.

DISCOF is based on the DISCO F ratio of the between-sample and within-sample dispersion. Note that the F ration does not follow an F distribution, but is just called F ratio analogous to the ANOVA.

In both cases, small values of the statistic indicate similarity of the datasets and therefore, the null hypothesis of equal distributions is rejected for large values of the statistic.

This implementation is a wrapper function around the function disco that modifies the in- and output of that function to match the other functions provided in this package. For more details see the disco.

Value

An object of class disco with the following components:

call	The function call
method	Description of the test
statistic	Vector of observed values of the test statistic
p.value	Vector of Bootstrap p values
k	Number of samples
N	Number of observations
between	Between-sample distance components
withins	One-way within-sample distance components
within	Within-sample distance component
total	Total dispersion
Df.trt	Degrees of freedom for treatments
Df.e	Degrees of freedom for error
index	Alpha (exponent on distance)
factor.names	Factor names
factor.levels	Factor levels
sample.sizes	Sample sizes
stats	Matrix containing decomposition

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	Yes

References

Szekely, G. J. and Rizzo, M. L. (2004) Testing for Equal Distributions in High Dimension, InterStat, November (5).

Rizzo, M. L. and Szekely, G. J. (2010). DISCO Analysis: A Nonparametric Extension of Analysis of Variance, Annals of Applied Statistics, 4(2), 1034-1055. doi:10.1214/09-AOAS245

Szekely, G. J. (2000) Technical Report 03-05: E-statistics: Energy of Statistical Samples, Department of Mathematics and Statistics, Bowling Green State University.

Rizzo, M., Szekely, G. (2022). energy: E-Statistics: Multivariate Inference via the Energy of Data. R package version 1.7-11, https://CRAN.R-project.org/package=energy.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

DISCOB, Energy

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform DISCO tests
if(requireNamespace("energy", quietly = TRUE)) {
  DISCOF(X1, X2, n.perm = 100)
}

---

Rank-Based Energy Test (Deb and Sen, 2021)

Description

Performs the multivariate rank-based multivariate two-sample test using measure transportation by Deb and Sen (2021).

Usage

DS(X1, X2, n.perm = 0, rand.gen = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
n.perm	Number of permutations for permuation test (default: 0, no permutation test performed)
rand.gen	Function that generates a grid of (random) numbers in $(0,1)$ of dimension $n \times k$ ($n$ and $k$ are inputs of this function). Default is NULL in which case, randtoolbox::halton is used.
seed	Random seed (default: 42)

Details

The test proposed by Deb and Sen (2021) is a rank-based version of the Energy statistic (Székely and Rizzo, 2004) that does not rely on any moment assumptions. Its test statistic is the Energy statistic applied to the rank map of both samples. The multivariate ranks are computed using optimal transport with a multivariate uniform distribution as the reference distribution.

For the rank version of the Energy statistic it still holds that the value zero is attained if and only if the two distributions coincide. Therefore, low values of the empirical test statistic indicate similarity between the datasets and the null hypothesis of equal distributions is rejected for large values.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

Note

The implementation is a modification of the code supplied by Deb and Sen (2021) for the simulation study presented in the original article. It generalizes the implementation and includes small modifications for computation speed.

Author(s)

Original implementation by Nabarun Deb, Bodhisattva Sen

Minor modifications by Marieke Stolte

References

Original implementation: https://github.com/NabarunD/MultiDistFree

Deb, N. and Sen, B. (2021). Multivariate Rank-Based Distribution-Free Nonparametric Testing Using Measure Transportation, Journal of the American Statistical Association. doi:10.1080/01621459.2021.1923508.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Energy

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Deb and Sen test 
if(requireNamespace("randtoolbox", quietly = TRUE) & 
    requireNamespace("clue", quietly = TRUE)) {
  DS(X1, X2, n.perm = 100)
}

---

Energy Statistic and Test

Description

Performs the Energy statistic multi-sample test (Székely and Rizzo, 2004). The implementation here uses the eqdist.etest implementation from the energy package.

Usage

Energy(X1, X2, ..., n.perm = 0, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
...	Further datasets as matrices or data.frames
n.perm	Number of permutations for Bootstrap test (default: 0, no Bootstrap test performed)
seed	Random seed (default: 42)

Details

The Energy statistic (Székely and Rizzo, 2004) for two datasets $X_1$ and $X_2$ is defined as

$T_{n_1, n_2} = \frac{n_1 n_2}{n_1+n_2}\left(\frac{1}{n_1 n_2}\sum_{i=1}^{n_1}\sum_{j=1}^{n_2} ||X_{1i} - X_{2j}|| - \frac{1}{2n_1^2}\sum_{i,j=1}^{n_1} ||X_{1i} - X_{1j}|| - \frac{1}{2n_2^2}\sum_{i,j=1}^{n_2} ||X_{2i} - X_{2j}||\right).$

This is equal to the Cramér test statistitic (Baringhaus and Franz, 2004). The multi-sample version is defined as the sum of the Energy statistics for all pairs of samples.

The population Energy statistic for two distributions is equal to zero if and only if the two distributions coincide. Therefore, small values of the empirical statistic indicate similarity between datasets and the permutation test rejects the null hypothesis of equal distributions for large values.

This implementation is a wrapper function around the function eqdist.etest that modifies the in- and output of that function to match the other functions provided in this package. For more details see the eqdist.etest.

Value

An object of class htest with the following components:

call	The function call
statistic	Observed value of the test statistic
p.value	Bootstrap p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	Yes

Note

The test based on the Energy statistic (Székely and Rizzo, 2004) is equivalent to the Cramér test (Baringhaus and Franz, 2004).

References

Szekely, G. J. and Rizzo, M. L. (2004) Testing for Equal Distributions in High Dimension, InterStat, November (5).

Szekely, G. J. (2000) Technical Report 03-05: E-statistics: Energy of Statistical Samples, Department of Mathematics and Statistics, Bowling Green State University.

Rizzo, M., Szekely, G. (2022). energy: E-Statistics: Multivariate Inference via the Energy of Data. R package version 1.7-11, https://CRAN.R-project.org/package=energy.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Cramer, DISCOB, DISCOF

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Energy test
if(requireNamespace("energy", quietly = TRUE)) {
  Energy(X1, X2, n.perm = 100)
}

---

Engineer Metric

Description

The function implements the $L_q$-engineer metric for comparing two multivariate distributions.

Usage

engineerMetric(X1, X2, type = "F", seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
type	Character specifying the type of $L_q$-norm to use. Reasonable options are "O", "o", "1", for the $L_1$-norm, "I", and "i", for the $L_\infty$-norm, and "F", "f", "E", "e" (the default) for the $L_2$-norm (Euclidean norm).
seed	Random seed (default: 42). Method is deterministic, seed is only set for consistency with other methods.

Details

The engineer is a primary propability metric that is defined as

$\text{EN}(X_1, X_2; q) = \left[ \sum_{i = 1}^{p} \left| \text{E}\left(X_{1i}\right) - \text{E}\left(X_{2i}\right)\right|^q\right]^{\min(q, 1/q)} \text{ with } q> 0,$

where $X_{1i}, X_{2i}$ denote the $i$th component of the $p$-dimensional random vectors $X_1\sim F_1$ and $X_2\sim F_2$.

In the implementation, expectations are estimated by column means of the respective datasets.

Value

An object of class htest with the following components:

method	Description of the test
statistic	Observed value of the test statistic
data.name	The dataset names
method	Description of the test
alternative	The alternative hypothesis

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

Note

The seed argument is only included for consistency with other methods. The result of the metric calculation is deteministic.

References

Rachev, S. T. (1991). Probability metrics and the stability of stochastic models. John Wiley & Sons, Chichester.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

Jeffreys

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Calculate engineer metric
engineerMetric(X1, X2)

---

Friedman-Rafsky Test

Description

Performs the Friedman-Rafsky two-sample test (original edge-count test) for multivariate data (Friedman and Rafsky, 1979). The implementation here uses the g.tests implementation from the gTests package.

Usage

FR(X1, X2, dist.fun = stats::dist, graph.fun = MST, n.perm = 0, 
    dist.args = NULL, graph.args = NULL, seed = 42)

Arguments

X1	First dataset as matrix or data.frame
X2	Second dataset as matrix or data.frame
dist.fun	Function for calculating a distance matrix on the pooled dataset (default: stats::dist, Euclidean distance).
graph.fun	Function for calculating a similarity graph using the distance matrix on the pooled sample (default: MST, Minimum Spanning Tree).
n.perm	Number of permutations for permutation test (default: 0, asymptotic test is performed).
dist.args	Named list of further arguments passed to dist.fun (default: NULL).
graph.args	Named list of further arguments passed to graph.fun (default: NULL).
seed	Random seed (default: 42)

Details

The test is a multivariate extension of the univariate Wald Wolfowitz runs test. The test statistic is the number of edges connecting points from different datasets in a minimum spanning tree calculated on the pooled sample (standardized with expectation and SD under the null).

High values of the test statistic indicate similarity of the datasets. Thus, the null hypothesis of equal distributions is rejected for small values.

For n.perm = 0, an asymptotic test using the asymptotic normal approximation of the null distribution is performed. For n.perm > 0, a permutation test is performed.

This implementation is a wrapper function around the function g.tests that modifies the in- and output of that function to match the other functions provided in this package. For more details see the g.tests.

Value

An object of class htest with the following components:

statistic	Observed value of the test statistic
p.value	Asymptotic or permutation p value
alternative	The alternative hypothesis
method	Description of the test
data.name	The dataset names

Applicability

Target variable?	Numeric?	Categorical?	K-sample?
No	Yes	No	No

References

Friedman, J. H., and Rafsky, L. C. (1979). Multivariate Generalizations of the Wald-Wolfowitz and Smirnov Two-Sample Tests. The Annals of Statistics, 7(4), 697-717.

Chen, H., and Zhang, J. (2017). gTests: Graph-Based Two-Sample Tests. R package version 0.2, https://CRAN.R-project.org/package=gTests.

Stolte, M., Kappenberg, F., Rahnenführer, J., Bommert, A. (2024). Methods for quantifying dataset similarity: a review, taxonomy and comparison. Statist. Surv. 18, 163 - 298. doi:10.1214/24-SS149

See Also

CF for the generalized edge-count test, CCS for the weighted edge-count test, ZC for the maxtype edge-count test, gTests for performing all these edge-count tests at once, SH for performing the Schilling-Henze nearest neighbor test, CCS_cat, FR_cat, CF_cat, ZC_cat, and gTests_cat for versions of the test for categorical data

Examples

# Draw some data
X1 <- matrix(rnorm(1000), ncol = 10)
X2 <- matrix(rnorm(1000, mean = 0.5), ncol = 10)
# Perform Friedman-Rafsky test
if(requireNamespace("gTests", quietly = TRUE)) {
  FR(X1, X2)
}

---