Package 'LiftTest'

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates binomial random samples for the control group (with sample size $n_1$ and success probability $p_1$) and the treatment group (with sample size $n_2$ and success probability $p_2$).

Usage

gen.simu.data(n1, n2, p1, p2, summary=TRUE)

Arguments

n1	sample size of the control group
n2	sample size of the treatment group
p1	success probability of the control group
p2	success probability of the treatment group
summary	boolean variable. if TRUE it returns 2x2 contingency table; if FALSE it returns raw binomial random samples. By default, summary=TRUE.

Details

The a 2x2 contingency table is of the following form

col1	col 2
control sample size	control positive response count
treatment sample size	treatment positive response count

Value

A list of simulated data for the control group and the treatment group if summary=FALSE or a 2x2 contingency table if summary=TRUE

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
set.seed(1)
sim.data <- gen.simu.data(n1, n2, p1, p2)
sim.data

---

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates the asymptotic power of the proposed bootstrap test. Two methods are provided: the asymptotic power based on the relative lift and the asymptotic power the absolute lift. For more details, please refer to the paper Liu et al., (2023).

Usage

get.asymp.power(n1, n2, p1, p2, method='relative', alpha=0.05)

Arguments

n1	sample size of the control group
n2	sample size of the treatment group
p1	success probability of the control group
p2	success probability of the treatment group
method	two methods are provided: method = c($\texttt{`relative'}, \texttt{`absolute'}$). $\texttt{`relative'}$ means min sample size based on the relative lift. $\texttt{`absolute'}$ means min sample size based on the absolute lift.
alpha	significance level. By default alpha = 0.05.

Details

Let $N = n_1 + n_2$ and $\kappa = n_1/N$. We define

$\sigma_{a,n} = \sqrt{n_1^{-1}p_1(1-p_1) + n_2^{-1}p_2(1-p_2)},$

$\bar\sigma_{a,n} = \sqrt{(n_1^{-1} + n_2^{-1})\bar p(1-\bar p)}.$

where $\bar p = \kappa p_1 + (1-\kappa) p_2$. $\sigma_{a,n}$ is the standard deviation of the absolute lift and $\bar\sigma_{a,n}$ can be viewed as the standard deviation of the combined sample of the control and treatment groups. Let $\delta_a = p_2 - p_1$ be the absolute lift. The asymptotic power function based on the absolute lift is given by

$\beta_{Absolute}(\delta_a) \approx \Phi\left( -cz_{\alpha/2} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi\left( -cz_{\alpha/2} - \frac{\delta_a}{\sigma_{a,n}} \right).$

The asymptotic power function based on the relative lift is given by

$\beta_{Relative}(\delta_a) \approx \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} - \frac{\delta_a}{\sigma_{a,n}} \right),$

where $\Phi(\cdot)$ is the CDF of the standard normal distribution $N(0,1)$, $z_{\alpha/2}$ is the upper $(1-\alpha/2)$ quantile of $N(0,1)$, and $c = {\bar\sigma_{a,n}}/\sigma_{a,n}$.

Value

Return the asymptotic power

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
get.asymp.power(n1, n2, p1, p2, method='relative')

---

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates the minimum sample size required to obtain a statistically significant result for a given power. For more details, please refer to the paper Liu et al., (2023).

Usage

get.min.size(p1, p2, p_treat, method='relative', power=0.8, alpha=0.05)

Arguments

p1	success probability of the control group
p2	success probability of the treatment group
p_treat	the percentage of the treatment group
method	two methods are provided: method = c($\texttt{`relative'}$, $\texttt{`absolute'}$). $\texttt{`relative'}$ means min sample size based on the relative lift. $\texttt{`absolute'}$ means min sample size based on the absolute lift.
power	the power you want to achieve. Industry standard is power = 0.8, which is also the default value
alpha	significance level. By default alpha = 0.05

Details

The minimum required sample size is approximated by the asymptotic power function. Let $N = n_1 + n_2$ and $\kappa = n_1/N$. We define

$\sigma_{a,n} = \sqrt{n_1^{-1}p_1(1-p_1) + n_2^{-1}p_2(1-p_2)},$

$\bar\sigma_{a,n} = \sqrt{(n_1^{-1} + n_2^{-1})\bar p(1-\bar p)}.$

where $\bar p = \kappa p_1 + (1-\kappa) p_2$. $\sigma_{a,n}$ is the standard deviation of the absolute lift and $\bar\sigma_{a,n}$ can be viewed as the standard deviation of the combined sample of the control and treatment groups. Let $\delta_a = p_2 - p_1$ be the absolute lift. The asymptotic power function based on the absolute lift is given by

$\beta_{Absolute}(\delta_a) \approx \Phi\left( -cz_{\alpha/2} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi\left( -cz_{\alpha/2} - \frac{\delta_a}{\sigma_{a,n}} \right).$

The asymptotic power function based on the relative lift is given by

$\beta_{Relative}(\delta_a) \approx \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} - \frac{\delta_a}{\sigma_{a,n}} \right),$

where $\Phi(\cdot)$ is the CDF of the standard normal distribution $N(0,1)$, $z_{\alpha/2}$ is the upper $(1-\alpha/2)$ quantile of $N(0,1)$, and $c = {\bar\sigma_{a,n}}/\sigma_{a,n}$.

Given a power (say power=0.80), it is difficult to get a closed form of the minimum sample size. Note that when $\delta_a > 0$, the first term of the power function dominates the second term, so we can ignore the second term and derive the closed form for the minimum sample size. Similarly, when $\delta_a < 0$, the second term of the power function dominates the first term, so we can ignore the first term. In particular, the closed form for the minimum sample size is given by

$N_{Relative} = \left( \frac{p_1(1-p_1)}{\kappa} + \frac{p_2(1-p_2)}{(1-\kappa)} \right) \left( \Phi^{-1}(\beta)p_1/\bar p + cz_{\alpha/2} \right)^2 / \delta_a^2,$

$N_{Absolute} = \left( \frac{p_1(1-p_1)}{\kappa} + \frac{p_2(1-p_2)}{(1-\kappa)} \right) \left( \Phi^{-1}(\beta) + cz_{\alpha/2} \right)^2 / \delta_a^2.$

Value

Return the required minimum sample size. This is the total sample size of control group + treatment group

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Examples

p1 <- 0.1; p2 <- 0.2
get.min.size(p1, p2, p_treat=0.5, method='relative', power=0.8, alpha=0.05)

---

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function implements several proportion tests that can be applied to Brand Lift Testing, including

1. $\mathbf{clt}$: Absolute lift based Z-test and relative lift based Z-test. The limiting distribution of Z-statistics are derived from the central limit theorem.

2. $\mathbf{bootstrap}$: Absolute lift based bootstrap test (BS-A) and relative lift based bootstrap test (BS-R), see Liu et al., (2023).

3. $\mathbf{bootstrapmean}$: Absolute lift based bootstrap mean test and relative lift based bootstrap mean test. (Efron and Tibshirani 1994).

4. $\mathbf{permutation}$: Absolute lift based permutation test and relative lift based permutation test. (Efron and Tibshirani 1994).

Learn more about the proportion tests in the section Details.

Usage

proportion.test(data, method, B)

Arguments

data	A 2x2 matrix with first column being (control sample size, treatment sample size) and the 2nd column being (control positive response count, treatment positive response count).
method	The method should be one of ("clt", "bootstrap", "bootstrapmean", "permutation")
B	Number of replications for bootstrap test or permutation test. Only required for methods "bootstrap", "bootstrapmean", "permutation".

Details

$\mathbf{clt}$: the classic Z-test based on normal approximation. The absolute lift based Z-test is defined as

$Z = \frac{\hat p_1 - \hat p_0}{\sqrt{{s_0^2}/{n_0} + {s_1^2}/{n_1}}},$

and he relative lift based Z-test is defined as

$Z_r = \frac{\hat p_1 / \hat p_0 - 1}{\sqrt{s_1^2/(n_1\hat p_0^2) + \hat p_1^2s_0^2/(n_0 \hat p_0^4)}},$

where $s_0^2 = \hat p_0(1-\hat p_0)$ and $s_1^2 = \hat p_1(1-\hat p_1)$.

$\mathbf{bootstrap}$: the bootstrap proportion tests proposed in Liu et al., (2023), see Algorithm 1 in their paper. There are two bootstrap tests: the absolute lift based bootstrap test BS-A and the relative lift based bootstrap test BS-R. Note that this type of bootstrap test is testing whether the distribution of the control group is the same as the distribution of the treatment group. In the binomial distribution case, it is equivalent to test whether the mean of the control group is the same as the mean of the treatment group.

$\mathbf{bootstrapmean}$ the bootstrap test to test whether the mean of the control group is the same as the mean of the treatment group. See Algorithm 16.2 of Efron and Tibshirani (1994).

$\mathbf{permutation}$ the permutation test to test whether the distribution of the control group is the same as the distribution of the treatment group. See Algorithm 15.1 of Efron and Tibshirani (1994).

Value

A list of absolute lift, relative lift, standardized absolute lift and their corresponding p-values. Standardized absolute lift equals absolute lift divided by its standard deviation. Only absolute lift and relative lift are available for method clt.

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Efron, Bradley, and Robert J. Tibshirani. An introduction to the bootstrap. CRC press, 1994.

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
set.seed(1)
sim.data <- gen.simu.data(n1, n2, p1, p2, summary = TRUE)
result <- proportion.test(sim.data, method = "bootstrap", B = 1000)
relative.lift <- result$lift$relative
relative.lift.pval <- result$pvalue$relative

---