Package “tpAUC”

pAUC:

The ROC curve is a well-established graphical tool used to evaluate performance of a classifier in accurately discriminating between subjects from different populations (e.g., diseased and healthy individuals). Let F and G be distribution functions of random variables X and Y corresponding to independent populations. Let G⁻¹(t) = inf {y : G(y) ≥ t} be the quantile function of G, 0 < t < 1. Let S_F(t) and S_G(t) be the corresponding survival functions S_F(t) = 1 − F(t) and S_G(t) = 1 − G(t). For t ∈ (0, 1), the ROC curve is defined as ROC(t) = 1 − F{G⁻¹(1 − t)} or ROC(t) = S_F{S_G⁻¹(t)}, where t is the value of FPR and S_G⁻¹(t) = G⁻¹(1 − t). The ROC curve is not a convenient tool for comparisions, in particular when two ROC curves cross. A summary measure of an ROC curve can be found by integrating the ROC curve over the the range of FPR values to obtain the area under the ROC curve as AUC = ∫₀¹ROC(t)dt = ∫_∞^−∞S_F(u)dS_G(u). For economical and practical purposes, it is common to hold the FPR to a low level. When interest is restricted to a sub-region of the ROC space, the partial area under the ROC curve, pAUC(P₀) = ∫₀^P₀ROC(p)dp for the threshold value of FPR P₀ ∈ (0, 1), can provide a useful summary measure.

Let X = {X_i, i = 1, ..., m} and Y = {Y_i, i = 1, ..., n} be random samples from the distribution functions F(x) and G(y), respectively. A Mann-Whitney nonparamteric method for pAUC is (method='WM') $$ \widehat{pAUC}(P_0 ) = \frac{1}{mn}\sum^{m}_{i=1}\sum^{n}_{j=1}I(X_i\ge Y_j)I\{Y_j\ge S^{-1}_{G,n}(P_0)\}, $$ where S_G, n⁻¹(t) = inf {x ∈ R; t ≥ S_G, n(x)} and S_F, m(⋅) and S_G, n(⋅) are estimators of S_F and S_G based on empirical distributions.

Wang and Chang (2011) propose the following method (method = 'expect'), $$\widetilde{pAUC}(P_0) = P_{0} - \frac{1}{m} \overset{m}{\underset{i=1}{\sum}} \min \{ S_{G, n}(X_{i}), P_{0}\}.$$

Yang et al. (2017) propose a jackknife method (method = 'jackknife') based on $\widetilde{pAUC}(P_0)$, in particular, $$ \widetilde{pAUC}_{jack}(P_0)=\frac{1}{n+m}\sum^{n+m}_{h=1}{V}_h(P_0), $$ where $$ {V}_h(P_0)=(n+m)\widetilde{pAUC}(P_0 )-(n+m-1)\widetilde{pAUC}_h(P_0), $$ and $$ \widetilde{pAUC}_h(P_0)=\left\{\begin{array}{ll} P_{0} - \frac{1}{m-1} \sum \limits_{ i \ne h }^m \min \{ S_{G, n}(X_{i}), P_{0}\} & ~\text{ $1 \leq h \leq m$}\\ P_{0} - \frac{1}{m} \sum \limits_{ i=1 }^m \min \{ S_{G,n-1, h-m}(X_{i}), P_{0}\} & ~ \text{$m+1 \leq h \leq m+n,$} \end{array} \right. $$ where $$S_{G,n-1, h-m}(X_{i})=\frac{1}{n-1} \overset{n}{\underset{j=1, j \neq h-m}{\sum}}I(Y_j>X_{i}). $$

pODC:

The ordinal dominance curve (ODC) introduced by Bamber (1973), describes the association between true negative rate (TNR) and false negative rate (FNR), ODC(t) = G{F⁻¹(t)} where t ∈ (0, 1). The area under the ODC, ∫₀¹ODC(t)dt = ∫_−∞^∞G(u)dF(u), is a commonly used summary measure. A partial area under the ODC (pODC) from 0 to P₀ is taken as pODC(P₀) = ∫₀^P₀ODC(t)dt.

A Mann-Whitney nonparamteric method for pAUC is (method='WM') $$\widehat{pODC}(P_0)=\frac{1}{mn}\sum^{m}_{i=1}\sum^{n}_{j=1}I(Y_j\le X_i )I\{X_i\le F^{-1}_{m}(P_0)\},$$ where F_m⁻¹(P₀) is an empirical quantile estimate at P₀ and F_m(⋅) and G_n(⋅) are the empirical distributions of F(⋅) and G(⋅).

Yang et al. (2017) propose the following method (method = 'expect'), $$\widetilde{pODC}(P_0) = P_{0} - \frac{1}{n} \overset{n}{\underset{j=1}{\sum}} \min \{ F_m(Y_{j}), P_{0}\}.$$ Yang et al. (2017) propose a jackknife method (method = 'jackknife') based on $\widetilde{pODC}(P_0)$, in particular, $$ \widetilde{pODC}_{jack}(P_0)=\frac{1}{n+m}\sum^{n+m}_{h=1}\check{U}_h(P_0), $$ where $$ \check{U}_h(P_0)=(n+m)\widetilde{pODC}(P_0)-(n+m-1)\widetilde{pODC}_h(P_0) $$ and $$ \widetilde{pODC}_h(P_0)=\left\{\begin{array}{ll} P_{0} - \frac{1}{n-1} \sum \limits_{j \ne h}^n \min \{ F_{m}(Y_{j}), P_{0}\} & ~\text{ $1 \leq h \leq n$}\\ P_{0} - \frac{1}{n} \sum \limits_{j = 1 }^n \min \{F_{m-1, h-n}(Y_{j}), P_{0}\} & ~ \text{$n+1 \leq h \leq m+n,$} \end{array} \right. $$ where $$F_{m-1, h-n}(Y_{j})=\frac{1}{m-1} \overset{m}{\underset{i=1, i \neq h-n}{\sum}}I(X_{i} \leq Y_j). $$

two-way pAUC:

The definition and estimation of two-way pAUC are proposed intuitively. Given bounds p₀ and q₀, two-way pAUC is formulated as U(p₀, q₀) = ∫_{SG{SF⁻¹(q0)}}^p₀S_F{S_G⁻¹(u)}du − [p₀ − S_G{S_F⁻¹(q₀)}]q₀. Alternatively, from a probability perspective, U(p₀, q₀) can be transformed as: P{Y < X, X ≤ S_F⁻¹(q₀), Y ≥ S_G⁻¹(p₀)}. A trimmed Mann-Whitney U-statistics estimator directly following the above expression is $$ \frac{1}{mn}\sum^{m}_{i=1}\sum^{n}_{j=1}V_{i,j} (p_0 , q_0), $$ where V_i, j(p₀, q₀) = I{Y_j ≤ X_i, X_i ≤ S_F, m⁻¹(q₀), Y_j ≥ S_G, n⁻¹(p₀)}.

Back to Top

pAUC:

Yang et al. (2017) prove that, under certain conditions, $$ \sqrt{m+n}\{\widehat{pAUC}(P_0)-pAUC(P_0)\}\stackrel{d}{\to}N\left\{0,\frac{\sigma^{2}_{1}(P_0)}{\lambda}+\frac{\sigma^{2}_{2}(P_0)}{1-\lambda}\right\}, m,n\to \infty, \nonumber $$ where $\frac{m}{m+n}\to \lambda$, σ₁²(P₀) = ∫_+∞^{S_G−1(P₀)}{P₀ − S_G(t)}²dS_F(t) − {∫_+∞^{S_G−1(P₀)}S_F(t)dS_G(t)}², and σ₂²(P₀) = ∫_+∞^{S_G−1(P₀)}[S_F(t) − S_F{S_G⁻¹(P₀)}]²dS_G(t) − (∫_+∞^{S_G−1(P₀)}[S_F(t) − S_F{S_G⁻¹(P₀)}]dS_G(t))². Moveover, under same conditions, $$\sqrt{m+n}\{\widetilde{pAUC}_{}(P_0)-pAUC(P_0)\}\stackrel{d}{\to}N\left\{0,\frac{\sigma^{2}_{1}(P_0)}{\lambda}+\frac{\sigma^{2}_{2}(P_0)}{1-\lambda}\right\}, m,n\to \infty.$$ and $$ \sqrt{m+n}\{\widetilde{pAUC}_{jack}(P_0)-pAUC(P_0)\}\stackrel{d}{\to}N\left\{0,\frac{\sigma^{2}_{1}(P_0)}{\lambda}+\frac{\sigma^{2}_{2}(P_0)}{1-\lambda}\right\}, m,n\to \infty. $$
Consider the jackknife variance estimator $$S^2_{\widetilde{pAUC}}={(m+n)}^{-1}\sum^{m+n}_{h=1}\{{V}_h(P_0)-\widetilde{pAUC}_{jack}(P_0)\}^2.$$ Yang et al. (2017) prove that $$ S^2_{\widetilde{pAUC}}(P_0)=\frac{\sigma^{2}_{1}(P_0)}{\lambda}+\frac{\sigma^{2}_{2}(P_0)}{1-\lambda}+o_p(1). $$ Therefore, $$ \frac{\sqrt{m+n}\{\widetilde{pAUC}_{jack}(P_0)-pAUC(P_0)\}}{\sqrt{S^2_{\widetilde{pAUC}}(P_0)}}\stackrel{d}{\to}N(0,1). $$

pODC:

In ODC cases, we have $$ \sqrt{m+n}\{\widehat{pODC}(P_0)-pODC(P_0)\}\stackrel{d}{\to}N\left(0,\frac{\sigma^{2}_{3}}{1-\lambda}+\frac{\sigma^{2}_{4}}{\lambda}\right), m,n\to \infty, $$ where σ₃² = ∫_−∞^F−1(P₀){P₀ − F(t)}²dG(t) − {∫_−∞^F−1(P₀)G(t)dF(t)}², and σ₄² = ∫_−∞^F−1(P₀)[G(t) − G{F⁻¹(P₀)}]²dF(t) − (∫_−∞^F−1(P₀)[G(t) − G{F⁻¹(P₀)}]dF(t))². Similarly, $$ \sqrt{m+n}\{\widetilde{pODC}_{}(P_0)-pODC(P_0)\}\stackrel{d}{\to}N\left\{0,\frac{\sigma^{2}_{3}(P_0)}{1- \lambda}+\frac{\sigma^{2}_{4}(P_0)}{\lambda}\right\}, $$ and $$\sqrt{m+n}\{\widetilde{pODC}_{jack}(P_0)-pODC(P_0)\}\stackrel{d}{\to}N\left(0,\frac{\sigma^{2}_{3}}{1- \lambda}+\frac{\sigma^{2}_{4}}{\lambda}\right).$$ Together with $$ \begin{align*} S^2_{\widetilde{pODC}}& ={(m+n)}^{-1}\sum^{m+n}_{h=1}\{\check{U}_h(P_0)-\widetilde{pODC}_{jack}(P_0)\}^2\\ &=\frac{\sigma^{2}_{3}(P_0)}{1-\lambda}+\frac{\sigma^{2}_{4}(P_0)}{\lambda}+o_p(1). \end{align*} $$ and $$ \frac{\sqrt{m+n}\{\widetilde{pODC}_{jack}(P_0)-pODC(P_0)\}}{\sqrt{S^2_{\widetilde{pODC}}(P_0)}}\stackrel{d}{\to}N(0,1). $$

two-way pAUC:

From Yang et al. (2016), we have, under certain conditions, $$ \sqrt{m+n}\{\hat{U}(p_0,q_0)-U(p_0,q_0)\}\stackrel{d}{\to}N\left\{0,\frac{\sigma^2_5}{\lambda}+\frac{\sigma^2_6}{1-\lambda}\right\}, \quad \text{as } \;\; m,n\to \infty, $$ where $$ \begin{align} \sigma^2_5= &F\{G^{-1}(1-p_0)\}[G\{F^{-1}(1-q_0)\}-(1-p_0)]^2+\int^{F^{-1}(1-q_0)}_{G^{-1}(1-p_0)}[G\{F^{-1}(1-q_0)\}-G(t)]^2dF(t)\nonumber\\ &-\left\{\int^{F^{-1}(1-q_0)}_{G^{-1}(1-p_0)}F(t)dG(t)\right\}^2\nonumber, \end{align} $$ and $$ \begin{align} \sigma^2_6=&[1-q_0-F\{G^{-1}(1-p_0)\}]^2(1-p_0)\nonumber+ \int^{F^{-1}(1-q_0)}_{G^{-1}(1-p_0)}\{1-q_0-F(t)\}^2dG(t) \\ \nonumber & -\left\{\int^{F^{-1}(1-q_0)}_{G^{-1}(1-p_0)}G(t)dF(t)\right\}^2\nonumber. \end{align} $$

Back to Top