Package 'metrica'

Description

It estimates the agreement coefficient suggested by Ji & Gallo (2006) for a continuous predicted-observed dataset.

Usage

AC(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The Ji and Gallo's AC measures general agreement, including both accuracy and precision. It is normalized, dimensionless, positively bounded (-infinity;1), and symmetric. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Ji & Gallo (2006). An agreement coefficient for image comparison. Photogramm. Eng. Remote Sensing 7, 823–833 doi:10.14358/PERS.72.7.823

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
df <- data.frame(obs = X, pred = Y)
AC(df, obs = X, pred = Y)

Description

It estimates the accuracy for a nominal/categorical predicted-observed dataset.

Usage

accuracy(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

Accuracy is the simplest and most popular classification metric in literature. It refers to a measure of the degree to which the predictions of a model matches the reality being modeled. The classification accuracy is calculated as the ratio between the number of correctly classified objects with respect to the total number of cases.

It is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low accuracy of predictions. It can be also expressed as percentage if multiplied by 100. It is estimated at a global level (not at the class level).

Accuracy presents limitations to address classification quality under unbalanced classes, and it is not able to distinguish among misclassification distributions. For those cases, it is advised to apply other metrics such as balanced accuracy (baccu), F-score (fscore), Matthews Correlation Coefficient (mcc), or Cohen's Kappa Coefficient (cohen_kappa).

Accuracy is directly related to the error_rate, since accuracy = 1 – error_rate.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Sammut & Webb (2017). Accuracy. In: Sammut C., Webb G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA. doi:10.1007/978-1-4899-7687-1_3

See Also

balacc fscore mcc khat

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE) )

# Get accuracy estimate for two-class case
accuracy(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get accuracy estimate for multi-class case
accuracy(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the Adjusted F-score for a nominal/categorical predicted-observed dataset.

Usage

agf(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The Adjusted F-score (or Adjusted F-measure) is an improvement over the F1-score especially when the data classes are imbalanced. This metric more properly accounts for the different misclassification costs across classes. It weights more the sensitivity (recall) metric than precision and gives strength to the false negative values. This index accounts for all elements of the original confusion matrix and provides more weight to patterns correctly classified in the minority class (positive).

It is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low performance. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Maratea, A., Petrosino, A., Manzo, M. (2014). Adjusted-F measure and kernel scaling for imbalanced data learning. Inf. Sci. 257: 331-341. doi:10.1016/j.ins.2013.04.016

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get F-score estimate for two-class case
agf(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get F-score estimate for each class for the multi-class case
agf(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get F-score estimate for the multi-class case at a global level
agf(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

The AUC estimates the area under the receiver operator curve (ROC) for a nominal/categorical predicted-observed dataset using the Mann-Whitney U-statistic.

Usage

AUC_roc(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The AUC tests whether positives are ranked higher than negatives. It is simply the area under the ROC curve.

The AUC estimation of this function follows the procedure described by Hand & Till (2001). The AUC_roc estimated following the trapezoid approach is equivalent to the average between recall and specificity (Powers, 2011), which is equivalent to the balanced accuracy (balacc):

$AUC_roc = \frac{(recall - FPR + 1)}{2} = \frac{recall+specificity}{2} = 1-\frac{FPR+FNR}{2}$

Interpretation: the AUC is equivalent to the probability that a randomly case from a given class (positive for binary) will have a smaller estimated probability of belonging to another class (negative for binary) compared to a randomly chosen member of the another class.

Values: the AUC is bounded between 0 and 1. The closer to 1 the better. Values close to 0 indicate inaccurate predictions. An AUC = 0.5 suggests no discrimination ability between classes; 0.7 < AUC < 0.8 is considered acceptable, 0.8 < AUC < 0.5 is considered excellent, and AUC > 0.9 is outstanding (Mandrekar, 2010).

For the multiclass cases, the AUC is equivalent to the average of AUC of each class (Hand & Till, 2001).

Finally, the AUC is directly related to the Gini-index (a.k.a. G1) since Gini + 1 = 2*AUC. (Hand & Till, 2001).

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Hanley, J.A., McNeil, J.A. (2017). The meaning and use of the area under a receiver operating characteristic (ROC) curve. _ Radiology 143(1): 29-36_ doi:10.1148/radiology.143.1.7063747

Hand, D.J., Till, R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. _ Machine Learning 45: 171-186_ doi:10.1023/A:1010920819831

Mandrekar, J.N. (2010). Receiver operating characteristic curve in diagnostic test assessment. _ J. Thoracic Oncology 5(9): 1315-1316_ doi:10.1097/JTO.0b013e3181ec173d

Powers, D.M.W. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation. Journal of Machine Learning Technologies 2(1): 37–63. doi:10.48550/arXiv.2010.16061

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE) )

# Get AUC estimate for two-class case
AUC_roc(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get AUC estimate for multi-class case
AUC_roc(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It calculates the intercept (B0) for the bivariate linear relationship between predicted and observed values following the SMA regression.

Usage

B0_sma(data = NULL, obs, pred, orientation = "PO", tidy = FALSE, na.rm = TRUE)

Arguments

Details

SMA is a symmetric linear regression (invariant results/interpretation to axis orientation) recommended to describe the bivariate scatter instead of OLS regression (classic linear model, which results vary with the axis orientation). For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Warton et al. (2006). Bivariate line-fitting methods for allometry. Biol. Rev. Camb. Philos. Soc. 81, 259–291. doi:10.1002/1521-4036(200203)44:2<161::AID-BIMJ161>3.0.CO;2-N

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
B0_sma(obs = X, pred = Y)

Description

It calculates the slope (B1) for the bivariate linear relationship between predicted and observed values following the SMA regression.

Usage

B1_sma(data = NULL, obs, pred, tidy = FALSE, orientation = "PO", na.rm = TRUE)

Arguments

Details

SMA is a symmetric linear regression (invariant results/interpretation to axis orientation) recommended to describe the bivariate scatter instead of OLS regression (classic linear model, which results vary with the axis orientation). For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Warton et al. (2006). Bivariate line-fitting methods for allometry. Biol. Rev. Camb. Philos. Soc. 81, 259–291. doi:10.1002/1521-4036(200203)44:2<161::AID-BIMJ161>3.0.CO;2-N

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
B1_sma(obs = X, pred = Y)

Description

It estimates the balanced accuracy for a nominal/categorical predicted-observed dataset.

Usage

balacc(data = NULL, obs, pred, pos_level = 2, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The balanced accuracy is the average between recall and specificity. It is particularly useful when the number of observation belonging to each class is despair or imbalanced, and when especial attention is given to the negative cases. It is bounded between 0 and 1. Values towards zero indicate low performance. The closer to 1 the better.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

García, V., Mollineda, R.A., Sánchez, J.S. (2009). Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions. In: Araujo, H., Mendonça, A.M., Pinho, A.J., Torres, M.I. (eds) Pattern Recognition and Image Analysis. IbPRIA 2009. Lecture Notes in Computer Science, vol 5524. Springer-Verlag Berlin Heidelberg. doi:10.1007/978-3-642-02172-5_57

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get balanced accuracy estimate for two-class case
balacc(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get balanced accuracy estimate for the multi-class case
balacc(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

This example dataset is a set of APSIM simulations of barley grain number (x1000 grains per squared meter), which presents high accuracy but medium precision. The experimental trials come from 11 site-years in 2 countries (Australia, and New Zealand). These data correspond to the latest, up-to-date, documentation and validation of version number 2020.03.27.4956. Data available at: doi:10.7910/DVN/EJS4M0. Further details can be found at the official APSIM Next Generation website: https://APSIMnextgeneration.netlify.app/modeldocumentation/

Usage

barley

Format

This data frame has 69 rows and the following 2 columns:

pred

predicted values

obs

observed values

Source

https://github.com/adriancorrendo/metrica

Description

It creates a scatter plot of the difference between observed and predicted values (obs-pred) vs. observed values.

Usage

bland_altman_plot(
  data = NULL,
  obs,
  pred,
  shape_type = NULL,
  shape_size = NULL,
  shape_color = NULL,
  shape_fill = NULL,
  zeroline_type = NULL,
  zeroline_size = NULL,
  zeroline_color = NULL,
  limitsline_type = NULL,
  limitsline_size = NULL,
  limitsline_color = NULL,
  na.rm = TRUE
)

Arguments

Details

For more details, see online-documentation

Value

an object of class ggplot.

References

Bland & Altman (1986). Statistical methods for assessing agreement between two methods of clinical measurement The Lancet 327(8476), 307-310 doi:10.1016/S0140-6736(86)90837-8

See Also

ggplot,geom_point,aes

Examples

X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 10)
bland_altman_plot(obs = X, pred = Y)

Description

It estimates the Bookmaker Informedness (a.k.a. Youden's J-index) for a nominal/categorical predicted-observed dataset.

jindex estimates the Youden's J statistic or Youden's J Index (equivalent to Bookmaker Informedness bmi)

Usage

bmi(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

jindex(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The Bookmaker Informedness (or Youden's J index) it is a suitable metric when the number of cases for each class is uneven.

The general formula applied to both binary and multiclass cases is:

$bmi = recall + specificity - 1$

It is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low performance. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Youden, W.J. (1950). Index for rating diagnostic tests. . Cancer 3: 32-35. doi:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get Informedness estimate for two-class case
bmi(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get Informedness estimate for each class for the multi-class case
bmi(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE, atom = TRUE)

# Get Informedness estimate for the multi-class case at a global level
bmi(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the Concordance Correlation Coefficient (CCC) for a continuous predicted-observed dataset.

Usage

CCC(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The CCC it is a normalized coefficient that tests general agreement. It presents both precision (r) and accuracy (Xa) components. It is positively bounded to 1. The closer to 1 the better. Values towards zero indicate low correlation between observations and predictions. Negative values would indicate a negative relationship between predicted and observed. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Lin (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45 (1), 255–268. doi:10.2307/2532051

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
CCC(obs = X, pred = Y)

Description

This example dataset is a set of APSIM simulations of chickpea aboveground dry mass (kg per hectare), which exhibits low accuracy and medium-low precision as it represents a model still under development. The experimental trials come from 7 site-years in 3 countries (Australia, India, and New Zealand). These data correspond to the latest, up-to-date, documentation and validation of version number 2020.03.27.4956. Data available at: doi:10.7910/DVN/EJS4M0. Further details can be found at the official APSIM Next Generation website: https://APSIMnextgeneration.netlify.app/modeldocumentation/

Usage

chickpea

Format

This data frame has 39 rows and the following 2 columns:

pred

predicted values

obs

observed values

Source

https://github.com/adriancorrendo/metrica

Description

It creates a confusion matrix table or plot displaying the agreement between the observed and the predicted classes by the model.

Usage

confusion_matrix(
  data = NULL,
  obs,
  pred,
  plot = FALSE,
  unit = "count",
  colors = c(low = NULL, high = NULL),
  print_metrics = FALSE,
  metrics_list = c("accuracy", "precision", "recall"),
  position_metrics = "top",
  na.rm = TRUE
)

Arguments

Details

A confusion matrix is a method for summarizing the predictive performance of a classification algorithm. It is particularly useful if you have an unbalanced number of observations belonging to each class or if you have a multinomial dataset (more than two classes in your dataset. A confusion matrix can give you a good hint about the types of errors that your model is making. See online-documentation

Value

An object of class data.frame when plot = FALSE, or of type ggplot when plot = TRUE.

References

Ting K.M. (2017). Confusion Matrix. In: Sammut C., Webb G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA. doi:10.1007/978-1-4899-7687-1_50

See Also

eval_tidy, defusing-advanced select

Examples

set.seed(183)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE),
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100,
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE))

# Plot two-class confusion matrix
confusion_matrix(data = binomial_case, obs = labels, pred = predictions, 
plot = TRUE, colors = c(low="pink" , high="steelblue"), unit = "count")

# Plot multi-class confusion matrix
confusion_matrix(data = multinomial_case, obs = labels, pred = predictions, 
plot = TRUE, colors = c(low="#f9dbbd" , high="#735d78"), unit = "count")

Description

It estimates the Critical Success Index (a.k.a. threat score, Jaccard Index) for a nominal/categorical predicted-observed dataset.

jaccardindex estimates the Jaccard similarity coefficient or Jaccard's Index (equivalent to the Critical Success Index csi).

Usage

csi(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

jaccardindex(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The csi is also known as the threat score or the Jaccard Index. It is a metric especially useful for binary classification tasks, representing the proportion of true positive (TP) cases with respect to the sum of predicted positive (PP = TP + FP) and true negative (TN) cases.

$csi = \frac{TP}{TP + TN + FP}$

It is bounded between 0 and 1. The closer to 1 the better the classification performance, while zero represents the worst result.

It has been extensively used in meteorology (NOOA) as a verification measure of categorical forecast performance equal to the total number of correct event forecast (hits = TP) divided by the total number of event forecasts plus the number of misses (hits + false alarms + misses = TP + FP + TN). However, the csi has been criticized for not representing an unbiased measure of forecast skill (Schaefer, 1990).

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

NOOA. Forecast Verification Glossary. Space Weather Prediction Center, NOOA. https://www.swpc.noaa.gov/sites/default/files/images/u30/Forecast%20Verification%20Glossary.pdf

Schaefer, J.T. (1990). The critical success index as an indicator of warning skill. Weather and Forecasting 5(4): 570-575. doi:10.1175/1520-0434(1990)005<0570:TCSIAA>2.0.CO;2

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Get csi estimate for two-class case
csi(data = binomial_case, obs = labels, pred = predictions)

Description

It estimates the Willmott's index of agreement (d) for a continuous predicted-observed dataset.

Usage

d(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The d index it is a normalized, dimensionless metric that tests general agreement. It measures both accuracy and precision using squared residuals. It is bounded between 0 and 1. The disadvantage is that d is an asymmetric index, that is, dependent to what is orientation of predicted and observed values. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Willmott (1981). On the validation of models. Phys. Geogr. 2, 184–194. doi:10.1080/02723646.1981.10642213

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
d(obs = X, pred = Y)

Description

It estimates the modified index of agreement (d1) using absolute differences following Willmott et al. (1985).

Usage

d1(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

Similar to d, the d1 index it is a normalized, dimensionless metric that tests general agreement. The difference with d, is that d1 uses absolute residuals instead of squared residuals. It is bounded between 0 and 1. The disadvantage is that d is an asymmetric index, that is, dependent to the orientation of predicted and observed values. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Willmott et al. (1985). Statistics for the evaluation and comparison of models. J. Geophys. Res. 90, 8995. doi:10.1029/jc090ic05p08995

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
d1(obs = X, pred = Y)

Description

It estimates the refined index of agreement (d1r) following Willmott et al. (2012).

Usage

d1r(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

Similar to d, and d1, the d1r index it is a normalized, dimensionless metric that tests general agreement. The difference is that d1r modifies the denominator of the formula (potential error), normalizing the mean absolute error (numerator) by two-times the mean absolute deviation of observed values. It is bounded between 0 and 1. The disadvantage is that d1r is an asymmetric index, that is, dependent to the orientation of predicted and observed values. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Willmott et al. (2012). A refined index of model performance. Int. J. Climatol. 32, 2088–2094. doi:10.1002/joc.2419

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
d1r(obs = X, pred = Y)

Description

It estimates the Distance Correlation coefficient (dcorr) for a continuous predicted-observed dataset.

Usage

dcorr(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The dcorr function is a wrapper for the dcor function from the energy-package. See Rizzo & Szekely (2022). The distance correlation (dcorr) coefficient is a novel measure of dependence between random vectors introduced by Szekely et al. (2007).

The dcorr is characterized for being symmetric, which is relevant for the predicted-observed case (PO).

For all distributions with finite first moments, distance correlation $\mathcal R$ generalizes the idea of correlation in two fundamental ways:

(1) $\mathcal R(P,O)$ is defined for $P$ and $O$ in arbitrary dimension.

(2) $\mathcal R(P,O)=0$ characterizes independence of $P$ and $O$.

Distance correlation satisfies $0 \le \mathcal R \le 1$, and $\mathcal R = 0$ only if $P$ and $O$ are independent. Distance covariance $\mathcal V$ provides a new approach to the problem of testing the joint independence of random vectors. The formal definitions of the population coefficients $\mathcal V$ and $\mathcal R$ are given in Szekely et al. (2007).

The empirical distance correlation $\mathcal{R}_n(\mathbf{P,O})$ is the square root of

$\mathcal{R}^2_n(\mathbf{P,O})= \frac {\mathcal{V}^2_n(\mathbf{P,O})} {\sqrt{ \mathcal{V}^2_n (\mathbf{P}) \mathcal{V}^2_n(\mathbf{O})}}.$

For the formula and more details, see online-documentation and the energy-package

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Szekely, G.J., Rizzo, M.L., and Bakirov, N.K. (2007). Measuring and testing dependence by correaltion of distances. Annals of Statistics, Vol. 35(6): 2769-2794. doi:10.1214/009053607000000505.

Rizzo, M., and Szekely, G. (2022). energy: E-Statistics: Multivariate Inference via the Energy of Data. R package version 1.7-10. https://CRAN.R-project.org/package=energy.

See Also

eval_tidy, defusing-advanced dcor, energy

Examples

set.seed(1)
P <- rnorm(n = 100, mean = 0, sd = 10)
O <- P + rnorm(n=100, mean = 0, sd = 3)
dcorr(obs = P, pred = O)

Description

deltap estimates the Markedness or deltaP for a nominal/categorical predicted-observed dataset.

mk estimates the Markedness (equivalent to deltaP) for a nominal/categorical predicted-observed dataset.

Usage

deltap(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

mk(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The deltap is also known as the markedness. It is a metric that quantifies the probability that a condition is marked by the predictor with respect to a random chance (Powers, 2011).

The deltap is related to precision (or positive predictive values -ppv-) and its inverse (the negative predictive value -npv-) as follows:

$deltap = PPV + NPV - 1 = precision + npv - 1$

The higher the deltap the better the classification performance.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Powers, D.M.W. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation. Journal of Machine Learning Technologies 2(1): 37–63. doi:10.48550/arXiv.2010.16061

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Get deltap estimate for two-class case
deltap(data = binomial_case, obs = labels, pred = predictions)

Description

It draws a density area plot of predictions and observations with alternative axis orientation (P vs. O; O vs. P).

Usage

density_plot(
  data = NULL,
  obs,
  pred,
  n = 10,
  colors = c(low = NULL, high = NULL),
  orientation = "PO",
  print_metrics = FALSE,
  metrics_list = NULL,
  position_metrics = c(x = NULL, y = NULL),
  print_eq = TRUE,
  position_eq = c(x = NULL, y = NULL),
  eq_color = NULL,
  regline_type = NULL,
  regline_size = NULL,
  regline_color = NULL,
  na.rm = TRUE
)

Arguments

Details

It creates a density plot of predicted vs. observed values. The plot also includes the 1:1 line (solid line) and the linear regression line (dashed line). By default, it places the observed on the x-axis and the predicted on the y-axis (orientation = "PO"). This can be inverted by changing the argument orientation = “OP". For more details, see online-documentation

Value

Object of class ggplot.

See Also

ggplot,geom_point,aes

Examples

X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 10)
density_plot(obs = X, pred = Y)

Description

It estimates the E1 model efficiency using absolute differences.

Usage

E1(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The E1 is a type of model efficiency that modifies the Nash-Sutcliffe model efficiency by using absolute residuals instead of squared residuals. The E1 is used to overcome the NSE over-sensitivity to extreme values caused by the used of squared residuals. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Legates & McCabe (1999). Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resour. Res. doi:10.1029/1998WR900018

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
E1(obs = X, pred = Y)

Description

It estimates the Erel model efficiency using differences to observations.

Usage

Erel(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The Erel model efficiency normalizes both residuals (numerator) and observed deviations (denominator) by observed values before squaring them. Compared to the NSE, the Erel is suggested as more sensitive to systematic over- or under-predictions. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Krause et al. (2005). Comparison of different efficiency criteria for hydrological model assessment. Adv. Geosci. 5, 89–97. doi:10.5194/adgeo-5-89-2005

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
Erel(obs = X, pred = Y)

Description

It estimates the error rate for a nominal/categorical predicted-observed dataset.

Usage

error_rate(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The error rate represents the opposite of accuracy, referring to a measure of the degree to which the predictions miss-classify the reality. The classification error_rate is calculated as the ratio between the number of incorrectly classified objects with respect to the total number of objects. It is bounded between 0 and 1. The closer to 1 the worse. Values towards zero indicate low error_rate of predictions. It can be also expressed as percentage if multiplied by 100. It is estimated at a global level (not at the class level). The error rate is directly related to the accuracy, since error_rate = 1 – accuracy' . For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

(2017) Accuracy. In: Sammut C., Webb G.I. (eds) Encyclopedia of Machine Learning and Data Mining Springer, Boston, MA. doi:10.1007/978-1-4899-7687-1_3

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get error_rate estimate for two-class case
error_rate(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get error_rate estimate for multi-class case
error_rate(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the Fowlkes-Mallows Index for a nominal/categorical predicted-observed dataset.

Usage

fmi(data = NULL, obs, pred, pos_level = 2, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The fmi has gained popularity within the machine learning community to summarize into a single value the confusion matrix of a binary classification. It is particularly useful when the number of observations belonging to each class is uneven or imbalanced. It is characterized for being symmetric (i.e. no class has more relevance than the other). It is bounded between -1 and 1. The closer to 1 the better the classification performance.

The fmi is only available for the evaluation of binary cases (two classes). For multiclass cases, fmi will produce a NA and display a warning.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Fowlkes, Edward B; Mallows, Colin L (1983). A method for comparing two hierarchical clusterings. Journal of the American Statistical Association. 78 (383): 553–569. doi:10.1080/01621459.1983.10478008

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Get fmi estimate for two-class case
fmi(data = binomial_case, obs = labels, pred = predictions)

Description

It estimates the F-score for a nominal/categorical predicted-observed dataset.

Usage

fscore(
  data = NULL,
  obs,
  pred,
  B = 1,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The F-score (or F-measure) it is a more robust metric than the classic accuracy, especially when the number of cases for each class is uneven. It represents the harmonic mean of precision and recall. Thus, to achieve high values of F-score it is necessary to have both high precision and high recall.

The universal version of F-score employs a coefficient B, by which we can tune the precision-recall ratio. Values of B > 1 give more weight to recall, and B < 1 give more weight to precision.

For binomial/binary cases, fscore = TP / (TP + 0.5*(FP + FN))

The generalized formula applied to multiclass cases is:

$fscore = \frac{(1 + B ^ 2) * (precision * recall)} {((B ^ 2 * precision) + recall)}$

It is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low performance. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Goutte, C., Gaussier, E. (2005). A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In: D.E. Losada and J.M. Fernandez-Luna (Eds.): ECIR 2005 . Advances in Information Retrieval LNCS 3408, pp. 345–359, 2. Springer-Verlag Berlin Heidelberg. doi:10.1007/978-3-540-31865-1_25

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get F-score estimate for two-class case
fscore(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get F-score estimate for each class for the multi-class case
fscore(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get F-score estimate for the multi-class case at a global level
fscore(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the Geometric Mean score for a nominal/categorical predicted-observed dataset.

Usage

gmean(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The gmean is a metric especially useful for imbalanced classes because it measures the balance between the classification performance on both major (over-represented) as well as on minor (under-represented) classes. As stated above, it is particularly useful when the number of observations belonging to each class is uneven.

The gmean score is equivalent to the square-root of the product of specificity and recall (a.k.a. sensitivity).

$gmean = \sqrt{recall * specificity}$

It is bounded between 0 and 1. The closer to 1 the better the classification performance, while zero represents the worst.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

De Diego, I.M., Redondo, A.R., Fernández, R.R., Navarro, J., Moguerza, J.M. (2022). General Performance Score for classification problems. _ Appl. Intell. (2022)._ doi:10.1007/s10489-021-03041-7

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Get gmean estimate for two-class case
gmean(data = binomial_case, obs = labels, pred = predictions)

Description

Imports data from SQLite databases (*.db) applied by APSIM Next Generation. It reads one file at a time.

Usage

import_apsim_db(filename = "", folder = ".", value = "report", simplify = TRUE)

Arguments

Value

An object of class data.frame, but it depends on the argument ‘value’ above

Note

This function was adapted from apsimx package (Miguez, 2022). For reference, we recommend to check and use the function apsimx::read_apsimx() as an alternative. Source: https://github.com/femiguez/apsimx by F. Miguez.

References

Miguez, F. (2022) apsimx: Inspect, Read, Edit and Run 'APSIM' "Next Generation" and 'APSIM' Classic. R package version 2.3.1, https://CRAN.R-project.org/package=apsimx

Examples

## See [documentation](https://adriancorrendo.github.io/metrica/index.html)

Description

Function to import *.out files generated by APSIM Classic.

Usage

import_apsim_out(filepath)

Arguments

Details

This function will generate a data frame object from an APSIM Classic *.out file.

Value

Element of class data.frame.

Note

Note: It imports one file at a time.

Examples

## See [documentation](https://adriancorrendo.github.io/metrica/index.html)

Description

It estimates the IqRMSE for a continuous predicted-observed dataset.

Usage

iqRMSE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The iqRMSE normalizes the RMSE by the length of the inter-quartile range of observations (percentiles 25th to 75th). As an error metric, the lower the values the better. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
iqRMSE(obs = X, pred = Y)

Description

It estimates the KGE for a predicted-observed dataset.

Usage

KGE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The KGE is a normalized, dimensionless, model efficiency that measures general agreement. It presents accuracy, precision, and consistency components. It is symmetric (invariant to predicted observed orientation). It is positively bounded up to 1. The closer to 1 the better. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Kling et al. (2012). Runoff conditions in the upper Danube basin under an ensemble of climate change scenarios. Journal of Hydrology 424-425, 264-277. doi:10.1016/j.jhydrol.2012.01.011

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
KGE(obs = X, pred = Y)

Description

It estimates the Cohen's Kappa Coefficient for a nominal/categorical predicted-observed dataset.

Usage

khat(data = NULL, obs, pred, pos_level = 2, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The Cohen's Kappa Coefficient is the accuracy normalized by the possibility of agreement by chance. Thus, it is considered a more robust agreement measure than simply the accuracy. The kappa coefficient was originally described for evaluating agreement of classification between different "raters" (inter-rater reliability).

It is positively bounded to 1, but it is not negatively bounded. The closer to 1 the better as Kappa assumes its theoretical maximum value of 1 (perfect agreement) only when both observed and predicted values are equally distributed across the classes (i.e. identical row and column sums). Thus, the lower the kappa the lower the prediction quality.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. _ Educational and Psychological Measurement 20 (1): 37–46._ doi:10.1177/001316446002000104

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE))

# Get Cohen's Kappa Coefficient estimate for two-class case
khat(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get Cohen's Kappa Coefficient estimate for each class for the multi-class case
khat(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get Cohen's Kappa Coefficient estimate for the multi-class case at a global level
khat(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the agreement coefficient (lambda) suggested by Duveiller et al. (2016) for a continuous predicted-observed dataset.

Usage

lambda(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

lambda measures both accuracy and precision. It is normalized, dimensionless, bounded (-1;1), and symmetric (invariant to predicted-observed orientation). lambda is equivalent to CCC when r is greater or equal to 0. The closer to 1 the better. Values towards zero indicate low correlation between observations and predictions. Negative values would indicate a negative relationship between predicted and observed. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Duveiller et al. (2016). Revisiting the concept of a symmetric index of agreement for continuous datasets. Sci. Rep. 6, 1-14. doi:10.1038/srep19401

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
lambda(obs = X, pred = Y)

Description

This example dataset was obtained in 2022 over a small region in Kansas, using visual interpretation. The predicted values were the result of a Random Forest classification, with a 70/30 % split. Values equal to 1 are associated to vegetation, and values equal to 0 are other type of land cover.

Usage

land_cover

Format

This data frame has 284 rows and the following 2 columns:

predicted

predicted values

actual

observed values

Source

https://github.com/adriancorrendo/metrica

Description

It estimates the lack of correlation (LCS) component of the Mean Squared Error (MSE) proposed by Kobayashi & Salam (2000).

Usage

LCS(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The LCS represents the random component of the prediction error following Kobayashi & Salam (2000). The lower the value the less contribution to the MSE. However, it needs to be compared to MSE as its benchmark. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Kobayashi & Salam (2000). Comparing simulated and measured values using mean squared deviation and its components. Agron. J. 92, 345–352. doi:10.2134/agronj2000.922345x

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
LCS(obs = X, pred = Y)

Description

It estimates the positive likelihood ratio posLr, negative likelihood ratio negLr, and diagnostic odds ratio dor for a nominal/categorical predicted-observed dataset.

Usage

posLr(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

negLr(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

dor(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  atom = FALSE,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The likelihood ratios are metrics designed to assess the expectations of classification tasks. They are highly related to recall (sensitivity or true positive rate) and specificity (selectivity or true negative rate).

For a multiclass case, positive and negative results are class-wise. We can either hit the actual class, or the actual non-class. For example, a classification may have 3 potential results: cat, dog, or fish. Thus, when the actual value is dog, the only true positive is dog, but we can obtain a true negative either by classifying it as a cat or a fish. The posLr, negLr, and dor estimates are the mean across all classes.

The positive likelihood ratio (posLr) measures the odds of obtaining a positive prediction in cases that are actual positives.

The negative likelihood ratio (negLr) relates the odds of obtaining a negative prediction for actual non-negatives relative to the probability of actual negative cases obtaining a negative prediction result.

Finally, the diagnostic odds ratio (dor) measures the effectiveness of classification. It represents the odds of a positive prediction result in actual (observed) positive cases with respect to the odds of a positive prediction for the actual negative cases.

The ratios are define as follows:

$posLr = \frac{recall}{1+specificity} = \frac{TPR}{FPR}$

$negLr = \frac{1-recall}{specificity} = \frac{FNR}{TNR}$

$dor = \frac{posLr}{negLr}$

For more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

GlasaJeroen, A.S., Lijmer, G., Prins, M.H., Bonsel, G.J., Bossuyta, P.M.M. (2009). The diagnostic odds ratio: a single indicator of test performance. Journal of Clinical Epidemiology 56(11): 1129-1135. doi:10.1016/S0895-4356(03)00177-X

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get posLr, negLr, and dor for a two-class case
posLr(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)
negLr(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)
dor(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the MAE for a continuous predicted-observed dataset.

Usage

MAE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MAE measures both lack of accuracy and precision in absolute scale. It keeps the same units than the response variable. It is less sensitive to outliers than the MSE or RMSE. The lower the better. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Willmott & Matsuura (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim. Res. 30, 79–82. doi:10.3354/cr030079

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MAE(obs = X, pred = Y)

Description

This example data set includes maize phenology classes collected in Kansas during the 2018 growing season. The predicted values were obtained using a Random Forest classifier with a 70/30 split. The dataset includes 16 phenology stages from VT to R6. For more information please visit doi:10.3390/rs14030469.

Usage

maize_phenology

Format

This data frame has 103 rows and the following 2 columns:

predicted

predicted values

actual

observed values

Source

https://github.com/adriancorrendo/metrica

Description

It estimates the MAPE of a continuous predicted-observed dataset.

Usage

MAPE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MAPE is expressed in percentage units (independent scale), which it makes easy to explain and to compare performance across models with different response variables. MAPE is asymmetric (sensitive to axis orientation). The lower the better. As main disadvantage, MAPE produces infinite or undefined values for zero or close-to-zero observed values. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Kim & Kim (2016). A new metric of absolute percentage error for intermittent demand forecast. _Int. J. Forecast. 32, 669-679._doi:10.1016/j.ijforecast.2015.12.003

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MAPE(obs = X, pred = Y)

Description

It estimates the mean absolute error using the naive-error approach for a continuous predicted-observed dataset.

Usage

MASE(
  data = NULL,
  obs,
  pred,
  time = NULL,
  naive_step = 1,
  oob_mae = NULL,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The MASE is especially well suited for time series predictions. It can be used to compare forecast methods on a single series and also to compare forecast accuracy between series.

This metric never gives an infinite or undefined values (unless all observations over time are exactly the same!).

By default, the MASE scales the error based on in-sample MAE from the naive forecast method (random walk). The in-sample MAE is used in the denominator because it is always available and it effectively scales the errors.Since it is based on absolute error, it is less sensitive to outliers compared to the classic MSE.

$MASE = \frac{1}{n}(\frac{|O_i-P_i|}{ \frac{1}{T-1} \sum^T_{t=2}~|O_t - O_{t-1}| })$

If available, users may use and out-of-bag error from an independent dataset, which can be specified with the oob_mae arg. and will replace the denominator into the MASE equation.

MASE measures total error (i.e. both lack of accuracy and precision.). The lower the MASE below 1, the better the prediction quality. MASE = indicates no difference between the model and the naive forecast (or oob predictions). MASE > 1 indicate poor performance.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Hyndman, R.J., Koehler, A.B. (2006). Another look at measures of forecast accuracy. _ Int. J. Forecast_ doi:10.3354/cr030079

Examples

set.seed(1)
# Create a fake dataset
X <- rnorm(n = 100, mean = 8, sd = 10)
Y <- rnorm(n = 100, mean = 8.2, sd = 15)
Z <- seq(1, 100, by = 1)

time_data <- as.data.frame(list("observed" = X, "predicted" = Y, "time" = Z))
 
MASE(data = time_data, obs = observed, pred = predicted, time = time)

Description

It estimates the MBE for a continuous predicted-observed dataset.

Usage

MBE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MBE is one of the most widely used error metrics. It presents the same units than the response variable, and it is unbounded. It can be simply estimated as the difference between the means of predictions and observations. The closer to zero the better. Negative values indicate overestimation. Positive values indicate general underestimation. The disadvantages are that is only sensitive to additional bias, so the MBE may mask a poor performance if overestimation and underestimation co-exist (a type of proportional bias). For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MBE(obs = X, pred = Y)

Description

It estimates the mcc for a nominal/categorical predicted-observed dataset.

phi_coef estimates the Phi coefficient (equivalent to the Matthews Correlation Coefficient mcc).

Usage

mcc(data = NULL, obs, pred, pos_level = 2, tidy = FALSE, na.rm = TRUE)

phi_coef(data = NULL, obs, pred, pos_level = 2, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The mcc it is also known as the phi-coefficient. It has gained popularity within the machine learning community to summarize into a single value the confusion matrix of a binary classification.

It is particularly useful when the number of observations belonging to each class is uneven or imbalanced. It is characterized for being symmetric (i.e. no class has more relevance than the other). It is bounded between -1 and 1. The closer to 1 the better the classification performance.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Chicco, D., Jurman, G. (2020) The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 6 (2020). doi:10.1186/s12864-019-6413-7

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))

# Get mcc estimate for two-class case
mcc(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates a group of metrics characterizing the prediction performance for a continuous (regression) or categorical (classification) predicted-observed dataset. By default, it calculates all available metrics for either regression or classification.

Usage

metrics_summary(
  data = NULL,
  obs,
  pred,
  type = NULL,
  metrics_list = NULL,
  orientation = "PO",
  pos_level = 2,
  na.rm = TRUE
)

Arguments

Details

The user can choose to calculate a single metric, or to calculate all metrics at once. This function creates a data.frame with all (or selected) metrics in the metrica-package. If looking for specific metrics, the user can pass a list of desired metrics using the argument “metrics_list” (e.g. metrics_list = c("R2","MAE", "RMSE", "RSR", "NSE", "KGE")). For the entire list of available metrics with formula, see online-documentation

Value

an object of class data.frame containing all (or selected) metrics.

Examples

# Continuous variable (regression)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 10)
regression_case <- data.frame(obs = X, pred = Y)

# Get a metrics summary for a regression problem
metrics_summary(regression_case, obs = X, pred = Y, type = "regression")

# Categorical variable (classification)
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))

#' # Get a metrics summary for a regression problem
metrics_summary(binomial_case, obs = labels, pred = predictions,
type = "classification")

Description

It estimates the Maximal Information Coefficient (MIC) for a continuous predicted-observed dataset.

Usage

MIC(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MIC function is a wrapper for the mine_stat function of the minerva-package, a collection of Maximal Information-Based Nonparametric statistics (MINE). See Reshef et al. (2011).

For the predicted-observed case (PO), the MIC is defined as follows:

$\textrm{MIC}(D)=\max_{PO<B(n)} M(D)_{X,Y} = \max_{PO<B(n)} \frac{I^ * (D,P,O)} {log(\min{P,O})},$

where $B(n)=n^{\alpha}$ is the search-grid size, $I^*(D,P,O)$ is the maximum mutual information over all grids P-by-O, of the distribution induced by D on a grid having P and O bins (where the probability mass on a cell of the grid is the fraction of points of D falling in that cell). Albanese et al. (2013).

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Reshef, D., Reshef, Y., Finucane, H., Grossman, S., McVean, G., Turnbaugh, P., Lander, R., Mitzenmacher, M., and Sabeti, P. (2011). Detecting novel associations in large datasets. Science 334, 6062. doi:10.1126/science.1205438.

Albanese, D., M. Filosi, R. Visintainer, S. Riccadonna, G. Jurman, C. Furlanello. minerva and minepy: a C engine for the MINE suite and its R, Python and MATLAB wrappers. Bioinformatics (2013) 29(3):407-408. doi:10.1093/bioinformatics/bts707.

See Also

eval_tidy, defusing-advanced mine_stat

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MIC(obs = X, pred = Y)

Description

It estimates the MLA, the systematic error component to the Mean Squared Error (MSE), for a continuous predicted-observed dataset following Correndo et al. (2021).

Usage

MLA(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MLA represents the systematic (bias) component of the MSE. It is obtained via a symmetric decomposition of the MSE (invariant to predicted-observed orientation) using a symmetric regression line. The MLA is equal to the sum of systematic differences divided by the sample size (n). The greater the value the greater the bias of the predictions. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MLA(obs = X, pred = Y)

Description

It estimates the MLP, the unsystematic error component to the Mean Squared Error (MSE), for a continuous predicted-observed dataset following Correndo et al. (2021).

Usage

MLP(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MLP represents the unsystematic (random) component of the MSE. It is obtained via a symmetric decomposition of the MSE (invariant to predicted-observed orientation) using a symmetric regression line. The MLP is equal to the sum of unsystematic differences divided by the sample size (n). The greater the value the greater the random noise of the predictions. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
MLP(obs = X, pred = Y)

Description

It estimates the MSE for a continuous predicted-observed dataset.

Usage

MSE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The MSE, also known as MSD, measures general agreement, as includes both variance (lack of precision) and bias (lack of accuracy). The MSE of predictions could be decomposed following a variety of approaches (e.g. Willmott et al. 1981; Correndo et al. 2021). Its calculation is simple, the sum of squared differences between predictions and observations divided by the sample size (n). The greater the value the worse the predicted performance. Unfortunately, the units of MSE do not have a direct interpretation. For a more direct interpretation, the square root of MSE (RMSE) has the same units as the variable of interest. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Willmott (1981). On the validation of models. Phys. Geogr. 2, 184–194. doi:10.1080/02723646.1981.10642213

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
MSE(obs = X, pred = Y)

Description

npv estimates the npv (a.k.a. positive predictive value -PPV-) for a nominal/categorical predicted-observed dataset.

FOR estimates the false omission rate, which is the complement of the negative predictive value for a nominal/categorical predicted-observed dataset.

Usage

npv(
  data = NULL,
  obs,
  pred,
  atom = FALSE,
  pos_level = 2,
  tidy = FALSE,
  na.rm = TRUE
)

FOR(
  data = NULL,
  obs,
  pred,
  atom = FALSE,
  pos_level = 2,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The npv is a non-normalized coefficient that represents the ratio between the correctly predicted cases (or true positive -TP- for binary cases) to the total predicted observations for a given class (or total predicted positive -PP- for binary cases) or at overall level.

For binomial cases, $npv = \frac{TP}{PP} = \frac{TP}{TP + FP}$

The npv metric is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low npv of predictions. It can be estimated for each particular class or at a global level.

The false omission rate (FOR) represents the proportion of false negatives with respect to the number of negative predictions (PN).

For binomial cases, $FOR = 1 - npv = \frac{FN}{PN} = \frac{FN}{TN + FN}$

The npv metric is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low npv of predictions.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Wang H., Zheng H. (2013). Negative Predictive Value. In: Dubitzky W., Wolkenhauer O., Cho KH., Yokota H. (eds) Encyclopedia of Systems Biology. _ Springer, New York, NY._ doi:10.1007/978-1-4419-9863-7_234

Trevethan, R. (2017). Sensitivity, Specificity, and Predictive Values: Foundations, Pliabilities, and Pitfalls _ in Research and Practice. Front. Public Health 5:307_ doi:10.3389/fpubh.2017.00307

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE))

# Get npv estimate for two-class case
npv(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get fdr estimate for two-class case
FDR(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get npv estimate for each class for the multi-class case
npv(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE, atom = TRUE)

# Get npv estimate for the multi-class case at a global level
npv(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE, atom = TRUE)

Description

It estimates the model efficiency suggested by Nash & Sutcliffe (1970) for a continuous predicted-observed dataset.

Usage

NSE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The NSE measures general agreement. It is normalized (by the variance of the observations) and dimensionless. It is calculated using the absolute squared differences between the predictions and observations, which has been suggested as an issue due to over-sensitivity to outliers. It goes form -infinity to 1. The closer to 1 the better the prediction performance. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Nash & Sutcliffe (1970). River flow forecasting through conceptual models part I - A discussion of principles. J. Hydrol. 10(3), 292-290. doi:10.1016/0022-1694(70)90255-6

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- rnorm(n = 100, mean = 0, sd = 9)
NSE(obs = X, pred = Y)

Description

It estimates the P4-metric for a nominal/categorical predicted-observed dataset.

Usage

p4(
  data = NULL,
  obs,
  pred,
  pos_level = 2,
  tidy = FALSE,
  na.rm = TRUE,
  atom = FALSE
)

Arguments

Details

The P4-metric it is a metric designed for binary classifiers. It is estimated from precision, recall, specificity, and npv (negative predictive value). The P4 it was designed to address criticism against the F-score, so it may be perceived as its extension. Unfortunately, it has not been generalized yet for multinomial cases.

For binomial/binary cases,

$p4 = \frac{(4 x TP x TN)} {(4 x TP x TN + (TP + TN) x FP + FN)}$

Or:

$p4 = \frac{4} {\frac{1}{precision} + \frac{1}{recall} + \frac{1}{specificity} + \frac{1}{npv} }$

The P4 metric has not been generalized for multinomial cases. The P4 metric is bounded between 0 and 1. The closer to 1 the better, which will require precision, recall, specificity and npv being close to 1. Values towards zero indicate low performance, which could be the product of only one of the four conditional probabilities being close to 0.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Sitarz, M. (2023). Extending F1 metric, probabilistic approach. _Adv. Artif. Intell. Mach. Learn., 3 (2):1025-1038._doi:10.54364/AAIML.2023.1161

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, replace = TRUE), 
predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, replace = TRUE),
predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE)    )

# Get P4-metric estimate for two-class case
p4(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

Description

It estimates the contribution of the proportional bias to the Mean Squared Error (MSE) following Correndo et al. (2021).

Usage

PAB(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The PAB represents contribution of additive bias () component of the MSE. The PAB = 100*((mO-mP)^2/MSE), where mO, and mP represent the mean of observations and predictions, respectively. The greater the value the greater the contribution of additive bias to the prediction error. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
PAB(obs = X, pred = Y)

Description

It estimates the PBE for a continuous predicted-observed dataset following Gupta et al. (1999).

Usage

PBE(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The PBE (%) is useful to identify systematic over or under predictions. It is an unbounded metric. The closer to zero the bias of predictions. Negative values indicate overestimation, while positive values indicate underestimation. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Gupta et al. (1999). Status of automatic calibration for hydrologic models: Comparison with multilevel expert calibration. J. Hydrologic Eng. 4(2): 135-143. doi:10.1061/(ASCE)1084-0699(1999)4:2(135)

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
PBE(obs = X, pred = Y)

Description

It estimates the PLA, the contribution of the systematic error to the Mean Squared Error (MSE) for a continuous predicted-observed dataset following Correndo et al. (2021).

Usage

PLA(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The PLA (%, 0-100) represents the contribution of the Mean Lack of Accuracy (MLA), the systematic (bias) component of the MSE. It is obtained via a symmetric decomposition of the MSE (invariant to predicted-observed orientation). The PLA can be further segregated into percentage additive bias (PAB) and percentage proportional bias (PPB). The greater the value the greater the contribution of systematic error to the MSE. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
PLA(obs = X, pred = Y)

Description

It estimates the PLP, the contribution of the unsystematic error to the Mean Squared Error (MSE) for a continuous predicted-observed dataset following Correndo et al. (2021).

Usage

PLP(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The PLP (%, 0-100) represents the contribution of the Mean Lack of Precision (MLP), the unsystematic (random) component of the MSE. It is obtained via a symmetric decomposition of the MSE (invariant to predicted-observed orientation). The greater the value the greater the contribution of unsystematic error to the MSE. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
PLP(obs = X, pred = Y)

Description

It estimates the contribution of the proportional bias to the Mean Squared Error (MSE) following Correndo et al. (2021).

Usage

PPB(data = NULL, obs, pred, tidy = FALSE, na.rm = TRUE)

Arguments

Details

The PPB (%) measures the contribution of proportional bias to the MSE. The PPB = 100*(((sO - sP)^2)/MSE), where sO, and sP are the sample variances of observations and predictions, respectively. The greater the value the greater the contribution of proportional bias to the prediction error. For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194. doi:10.1016/j.agsy.2021.103194

Examples

set.seed(1)
X <- rnorm(n = 100, mean = 0, sd = 10)
Y <- X + rnorm(n=100, mean = 0, sd = 3)
PPB(obs = X, pred = Y)

Description

precision estimates the precision (a.k.a. positive predictive value -ppv-) for a nominal/categorical predicted-observed dataset.

ppv estimates the Positive Predictive Value (equivalent to precision) for a nominal/categorical predicted-observed dataset.

FDR estimates the complement of precision (a.k.a. positive predictive value -PPV-) for a nominal/categorical predicted-observed dataset.

Usage

precision(
  data = NULL,
  obs,
  pred,
  tidy = FALSE,
  atom = FALSE,
  na.rm = TRUE,
  pos_level = 2
)

ppv(
  data = NULL,
  obs,
  pred,
  tidy = FALSE,
  atom = FALSE,
  na.rm = TRUE,
  pos_level = 2
)

FDR(
  data = NULL,
  obs,
  pred,
  atom = FALSE,
  pos_level = 2,
  tidy = FALSE,
  na.rm = TRUE
)

Arguments

Details

The precision is a non-normalized coefficient that represents the ratio between the correctly predicted cases (or true positive -TP- for binary cases) to the total predicted observations for a given class (or total predicted positive -PP- for binary cases) or at overall level.

For binomial cases, $precision = \frac{TP}{PP} = \frac{TP}{TP + FP}$

The precision metric is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low precision of predictions. It can be estimated for each particular class or at a global level.

The false detection rate or false discovery rate (FDR) represents the proportion of false positives with respect to the total number of cases predicted as positive.

For binomial cases, $FDR = 1 - precision = \frac{FP}{PP} = \frac{FP}{TP + FP}$

The precision metric is bounded between 0 and 1. The closer to 1 the better. Values towards zero indicate low precision of predictions.

For the formula and more details, see online-documentation

Value

an object of class numeric within a list (if tidy = FALSE) or within a ⁠data frame⁠ (if tidy = TRUE).

References

Ting K.M. (2017) Precision and Recall. In: Sammut C., Webb G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA. doi:10.1007/978-1-4899-7687-1_659

Examples

set.seed(123)
# Two-class
binomial_case <- data.frame(labels = sample(c("True","False"), 100, 
replace = TRUE), predictions = sample(c("True","False"), 100, replace = TRUE))
# Multi-class
multinomial_case <- data.frame(labels = sample(c("Red","Blue", "Green"), 100, 
replace = TRUE), predictions = sample(c("Red","Blue", "Green"), 100, replace = TRUE))

# Get precision estimate for two-class case
precision(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get FDR estimate for two-class case
FDR(data = binomial_case, obs = labels, pred = predictions, tidy = TRUE)

# Get precision estimate for each class for the multi-class case
precision(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE, atom = TRUE)

# Get precision estimate for the multi-class case at a global level
precision(data = multinomial_case, obs = labels, pred = predictions, tidy = TRUE, atom = TRUE)