--- title: "Weighted Chance-corrected Agreement Coefficients" author: "Kilem L. Gwet, Ph.D." date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{irrCAC: Computing weighted agreement coefficients} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ```{r setup} library(irrCAC) ``` # Abstract The **irrCAC** is an R package that provides several functions for calculating various chance-corrected agreement coefficients. This package closely follows the general framework of inter-rater reliability assessment presented by Gwet (2014). The document [overview.html](overview.html) provides a broad overview of the different ways you may compute various agreement coefficients using the irrCAC package. However, it is common for researchers to want to compute the weighted versions of agreement coefficients in order to account for the ordinal nature of some ratings. This document shows you how to specify the weights to comnpute the weighted kappa coefficient, the weighted $\mbox{AC}_2$ coefficient and many others. # # The different weights A set of weights to be applied to an agreement is generally defined in the form a matrix (i.e. a square table) whose dimension is defined par the number of categories into which the rater can classify each subject. Although the user can define her own custom weights, the inter-rater reliability literature offers predefined weights, which can be generated using functions in this package as follows: ```{r} identity.weights(1:3) radical.weights(1:3) linear.weights(1:3) ordinal.weights(1:3) quadratic.weights(1:3) circular.weights(1:3) bipolar.weights(1:3) ``` You may need to read chapter 3 of Gwet (2014) for a more detailed discussion of weighted agreement coefficients. Identifty weights yield unweighted coefficients and the most commonly-used weights are the quadratic and linear weights. # # Weighted Agreement Coefficients I will show here how to compute weighted agreement coefficients separately for each of the 3 ways of organizing your input ratings that are covered in this package. 1. *The contingency table*: this dataset describes ratings from 2 raters only in such a way that each row represents a rating used by rater 1 and each column a rating used by rater 2 (see **cont3x3abstractors**) for an example of such a dataset). 1. *The dataset of raw ratings*: this dataset is a listing of all ratings assigned each individual subject by each rater. The row represents one subject, the column represents one rater and each value is the rating that the rater assigned to the subject (you may view the package dataset **cac.raw4raters** as an example). 1. *The distribution of raters*: each row of this dataset represents a subject, each column a rating used by any of the raters and each dataset value is the the number of raters who assigned the specified rating to the subject (you may see **distrib.6raters** as an example) ## Weighting ratings from a contingency table Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset *cont3x3abstractors*. Using the *quadratic.weights* function presented above, you would proceed as follows: ```{r} cont3x3abstractors q <- nrow(cont3x3abstractors) kappa2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) scott2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) gwet.ac1.table(cont3x3abstractors,weights = quadratic.weights(1:q)) bp2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) krippen2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) pa2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) ``` * You may use any of the weight generating functions previously discusssed to change your weights. * You may also supply your own custom set of weights to these functions. No need to stick to the weights recommended in the inter-rater reliability literature. ## Weighting for a dataset of raw ratings Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset cac.raw4raters. You would proceed as follows: ```{r} pa.coeff.raw(cac.raw4raters,weights = "quadratic")$est gwet.ac1.raw(cac.raw4raters,weights = "quadratic")$est fleiss.kappa.raw(cac.raw4raters,weights = "quadratic")$est krippen.alpha.raw(cac.raw4raters,weights = "quadratic")$est conger.kappa.raw(cac.raw4raters,weights = "quadratic")$est bp.coeff.raw(cac.raw4raters,weights = "quadratic")$est ``` * If you want to use a set of weights other than quadratic weights you may replace "quadratic" with one of the following strings of characters: "ordinal", "linear", "radical", "ratio", "circular", or "bipolar". * You may also supply your own custom set of weights to these functions. In this case, the value of the **weight** parameter will not be a string of characters. Instead, it will be an actual matrix containing the weight values you have decided to use. ## Weighting when input data is the distribution of raters by subject and rating The calculation of weighted agreement coefficients for this type of datasets is very similar to the calculation of agreement coefficients with contingency tables. You need to use the weight-generating functions such as the *quadratic.weights()* fucntion to which the vector of ratings is supplied as parameter. As an example, suppose you want to compute various weighted agreement coefficients using quadratic weights and the *distrib.6raters* dataset. This would be accomplished as follows: ```{r} q <- ncol(distrib.6raters) gwet.ac1.dist(distrib.6raters,weights = quadratic.weights(1:q)) fleiss.kappa.dist(distrib.6raters,weights = quadratic.weights(1:q)) krippen.alpha.dist(distrib.6raters,weights = quadratic.weights(1:q)) bp.coeff.dist(distrib.6raters,weights = quadratic.weights(1:q)) ``` # References: 1. Gwet, K.L. (2014, [ISBN:978-0970806284](https://www.amazon.com/Handbook-Inter-Rater-Reliability-Definitive-Measuring/dp/0970806280/)). "*Handbook of Inter-Rater Reliability*," 4th Edition. Advanced Analytics, LLC