--- title: "Getting started with misskappa" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting started with misskappa} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r} #| include: false knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` `misskappa` estimates weighted agreement coefficients (Conger, Fleiss, Brennan–Prediger) when some ratings are **missing**, with standard errors, confidence intervals, and equality tests. ```{r} #| message: false library(misskappa) ``` ## A first agreement coefficient `dat.gwet2014` is a small rating matrix (20 subjects, five raters, four ordered categories) with some ratings missing (`NA`). The default estimator, inverse-probability weighting (`"ipw"`), is consistent when ratings are missing completely at random (MCAR) and is the recommended starting point: ```{r} #| warning: false fit <- agreement(dat.gwet2014, weight = "linear") fit ``` Ratings can be numbers, strings, or factors. `categories = 0:4` would declare a fifth, unused category, which changes Brennan–Prediger and the linear weights. The print method shows each coefficient with its standard error and a 95% confidence interval. The pieces are available directly: ```{r} coef(fit) sqrt(diag(vcov(fit))) ``` ### Confidence intervals `confint()` gives natural-scale Wald intervals, or a variance-stabilizing Fisher *z* interval that always stays inside `(-1, 1)`, useful when a coefficient is close to the boundary: ```{r} confint(fit) confint(fit, transform = "fisher") ``` On complete data with few subjects, `confint(fit, method = "jackknife")` gives exact delete-one jackknife intervals with a *t* critical value instead. `?confint.misskappa_estimate` lists the fits that support it. For a tidy summary, `as.data.frame(fit)` returns one row per coefficient with its estimate, standard error, and interval. ## Choosing an estimator Start with `"ipw"`. The others suit specific situations: | Function and data | `estimator` | Assumption | |---|---|---| | `agreement()`, categorical ratings | `"ipw"` | MCAR, rater-varying observation rates | | `agreement()`, categorical ratings | `"cat_fiml"` | Ignorable (MCAR or MAR), saturated multinomial | | `agreement()`, scored ratings | `"pairwise"` | MCAR, distribution-free, saturated covariance | | `agreement()`, scored ratings | `"nt_fiml"` | Ignorable, normal-theory, saturated covariance | | `agreement()`, scored ratings | `"nt_fiml_cs"` | Ignorable, normal-theory, compound-symmetry covariance | | `agreement_counts()`, category counts | `"pairwise"` | MCAR, exchangeable raters | | `agreement_counts()`, category counts | `"cat_fiml"` | Ignorable, exchangeable raters | | `agreement_counts()`, category counts | `"nt_fiml_cs"` | Ignorable, exchangeable raters, normal-theory | Use `"cat_fiml"` when missingness may depend on the observed ratings (MAR) and there are few raters and categories. The scored estimators treat the categories as numbers and use the quadratic weight. They report Brennan–Prediger only when the scale is declared: ```{r} agreement(dat.zapf2016, estimator = "pairwise") agreement(dat.zapf2016, estimator = "nt_fiml", categories = 1:5) ``` `"nt_fiml_cs"` assumes compound symmetry, which lets it work on planned-incomplete designs where some rater pairs never rate the same subject. The [estimators article](https://jonasmoss.github.io/misskappa/articles/missingness-estimators.html) explains the trade-offs and works through such a design. ## Counts-format data When raters are interchangeable and you only have per-subject category counts, use `agreement_counts()`. `dat.fleiss1971` is the classic psychiatric-diagnosis table (30 subjects, six raters, five categories): ```{r} agreement_counts(dat.fleiss1971) ``` `ratings_to_counts()` turns a subjects-by-raters matrix into this format. ## Beyond pairwise agreement Set `g > 2` for kernels that compare `g` raters at once, or pass a subjects-by-raters-by-features array for vector-valued ratings. The [beyond pairwise agreement](https://jonasmoss.github.io/misskappa/articles/beyond-pairwise.html) article covers both. ## Comparing coefficients Every fit carries a per-subject influence matrix `fit$psi` with `vcov == crossprod(psi) / n^2`. That is what a Wald test needs, so `agreement_test()` compares a coefficient across fits and returns an `htest`. Only the covariance of the contrast changes between designs: * **one sample**: the coefficient's own variance * **independent samples**: the per-fit variances add (block diagonal) * **paired / same-subject**: the cross-fit dependence is read from the stacked influence functions `fit$psi` ### One sample: is a coefficient equal to a fixed value? Pass a single fit and a `value`. This tests whether the (nominal) Conger coefficient for the Gwet (2014) data equals 0.5. Conger is the first coefficient of a `agreement()` fit, so it is the default `coef`: ```{r} agreement_test(agreement(dat.gwet2014, estimator = "ipw"), value = 0.5) ``` ### Independent samples: two studies, different subjects When the fits come from disjoint sets of subjects the variances add. This is the default (`paired = FALSE`). Here the Conger coefficient is compared between the Gwet (2014) and Klein (2018) studies: ```{r} agreement_test( gwet = agreement(dat.gwet2014, estimator = "ipw"), klein = agreement(dat.klein2018, estimator = "ipw") ) ``` ### Paired: same subjects, two coefficients When both coefficients are computed on the *same* subjects, the two estimates are correlated and ignoring that dependence gives the wrong standard error. `paired = TRUE` takes the dependence from the influence functions. A clean illustration on **real crowdsourced data**: in `dat.mcduff2019`, MTurk judges rate smiling images on two attributes (*smile presence* and *image positivity*). Each image is rated on a 1-6 scale by only a handful of the 121 judges, and almost no two judges share images, so the judges are treated as exchangeable and each image is summarised by its category counts. Because both coefficients are estimated on the same items, the contrast is dependent. ```{r} to_counts <- function(attribute) { m <- with(dat.mcduff2019, tapply(get(attribute), list(item, judge), function(z) z[1])) ratings_to_counts(matrix(as.numeric(m), nrow = nrow(m))) } smile <- agreement_counts(to_counts("rating_smile"), weight = "quadratic") positive <- agreement_counts(to_counts("rating_positive"), weight = "quadratic") agreement_test(smile = smile, positive = positive, paired = TRUE) ``` ### *G*-way homogeneity: are several coefficients all equal? With more than two fits the test becomes the joint hypothesis that they are all equal, with `G - 1` degrees of freedom. Is the two-rater quadratic coefficient constant across all rater pairs? The Zapf (2016) data has four raters and hence six pairs, all on the same 50 subjects: ```{r} pairs <- combn(4, 2) pair_fits <- apply(pairs, 2, function(rs) agreement(dat.zapf2016[, rs], estimator = "pairwise"), simplify = FALSE) names(pair_fits) <- apply(pairs, 2, paste0, collapse = "") do.call(agreement_test, c(pair_fits, list(paired = TRUE))) ``` ### Custom contrasts `agreement_test()` covers equality of one coefficient. For anything else, build the test from the influence rows: a linear contrast of coefficients has influence function equal to the same linear combination of the columns of `psi` (`stats::influence(fit)` returns the same matrix). Are Conger and Brennan-Prediger equal within one fit? ```{r} fit <- agreement(dat.tenhove2025, estimator = "nt_fiml_cs", categories = 1:5) psi <- influence(fit) est <- coef(fit)[["Conger"]] - coef(fit)[["Brennan-Prediger"]] se <- sqrt(sum((psi[, "Conger"] - psi[, "Brennan-Prediger"])^2)) / nrow(psi) c(estimate = est, se = se, z = est / se) ``` For fits on the same subjects, subtract their influence columns row by row in the same way; the rows are named by subject.