Getting started with misskappa

misskappa estimates weighted agreement coefficients (Conger, Fleiss, Brennan–Prediger) when some ratings are missing, with standard errors, confidence intervals, and equality tests.

library(misskappa)

A first agreement coefficient

dat.gwet2014 is a small rating matrix (20 subjects, five raters, four ordered categories) with some ratings missing (NA). The default estimator, inverse-probability weighting ("ipw"), is consistent when ratings are missing completely at random (MCAR) and is the recommended starting point:

fit <- agreement(dat.gwet2014, weight = "linear")
fit
#> misskappa: estimator = ipw, weight = linear, n = 20
#>                  estimate     se  lower  upper
#> Conger             0.5879 0.0772 0.4365 0.7393
#> Fleiss             0.5858 0.0774 0.4341 0.7375
#> Brennan-Prediger   0.6397 0.0767 0.4894 0.7900

Ratings can be numbers, strings, or factors. categories = 0:4 would declare a fifth, unused category, which changes Brennan–Prediger and the linear weights. The print method shows each coefficient with its standard error and a 95% confidence interval. The pieces are available directly:

coef(fit)
#>           Conger           Fleiss Brennan-Prediger 
#>        0.5879222        0.5858274        0.6397286
sqrt(diag(vcov(fit)))
#>           Conger           Fleiss Brennan-Prediger 
#>       0.07724798       0.07739332       0.07668414

Confidence intervals

confint() gives natural-scale Wald intervals, or a variance-stabilizing Fisher z interval that always stays inside (-1, 1), useful when a coefficient is close to the boundary:

confint(fit)
#>                      2.5 %    97.5 %
#> Conger           0.4365189 0.7393254
#> Fleiss           0.4341393 0.7375156
#> Brennan-Prediger 0.4894304 0.7900267
#> attr(,"method")
#> [1] "wald"
#> attr(,"transform")
#> [1] "none"
#> attr(,"boundary_fallback")
#>           Conger           Fleiss Brennan-Prediger 
#>            FALSE            FALSE            FALSE
confint(fit, transform = "fisher")
#>                      2.5 %    97.5 %
#> Conger           0.4162142 0.7191417
#> Fleiss           0.4139274 0.7173856
#> Brennan-Prediger 0.4647036 0.7666434
#> attr(,"method")
#> [1] "wald"
#> attr(,"transform")
#> [1] "fisher"
#> attr(,"boundary_fallback")
#>           Conger           Fleiss Brennan-Prediger 
#>            FALSE            FALSE            FALSE

On complete data with few subjects, confint(fit, method = "jackknife") gives exact delete-one jackknife intervals with a t critical value instead. ?confint.misskappa_estimate lists the fits that support it. For a tidy summary, as.data.frame(fit) returns one row per coefficient with its estimate, standard error, and interval.

Choosing an estimator

Start with "ipw". The others suit specific situations:

Function and data estimator Assumption
agreement(), categorical ratings "ipw" MCAR, rater-varying observation rates
agreement(), categorical ratings "cat_fiml" Ignorable (MCAR or MAR), saturated multinomial
agreement(), scored ratings "pairwise" MCAR, distribution-free, saturated covariance
agreement(), scored ratings "nt_fiml" Ignorable, normal-theory, saturated covariance
agreement(), scored ratings "nt_fiml_cs" Ignorable, normal-theory, compound-symmetry covariance
agreement_counts(), category counts "pairwise" MCAR, exchangeable raters
agreement_counts(), category counts "cat_fiml" Ignorable, exchangeable raters
agreement_counts(), category counts "nt_fiml_cs" Ignorable, exchangeable raters, normal-theory

Use "cat_fiml" when missingness may depend on the observed ratings (MAR) and there are few raters and categories. The scored estimators treat the categories as numbers and use the quadratic weight. They report Brennan–Prediger only when the scale is declared:

agreement(dat.zapf2016, estimator = "pairwise")
#> misskappa: estimator = pairwise, weight = quadratic, n = 50
#>        estimate     se  lower upper
#> Conger   0.8985 0.0278 0.8439 0.953
#> Fleiss   0.8984 0.0279 0.8437 0.953
agreement(dat.zapf2016, estimator = "nt_fiml", categories = 1:5)
#> misskappa: estimator = nt_fiml, weight = quadratic, n = 50
#>                  estimate     se  lower  upper
#> Conger             0.8985 0.0278 0.8439 0.9530
#> Fleiss             0.8984 0.0279 0.8437 0.9530
#> Brennan-Prediger   0.8675 0.0313 0.8062 0.9288

"nt_fiml_cs" assumes compound symmetry, which lets it work on planned-incomplete designs where some rater pairs never rate the same subject. The estimators article explains the trade-offs and works through such a design.

Counts-format data

When raters are interchangeable and you only have per-subject category counts, use agreement_counts(). dat.fleiss1971 is the classic psychiatric-diagnosis table (30 subjects, six raters, five categories):

agreement_counts(dat.fleiss1971)
#> misskappa: estimator = pairwise, weight = nominal, n = 30
#>                  estimate     se  lower  upper
#> Fleiss             0.4302 0.0533 0.3258 0.5347
#> Brennan-Prediger   0.4444 0.0542 0.3382 0.5507

ratings_to_counts() turns a subjects-by-raters matrix into this format.

Beyond pairwise agreement

Set g > 2 for kernels that compare g raters at once, or pass a subjects-by-raters-by-features array for vector-valued ratings. The beyond pairwise agreement article covers both.

Comparing coefficients

Every fit carries a per-subject influence matrix fit$psi with vcov == crossprod(psi) / n^2. That is what a Wald test needs, so agreement_test() compares a coefficient across fits and returns an htest. Only the covariance of the contrast changes between designs:

  • one sample: the coefficient’s own variance
  • independent samples: the per-fit variances add (block diagonal)
  • paired / same-subject: the cross-fit dependence is read from the stacked influence functions fit$psi

One sample: is a coefficient equal to a fixed value?

Pass a single fit and a value. This tests whether the (nominal) Conger coefficient for the Gwet (2014) data equals 0.5. Conger is the first coefficient of a agreement() fit, so it is the default coef:

agreement_test(agreement(dat.gwet2014, estimator = "ipw"), value = 0.5)
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
#> 
#>  One-sample Wald test of Conger
#> 
#> data:  fit1
#> X-squared = 0.46985, df = 1, p-value = 0.4931
#> alternative hypothesis: true Conger is not equal to 0.5
#> sample estimates:
#>      fit1 
#> 0.4233051

Independent samples: two studies, different subjects

When the fits come from disjoint sets of subjects the variances add. This is the default (paired = FALSE). Here the Conger coefficient is compared between the Gwet (2014) and Klein (2018) studies:

agreement_test(
  gwet  = agreement(dat.gwet2014,  estimator = "ipw"),
  klein = agreement(dat.klein2018, estimator = "ipw")
)
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
#> 
#>  Independent-sample Wald test of equal Conger across 2 fits
#> 
#> data:  gwet, klein
#> X-squared = 0.0019646, df = 1, p-value = 0.9646
#> alternative hypothesis: true difference in Conger is not equal to 0
#> sample estimates:
#>      gwet     klein 
#> 0.4233051 0.4301049

Paired: same subjects, two coefficients

When both coefficients are computed on the same subjects, the two estimates are correlated and ignoring that dependence gives the wrong standard error. paired = TRUE takes the dependence from the influence functions.

A clean illustration on real crowdsourced data: in dat.mcduff2019, MTurk judges rate smiling images on two attributes (smile presence and image positivity). Each image is rated on a 1-6 scale by only a handful of the 121 judges, and almost no two judges share images, so the judges are treated as exchangeable and each image is summarised by its category counts. Because both coefficients are estimated on the same items, the contrast is dependent.

to_counts <- function(attribute) {
  m <- with(dat.mcduff2019,
            tapply(get(attribute), list(item, judge), function(z) z[1]))
  ratings_to_counts(matrix(as.numeric(m), nrow = nrow(m)))
}
smile    <- agreement_counts(to_counts("rating_smile"),    weight = "quadratic")
positive <- agreement_counts(to_counts("rating_positive"), weight = "quadratic")

agreement_test(smile = smile, positive = positive, paired = TRUE)
#> 
#>  Paired Wald test of equal Fleiss across 2 fits
#> 
#> data:  smile, positive
#> X-squared = 0.013445, df = 1, p-value = 0.9077
#> alternative hypothesis: true difference in Fleiss is not equal to 0
#> sample estimates:
#>     smile  positive 
#> 0.6421397 0.6451303

G-way homogeneity: are several coefficients all equal?

With more than two fits the test becomes the joint hypothesis that they are all equal, with G - 1 degrees of freedom.

Is the two-rater quadratic coefficient constant across all rater pairs? The Zapf (2016) data has four raters and hence six pairs, all on the same 50 subjects:

pairs <- combn(4, 2)
pair_fits <- apply(pairs, 2, function(rs)
  agreement(dat.zapf2016[, rs], estimator = "pairwise"), simplify = FALSE)
names(pair_fits) <- apply(pairs, 2, paste0, collapse = "")
do.call(agreement_test, c(pair_fits, list(paired = TRUE)))
#> 
#>  Paired Wald test of equal Conger across 6 fits
#> 
#> data:  12, 13, 14, 23, 24, 34
#> X-squared = 5.4178, df = 5, p-value = 0.367
#> sample estimates:
#>        12        13        14        23        24        34 
#> 0.9309091 0.8811545 0.9012694 0.8786611 0.9403509 0.8466753

Custom contrasts

agreement_test() covers equality of one coefficient. For anything else, build the test from the influence rows: a linear contrast of coefficients has influence function equal to the same linear combination of the columns of psi (stats::influence(fit) returns the same matrix). Are Conger and Brennan-Prediger equal within one fit?

fit <- agreement(dat.tenhove2025, estimator = "nt_fiml_cs", categories = 1:5)
psi <- influence(fit)
est <- coef(fit)[["Conger"]] - coef(fit)[["Brennan-Prediger"]]
se  <- sqrt(sum((psi[, "Conger"] - psi[, "Brennan-Prediger"])^2)) / nrow(psi)
c(estimate = est, se = se, z = est / se)
#>    estimate          se           z 
#> -0.12743311  0.03771282 -3.37903988

For fits on the same subjects, subtract their influence columns row by row in the same way; the rows are named by subject.