Package 'scindex'

Estimate SDT Decision Thresholds

Description

Internal utility for estimating signal-detection-theoretic decision thresholds from binary ratings and observed truth labels.

Usage

estimate_sdt_thresholds(
  data,
  item_col,
  rater_col,
  rating_col,
  truth_col,
  positive = 1
)

Arguments

data	Standardized long-format data frame.
item_col	Character string identifying the item column.
rater_col	Character string identifying the rater column.
rating_col	Character string identifying the rating column.
truth_col	Character string identifying the truth column.
positive	Positive class value.

Details

For each rater, the function computes:

• hit rate;

• false-alarm rate;

• SDT decision threshold.

Thresholds are estimated as:

$t_i = -\frac{1}{2}[z(H_i) + z(F_i)]$

where:

• $H_i$ is the hit rate;

• $F_i$ is the false-alarm rate;

• $z(\cdot)$ is the inverse standard normal CDF.

A log-linear correction is applied to avoid infinite z-values.

Value

A list containing:

thresholds

Data frame with SDT quantities for each rater.

mean_threshold

Mean threshold across raters.

threshold_variance

Variance of thresholds.

References

DeCarlo, L. T. (1998). Signal detection theory and generalized linear models. Psychological Methods, 3(2), 186–205. https://doi.org/10.1037/1082-989X.3.2.186

Macmillan, N. A., & Creelman, C. D. (2004). Detection theory: A user's guide (2nd ed.). New York, NY: Psychology Press. https://doi.org/10.4324/9781410611147

---

Print a scindex Object

Description

Print method for objects of class "scindex_fit".

Usage

## S3 method for class 'scindex_fit'
print(x, digits = 3, ...)

Arguments

x	An object of class "scindex_fit".
digits	Number of digits to print.
...	Additional arguments (unused).

Details

The method displays:

• agreement coefficient;

• agreement type;

• SCI when available;

• basic sample information.

Value

Invisibly returns x.

---

Compute Agreement and the Strategic Convergence Index

Description

scindex() computes classical inter-rater agreement and, when a ground-truth variable is available, the Strategic Convergence Index (SCI). The function automatically returns Cohen's kappa for two raters and Fleiss' kappa for more than two raters.

Usage

scindex(
  data,
  item,
  rater,
  rating,
  truth = NULL,
  positive = 1,
  variance_bounds = NULL
)

Arguments

data	A data frame containing item identifiers, rater identifiers, ratings, and optionally ground-truth labels.
item	Unquoted column name identifying the rated item or case.
rater	Unquoted column name identifying the rater.
rating	Unquoted column name containing the categorical rating. In the current version, ratings must be binary.
truth	Optional unquoted column name containing the ground-truth label. If supplied, SDT thresholds and SCI are computed.
positive	Value identifying the positive class. Defaults to 1.
variance_bounds	Optional numeric vector of length 2 defining the lower and upper bounds used to compute the reference variance for SCI. If NULL, bounds are derived from the estimated thresholds.

Details

SCI is computed from signal-detection-theoretic decision thresholds and should be interpreted as a model-based index of convergence in rater decision criteria, not as a replacement for agreement coefficients.

Value

An object of class "scindex_fit" containing the agreement coefficient, its type, SDT thresholds when available, SCI when available, and diagnostic information.

Examples

ratings <- data.frame(
  item = rep(1:6, each = 2),
  rater = rep(c("A", "B"), times = 6),
  rating = c(1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1),
  truth = rep(c(1, 0, 1, 1, 0, 1), each = 2)
)

fit <- scindex(
  data = ratings,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit

---

Example Binary Inter-Rater Dataset

Description

Simulated binary classification ratings from two raters across eight items, including ground-truth labels.

Usage

data(scindex_example)

Format

A data frame with 16 rows and 4 variables:

item

Item identifier.

rater

Rater identifier.

rating

Binary rating assigned by the rater.

truth

Ground-truth binary classification.

Details

The dataset illustrates the computation of Cohen's kappa and the Strategic Convergence Index (SCI).

Source

Simulated example data.

Examples

data(scindex_example)

fit <- scindex(
  data = scindex_example,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit
summary(fit)

---

Example Binary Inter-Rater Dataset with Multiple Raters

Description

Simulated binary classification ratings from three raters across eight items, including ground-truth labels.

Usage

data(scindex_example_multi)

Format

A data frame with 24 rows and 4 variables:

item

Item identifier.

rater

Rater identifier.

rating

Binary rating assigned by the rater.

truth

Ground-truth binary classification.

Details

The dataset illustrates the computation of Fleiss' kappa and the Strategic Convergence Index (SCI) in a multi-rater setting.

Source

Simulated example data.

Examples

data(scindex_example_multi)

fit <- scindex(
  data = scindex_example_multi,
  item = item,
  rater = rater,
  rating = rating,
  truth = truth
)

fit
summary(fit)

---

Summarize a scindex Object

Description

Summary method for objects of class "scindex_fit".

Usage

## S3 method for class 'scindex_fit'
summary(object, digits = 3, ...)

Arguments

object	An object of class "scindex_fit".
digits	Number of digits to print.
...	Additional arguments (unused).

Details

The method returns a structured summary including the agreement coefficient, SCI when available, SDT threshold estimates, and basic diagnostic information.

Value

Invisibly returns object.

---