---
title: "04 — Data preparation, routing checks and localization"
author: "Pierpaolo Massoli"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{04 — Data preparation, routing checks and localization}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

> Operational examples assume that the SSPICE project root is the working directory. Operational chunks are not executed when building the vignette because they require the project data. Standalone chunks explicitly marked `eval=TRUE` are exceptions.

# Preparing and checking data

Step 02 prepares questionnaire data and auxiliary variables; Step 03 checks
the prepared data against the canonical routing DAG. A missing response may
be valid for an inactive question and invalid for an active question: the
diagnosis depends on the graph structure and observed response categories.

```{r routing-steps}
source(file.path("scripts", "02_prepare_data.R"))
source(file.path("scripts", "03_run_routing_check.R"))
```

`03_run_routing_check.R` saves `routing_check_results.rds`, `full_report.csv`,
`summary_by_record.csv`, `summary_by_variable.csv` and `execution_time.csv`
in `output/routing-check/`. Use the detailed report to identify the variables
involved and the summaries to investigate systematic metadata errors.

# Validating localization

`06_validate_violation_localization.R` compares localized violations with a
reference report when one is available in the configuration. Comparing results
with known truth is distinct from simply counting detected errors.

```{r localization}
source(file.path("scripts", "06_validate_violation_localization.R"))
```

Before attributing a violation to a record, inspect the item mapping and
external sources used by auxiliary variables: an inconsistent rule can
produce many apparent errors.

# Representation of prepared data

Retain the original questionnaire data. Preparation may replace OR/XOR members
with canonical nodes and convert text columns into presence indicators.
Matching the column names to the DAG is necessary, but does not verify that
identifier and code semantics have been preserved. See
[Building the routing file](routing-02b.html) for these conventions.

A ground-truth report is usable only when it refers to the actual records and
violations in the analysed dataset. A matching key-column name is not enough.
Do not associate synthetic violation ground truth with the real questionnaire
data. Without a consistent reference, report localization results without
supervised accuracy metrics.

# Reading the check object

`routing_check()` returns `full_report`, `summary_by_rec` and
`summary_by_var`. The full report contains one row per reported violation,
including the record key, `varname_label`, `routevar_label`, `errcode` and
`message`. When logical edits are requested, `violation_source` distinguishes
ROUTING from LOGICAL_EDIT. This column is not added in routing-only mode.

The record summary groups by identifier and message. Therefore its row count
is not necessarily the number of inconsistent records. Count distinct keys
in `full_report` instead. Consistent records are absent from these violation
summaries. See the overview's standalone example for a complete call.

The current checker requires a mapping CSV even when no recoding is needed;
a header-only file with `varname,item_route,item_dataset` is sufficient.
Numeric IDs must remain unique and unchanged throughout preparation.