---
title: "Getting Started with synadam"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with synadam}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Introduction

`synadam` generates synthetic ADaM (Analysis Data Model) datasets from real clinical
trial data, removing identifiable patient information while preserving the structure
of the original datasets.

**Important:** The synthetic data produced by `synadam` is **not realistic**. Values
are generated to match the structural characteristics and ranges of the original data
but do not preserve statistical distributions. This package is designed for use cases
where exact values do not matter, such as:

- Testing the generation of clinical tables.
- Validating data processing pipelines.
- Developing and debugging analysis code without access to real patient data.

## High-level design

`synadam` uses a two-phase "glimpse-then-simulate" approach:

1. **Glimpse**: Extract statistical summaries from real data (ranges, unique values,
   NA patterns).
2. **Simulate**: Generate synthetic data from those summaries.

This provides a clear, traceable interface between real and synthetic data, allowing
users and reviewers to inspect exactly what information is retained.

```{r, echo=FALSE, out.width="100%", fig.align="center"}
if (file.exists("images/synadam_overview.png")) {
  knitr::include_graphics("images/synadam_overview.png")
}
```

## Supported dataset types

| Type | Description | Example |
|------|-------------|---------|
| **ADSL** | Subject-level data (one row per subject) | Demographics, treatment assignments |
| **BDS** | Basic Data Structure (longitudinal) | Lab results (ADLB), vitals (ADVS) |
| **OCCDS** | Occurrence Data Structure (event-based) | Adverse events (ADAE), concomitant meds |
| **TTE** | Time-to-Event (survival analysis) | Overall survival (ADTTE) |

## End-to-end example

This example uses sample data inspired by the
[PHUSE CDISC Pilot Study](https://github.com/phuse-org/phuse-scripts), included
in `inst/extdata/` for demonstration.

```{r setup}
library(synadam)
```

```{r prepare-sas-files, include=FALSE}
# Convert sample CSVs to SAS format in a temp directory for the demo
temp_adam_dir <- file.path(tempdir(), "adam_data")
dir.create(temp_adam_dir, showWarnings = FALSE)
temp_output_dir <- file.path(tempdir(), "syn_output")

adsl <- read.csv(
  system.file("extdata", "adsl.csv", package = "synadam"),
  stringsAsFactors = FALSE
)
adlb <- read.csv(
  system.file("extdata", "adlb.csv", package = "synadam"),
  stringsAsFactors = FALSE
)
adae <- read.csv(
  system.file("extdata", "adae.csv", package = "synadam"),
  stringsAsFactors = FALSE
)

# Convert date columns
adsl$TRTSDT <- as.Date(adsl$TRTSDT)
adlb$ADT <- as.Date(adlb$ADT)
adae$ASTDT <- as.Date(adae$ASTDT)
adae$AENDT <- as.Date(adae$AENDT)

haven::write_sas(adsl, file.path(temp_adam_dir, "adsl.sas7bdat"))
haven::write_sas(adlb, file.path(temp_adam_dir, "adlb.sas7bdat"))
haven::write_sas(adae, file.path(temp_adam_dir, "adae.sas7bdat"))
```

### Step 1: Auto-generate a configuration file

Point `generate_study_config()` at a directory containing your `.sas7bdat` files.
It scans the files, infers dataset types and column roles using CDISC conventions,
and writes a draft YAML configuration:
 
```{r generate-config}
yaml_path <- synadam::generate_study_config(
  adam_dir = temp_adam_dir,
  output_dir = temp_output_dir,
  seed = 42
)
```

The generated YAML uses heuristics to classify columns into their roles. Fields with
weaker signal are tagged with inline `# REVIEW` comments so you know what to verify:

```{r show-config}
cat(readLines(yaml_path), sep = "\n")
```

### Step 2: Review and edit the configuration

Before running the simulation, review the generated YAML:

- Verify `dataset_type` is correct for each dataset.
- Check that `treatment_cols`, `flag_cols`, and `ordered_col_sets` are appropriate.
- Remove any `flag_cols` you don't need to preserve.
- Add any `ordered_col_sets` not auto-detected (e.g., `AEBODSYS`/`AEDECOD`).

### Step 3: Simulate the study

Once satisfied with the configuration, generate all synthetic datasets:

```{r simulate-study}
synadam::simulate_study(config_path = yaml_path)
```

Synthetic datasets are saved as individual `.rds` files in the output directory:

```{r read-results}
list.files(temp_output_dir, pattern = "\\.rds$")

syn_adsl <- readRDS(file.path(temp_output_dir, "syn_adsl.rds"))
head(syn_adsl)

# Each dataset carries a version attribute for traceability
attr(syn_adsl, "synadam_version")
```

## Writing the YAML configuration manually

If you prefer full control (or need to fine-tune the auto-generated draft), author
the YAML by hand. The schema supports all four dataset types:

```yaml
output_dir: "/path/to/output_directory"
seed: 42

datasets:
  adsl:
    dataset_type: "adsl"
    path: "/path/to/adsl.sas7bdat"
    id_cols: [USUBJID, SUBJID]
    treatment_cols: [TRT01A, TRT01AN]
    flag_cols: [SAFFL, ITTFL, EFFFL]         # Optional
    ordered_col_sets: [[REGION1, REGION1N]]   # Optional

  adlb:
    dataset_type: "bds"
    path: "/path/to/adlb.sas7bdat"
    id_cols: [USUBJID]
    param_cols: [PARAM, PARAMCD]
    visit_cols: [AVISIT, AVISITN]            # Optional
    flag_cols: [ANL01FL]                     # Optional

  adae:
    dataset_type: "occds"
    path: "/path/to/adae.sas7bdat"
    id_cols: [USUBJID]
    seq_col: AESEQ
    ordered_col_sets: [[AEBODSYS, AEDECOD]]  # Optional

  adtte:
    dataset_type: "tte"
    path: "/path/to/adtte.sas7bdat"
    param_cols: [PARAM, PARAMCD]
    censor_cols: [CNSR, EVNTDESC, CNSDTDSC]  # Optional
```

**Note**: Optional fields can be set to `[]` or omitted entirely.

## Granular control: glimpse and simulate individually

For more control, call the `glimpse_*()` and `simulate_*()` functions directly:

```{r individual-adsl}
adsl_data <- synadam::read_sas7bdat(file.path(temp_adam_dir, "adsl.sas7bdat"))

# Glimpse: extract structural summary
adsl_summary <- synadam::glimpse_adsl(
  adsl = adsl_data,
  id_cols = c("USUBJID", "SUBJID"),
  treatment_cols = c("TRT01A", "TRT01AN"),
  flag_cols = c("SAFFL", "ITTFL", "EFFFL"),
  ordered_col_sets = list(c("REGION1", "REGION1N")),
  seed = 42
)

# Simulate: generate synthetic data
syn_adsl <- synadam::simulate_adsl(adsl_summary, seed = 42)
head(syn_adsl)
```

For BDS, OCCDS, and TTE datasets:

- `glimpse_bds()` / `simulate_bds()` — longitudinal data (ADLB, ADVS)
- `glimpse_occds()` / `simulate_occds()` — occurrence data (ADAE, ADCM)
- `glimpse_tte()` / `simulate_tte()` — time-to-event data (ADTTE)

See `?glimpse_bds`, `?glimpse_occds`, `?glimpse_tte` for full documentation.

## Splitting glimpse from simulate

`simulate_study()` runs both phases in one call. To run them separately — for
example, to glimpse on a machine that has access to the real `.sas7bdat` files
and then simulate elsewhere — use `glimpse_study()` and
`simulate_study_from_summary()`.

```{r split-glimpse-simulate}
study_summary_path <- file.path(temp_output_dir, "study_summary.rds")

# Phase 1: glimpse only. Requires the real SAS files.
synadam::glimpse_study(
  config_path        = yaml_path,
  study_summary_path = study_summary_path
)

# The study summary is one .rds with all summaries plus seed and version.
str(readRDS(study_summary_path), max.level = 2)

# Phase 2: simulate from the study summary. Does not need the SAS files.
split_output_dir <- file.path(tempdir(), "syn_output_split")
synadam::simulate_study_from_summary(
  study_summary_path = study_summary_path,
  output_dir         = split_output_dir
)

list.files(split_output_dir, pattern = "\\.rds$")
```

Pass `seed` to `simulate_study_from_summary()` to draw a different synthetic
replicate from the same summaries; omit it to reuse the seed stored in the
study summary.

## Privacy protection

`synadam` implements multiple privacy mechanisms:

1. **Structure, not values**: Preserves column names, unique values, and ranges, but
   not distributions or clinical patterns.
2. **Singleton removal**: Unique values or combinations appearing only once are
   excluded and replaced with more common values.
3. **NA noise**: 5% of NA positions are randomly flipped to prevent exact pattern
   reproduction.

## Limitations

Synthetic data from `synadam` is **not suitable** for:

- **Realistic simulations** — does not preserve statistical distributions.
- **Statistical analysis** — results will not reflect real clinical findings.
- **Regulatory submissions** — unless explicitly approved.

## Disclaimer

⚠️ Users are responsible for correctly specifying all column categories in the
configuration. Incorrect configuration can result in private data leaking into
synthetic data.

- **ID columns** (`id_cols`): Must include all subject identifiers.
- **Ordered column sets** (`ordered_col_sets`): Should include the minimum sets of
  related columns required for your use case.

**Always review your configuration carefully and validate that synthetic data does not
contain identifiable information before sharing outside GxP environments.**