Active Learning with conflibertR

Labeling text is expensive. Active learning lets you focus annotation effort on the examples a model is most uncertain about, so each label has maximum impact on performance. conflibertR exposes this as a small, intuitive loop:

Start from a tiny labeled seed and an unlabeled pool.
Train a model on the seed; the package hands back the most uncertain samples from the pool.
Label those samples, submit them, and the model retrains and picks the next uncertain batch.
Repeat until the pool is exhausted or metrics plateau.
Save the final model as a HuggingFace checkpoint.

library(conflibertR)

Example data

The package bundles a small demo dataset: a 20-text labeled seed, a 61-text unlabeled pool, and a dev set. It also includes oracle labels for the pool so you can simulate a full loop without a human in the loop (for testing only; don’t use the oracle in real workflows).

demo <- conflibert_example("active")

nrow(demo$seed)      # 20 labeled seed texts
length(demo$pool)    # 61 unlabeled pool texts
nrow(demo$dev)       # 20 dev texts
length(demo$pool_labels)  # oracle labels (for simulation)

Starting a session

conflibert_active_start() trains a classifier on the seed and returns a session object containing the first uncertain batch. Each round, pass the session to conflibert_active_next() along with your labels.

session <- conflibert_active_start(
  seed       = demo$seed,
  pool       = demo$pool,
  dev        = demo$dev,
  model      = "ConfliBERT",
  task       = "binary",
  strategy   = "entropy",   # or "margin", "least_confidence"
  query_size = 10,
  epochs     = 1
)

session

The session’s $query is a tibble of texts to label next, with an uncertainty column showing how unsure the model is. $metrics tracks scores across rounds; $labeled_n / $pool_n track progress.

Labeling and iterating

For real labeling, the easiest route is the built-in Shiny gadget. It opens a modal dialog (or browser tab) showing every row of the current query with radio buttons for each class: click, submit, done.

labels  <- conflibert_active_label(session)
session <- conflibert_active_next(session, labels = labels)

You can also provide the labels by hand (useful for scripting, or if you prefer a console-only workflow):

# labels in the same order as session$query
labels  <- c(1, 0, 1, 0, 0, 1, 0, 1, 0, 1)
session <- conflibert_active_next(session, labels = labels)

For this vignette we use the bundled oracle to simulate labeling:

labels  <- unname(demo$pool_labels[session$query$text])
session <- conflibert_active_next(session, labels = labels)
session

Repeat until the pool is exhausted or the learning curve flattens. Here’s a short simulation loop:

for (round in 2:5) {
  if (session$done) break
  labels  <- unname(demo$pool_labels[session$query$text])
  session <- conflibert_active_next(session, labels = labels)
}

session$metrics

Visualizing progress

plot() produces a two-panel diagnostic: the learning curve on top (metrics vs labeled-set size) and the query uncertainty trend on the bottom. When mean uncertainty flattens, the model is no longer finding informative samples, a good signal to stop.

plot(session)

# or a single panel:
plot(session, which = "metrics")
plot(session, which = "uncertainty")

Query strategies

Three uncertainty strategies are available. Pass one via strategy:

"entropy" (default): highest Shannon entropy of the predicted class distribution. Works well for both binary and multiclass.
"margin": smallest gap between the top two class probabilities. Targets ambiguous samples on the decision boundary.
"least_confidence": lowest maximum class probability. Simplest strategy; a good baseline.

Diversity-aware batches

Pure uncertainty sampling can pick several near-duplicates in one batch, a problem when your pool has many similar texts. Pass diverse = TRUE to cluster the top-scoring candidates in the model’s embedding space and pick the highest-scoring sample from each cluster:

session <- conflibert_active_start(
  seed = demo$seed, pool = demo$pool, dev = demo$dev,
  strategy = "entropy",
  diverse  = TRUE,
  diversity_candidates = 30    # defaults to 3 * query_size
)

LoRA fine-tuning

For bigger base models or tighter GPU budgets, train only a low-rank adapter each round. The adapter is merged into the base model before every round ends, so scoring, saving, and reloading behave exactly like full fine-tuning:

session <- conflibert_active_start(
  seed = demo$seed, pool = demo$pool, dev = demo$dev,
  model      = "DeBERTa v3 Base",
  use_lora   = TRUE,
  lora_rank  = 8,
  lora_alpha = 16
)

Saving the model

Persist the final model as a standard HuggingFace checkpoint:

conflibert_active_save(session, "my_al_model")

You can reload it with any transformers tool, or point AutoModelForSequenceClassification.from_pretrained() at the directory from Python.

Tips

Start small. A seed of 10–50 texts is often enough; active learning shines when the pool is much larger than the labeled set.
Watch uncertainty trends, not just metrics. If dev metrics are noisy on small sets, the uncertainty panel often tells a clearer story about whether the model is still learning.
Parameters carry across rounds. The model, task, strategy, query size, and training hyperparameters are fixed when you call conflibert_active_start() and reused for every subsequent round.
Sessions are in-memory. The trained model lives inside the session object as a Python handle. Calling saveRDS() on the session won’t serialize the model; use conflibert_active_save() to persist it, and re-run rounds from a fresh session if needed.