sysctl on macOS, /proc/meminfo on Linux). It now reads the memory
budget straight from the connection's own memory_limit setting, which
defaults to 80% of system RAM and reflects any user override. This removes
the shell call that broke on systems without sysctl on the PATH (the
CRAN M1mac check failures, where batch planning errored with "argument is
of length zero"), and makes the budget detection work identically on all
platforms, including Windows.First stable release. The token core, the DuckDB backend, embedding matching, diagnostics, and calibration are feature-complete and the public API is stable. This release adds the documentation that makes the package usable end to end.
workshop_register, workshop_listings, workshop_panel, match_labels_example: synthetic woodworking-workshop data with planted difficulty tiers and ground-truth links, used throughout the articles. Each tier (containment, movers, phonetic twins, hub tokens) has a minority that measurably benefits from the feature it exercises.Staged entity resolution, region-free linking across blocks, and an always-on cost guard, plus embedding reuse and faster preparers.
Run strategies in order, carry residuals forward, resolve entities once at the end.
multi_stage_dedup() and multi_stage_search(): run an ordered list of strategies as successive passes. multi_stage_dedup() finds duplicates within one table; multi_stage_search() links records across tables, or across years of one pooled table with self = TRUE. Both take a mix of exact, fuzzy, and embedding strategies; multi_stage_search() supports collapse-and-continue so a slowly drifting name links one step at a time. Renames multi_stage_match().exact_strategy(): identical-token-set matching as a strategy, for a cheap first pass. Runs through detect_duplicates() and search_candidates() like any strategy. Optional containment matches a subset rather than an exact set, with a per-column min_containment_tokens floor.resolve_entities(): group an edge list into entities (connected components) and pick a representative per group.materialize_records(): fetch the original rows for a set of ids, the complement of extract_unmatched().plan_strategy(): compare blocking keys before matching. Reports each candidate's block sizes, comparison cost, and how many true twins stay co-blocked, without computing any scores.rarity_distribution(): report a column's token frequency and rarity before matching, with a suggested min_rarity.find_stopwords(): list a column's high-frequency, low-information tokens for filter_stopwords().duckdb_control(): one object for DuckDB execution tuning (batch size, scoring chunk key, per-chunk failure policy, progress), passed as control =. Replaces the loose batch arguments.Follow an entity across geographic blocks (movers, name drift, year to year) without giving up block-based cost control.
block_on_tokens(): block on a record's own rare name tokens instead of a fixed key, so two records sharing any rare token are compared wherever they sit. Mix it with plain column names in block_by.rarity_scope = "global": measure rarity across the whole corpus, so a distinctive name reads as strong evidence in any block and a common one stays weak.max_fanout / on_fanout: an automatic ceiling on comparison cost. When a hot or boilerplate token would fan a dense block into a near-quadratic join, joinery drops the offending tokens with a warning ("cap", the default) or stops ("abort"). On by default. Replaces max_comparisons.joinery.embedding_cache_dir to persist across sessions, or joinery.embedding_reuse = FALSE to opt out.clear_embedding_cache(): empty the cache, optionally on disk too.score_embeddings() scores all pairs in a block as one matrix product, dropping a few hundred thousand pairs from seconds to a fraction of a second.drop_short_tokens(): drop tokens below a length, useful after phonetic encoding.as_cologne(), as_soundex(), as_metaphone(), and as_nysiis() now encode token lists as well as raw strings, so you can encode after tokenizing.normalize_street() gains drop_house_numbers and drop_stopwords to strip address noise.word_tokens(), filter_stopwords(), drop_numeric_tokens(), token_shapes(), and extract_initials() now run group-wise over token tables.search_candidates() rejects overlapping id spaces and prepare_search_data() rejects duplicate ids, both of which corrupt results silently otherwise.resolve_entities() no longer drops singletons when ids mix integer and double forms.summarise_matches() (DuckDB) no longer produces an out-of-range histogram bin for scores just above 1.0.drop_joinery_temp_tables() is now exported.Internal consolidation after the calibration work, plus fixes surfaced by a full-scale Yellow-Pages panel build. Output schemas unchanged.
[0, sum(weights)].tbl |> filter(...)) are accepted everywhere.summarise_matches(entity_cols =): count duplicate groups whose listed columns are single-valued, separating real stopword clusters from cardinality artefacts.cli::cli_abort() with rlang argument checks across exported verbs.R/ reorganised under an eight-prefix naming scheme (see CLAUDE.md).An optional post-match filter that learns to drop false positives from a small labelled sample. The same verb works on token and embedding strategies.
match_features(): build a one-row-per-pair feature table from a match result, with token-overlap counts, auxiliary-side informativeness (aIP, after Doherr 2023), and string similarities.fit_filter() / apply_filter(): fit a logistic false-positive filter and apply it, choosing a threshold by Youden's J unless you set one.calibrate_matches(): one verb composing features, fit, and apply.calibrate(): evaluate a fitted filter on a labelled set; returns reliability, Brier score, log-loss, confusion matrix, and a threshold sweep.sample_matches() stratification, plus export_for_labelling() / import_labels() for a CSV round-trip.fit_filter() via joinery_recipe(). All tidymodels packages are optional; the glm path needs none.Verbs to answer four questions about a strategy and its results: will it work, did it work, why this pair, and where to look.
audit_strategy(): grade a strategy before matching.summarise_matches(): overview of a dedup or candidate result, unified across backends.explain_match(): per-token attribution of a single pair's score.sample_matches(): draw pairs by mode (high, low, borderline, ambiguous, top-gap, random).compare_stages(): per-stage coverage for multi-stage workflows.tinyplot functions, one per view.recommendations().Optional semantic matching that complements rather than replaces the token core. Use embeddings for fields where word-overlap fails (paraphrases, multilingual variants, fuzzy free-text descriptions) and combine them with token strategies via multi_stage_match().
embedding_strategy(): declarative strategy for embedding-based linkage, mirroring the ergonomics of search_strategy(). Specify one or more embedding columns, an optional block_by, an optional threshold, and an optional weights vector across embedding columns.normalize flag for users who want to keep raw magnitudes.detect_duplicates(), search_candidates(), and extract_unmatched() all accept an Embedding_Strategy and return the standard joinery output schemas (duplicate_group / match_id, score, rank, original columns).multi_stage_match() accepts a sequence of mixed Search_Strategy and Embedding_Strategy objects, threading residuals between stages and stopping early when either side is exhausted. Useful pattern: cheap token stage first, then embedding stage on the residual.block_by support for embeddings so cosine search runs within blocks (e.g. country, year bucket) instead of across the whole table.tidyllm (optional Suggests dependency): provider-agnostic helpers for Ollama, OpenAI, and other tidyllm-supported backends, so users can move from raw text to a matchable embedding column without leaving R.block_by SQL bug fixed.A maintenance release with no new user-facing features. The goal was to harden the test suite and close coverage gaps before resuming feature work on embeddings and diagnostics.
methods_duckdb.R coverage raised from 34% to 90%; full behavioural parity with the data.table backend now exercised by tests.embedding_methods_* coverage raised to 95%+ on both data.table and DuckDB backends.batch_duckdb brittleness diagnosed and fixed (see notes/batch_duckdb_brittleness.md). User-facing impact: small inputs no longer hit pathological batching behaviour.local_tests/.batch_duckdb small-table brittleness.This release implements advanced matching heuristics that significantly improve accuracy and robustness.
rIP Smoothing: Four smoothing methods for token weights:
smoothing(method = "log"): Log transformationsmoothing(method = "softmax", temperature = 1.0): Softmax with temperaturesmoothing(method = "offset", alpha = 0.1): Additive smoothingsmoothing(method = "none"): No smoothing (default)Containment: Control maximum matches per record:
max_candidates parameter limits top-N matchesFeedback Weighting: Penalize low token overlap:
feedback_strength parameter (0-1) controls intensity.score_pairs_sql() helper consolidates scoring logicdetect_duplicates() and search_candidates()