The narrative corpus behind morie_describe() was excluded from the
0.3.1 tarball to get this package under 5 MB, which left rmorie
carrying the only shipped copy. rmorie has no room for it either, so
the corpus is back where it belongs, and both packages sit in the
5-10 MB range CRAN accepts with a written justification.
The package shipped 9.2 MB, and most of the excess was duplication:
rmoriedata.sqlite (15 MB uncompressed) was retired by the
SQLite-to-Parquet migration and no shipped code reads it. Kept in the
repository, excluded from the tarball..csv.gz. load_siu_reports(format = "parquet") now reads the store
copy that morie_data_load() already serves, so the top-level
duplicate is gone. The CSV remains: it is the default format and a
genuinely different encoding, not a third copy of the same one.describe_corpus.Rds (1.6 MB, and it does not compress) is referenced
by no R code, Rd page or test. Excluded.Verified after the change: the Parquet path, the CSV path and the
siu_directors_reports store slug all return the same 5,157 x 65 corpus.
nanoparquet is gone from Imports. The bundled store is now read and
written by this package's own codec (R/aaa_parquet.R), so a plain
install no longer pulls a compiled Parquet dependency, and
load_siu_reports(format = "parquet") works on a machine that never
had one -- previously it turned a missing optional package into a hard
stop, despite the corpus being bundled in exactly that format.morie_data_load("vic_<key>"); they appear
in morie_data_catalog() like any other slug.data-raw/build_vic_tables.R.morie_datasets_vic_table() defaults to offline = TRUE, and with
nothing bundled it returned a 0-row frame on any machine that had not
already downloaded the workbook.R CMD check on R before 4.6 no longer notes rmoriebricklayer as an unused import.morie_data_load() is mdload(), morie_dp_laplace_count() is
dplapcnt(), load_chicago_data() is chiload(), quasi_identifiers is qids, and so on.
Each help page gives the full form of its short name. The old names are not kept.hostlogin()), mdask() and the downloads that carry the data key need
rmoriebricklayer 1.0.0; with an older one they say so, and attaching rmoriedata prints a
one-line note on how to update.morie_data_hosted_login() needs the data
access terms accepted first, then your name, organisation and why you need the data (and, for the
restricted tables, a supervisor, department chair, director or coordinator). Pass them as
arguments (accept_terms = TRUE, name = , purpose = , ...) or answer at the console.ask() is built on (0.5.9 was
accepted, so a build against an older one ran ask() on the retired hosted tier).ask() uses a language model you bring (a provider's key, Ollama, or your own server) through
rmoriebricklayer, and the model is given rmoriedata's and rmoriebricklayer's help with tools to
read it. Its backend argument is now route ("auto", "own", "ollama" or a provider id
such as "anthropic"); the hosted MORIE tier and the rmorie command-line fallback are gone.morie_data_hosted_load() uses a downloaded table only when it matches the SHA-256 the
manifest lists (fetching the manifest again once before calling a file bad); a mismatch
deletes the copy and says so. A restricted table (the ICPSR studies, NCVS 38604, the a2aj
case law) refused with 403 says to accept the terms at https://data.rmorie.com/terms.morie_data_llm_config() shows and saves the language-model settings (route, and the
address, key and model of each route), shared with rmoriebricklayer, rmorie and morie.A hex logo (man/figures/logo.png, source data-raw/hex_logo.svg), on the README and the
package website, with its favicons; the website indexes morie_dp_budget() and morie_dp_spent().
A garbled download is refused, not padded. The full Chicago export and the tables from
data.rmorie.com are read with fill = FALSE: a row with a field missing or one too many was
padded into the result (and, for Chicago, written to the cross-session cache); it is now an
error, and a damaged hosted table's local copy is removed so the next call fetches it again.
The Chicago tests also pin the 1% row-count tolerance at its edge (495 of 500 accepted, 494
refused) and the request size (the service's count, at least 5,000,000 rows).
Privacy verifiers (review of 0.3.5). morie_k_anonymity_verify() treats a missing
quasi-identifier as a level of its own: stats::aggregate() had dropped every row with an NA
key, so a class of one person could pass k = 3; class sizes now always sum to nrow(data).
morie_l_diversity_verify() counts only known sensitive values ("HIV", NA, NA is diversity 1,
not 2). morie_cell_suppress() adds complementary suppressions to a fixed point and then audits
the result exactly: a hidden cell the row and column totals determine (its indicator in the row
space of the margin constraints) gets another complement, and the result carries
recoverable_mask, n_recoverable and a protected verdict to test before publishing. The
0.3.5 passes left a lone hidden cell in a row or column, from which the primary value followed
by two subtractions; zero cells are no longer used as complements. Checked against brute-force
enumeration of every margin-consistent fill on 464 random tables: no disagreement.
Differential privacy. Noise comes from the operating system's CSPRNG
(rmoriebricklayer::random_bytes()), not R's seeded generator, so set.seed() no longer
reproduces a release. Counts and histograms use the exact discrete Laplace sampler of Canonne,
Kamath and Steinke (2020) in integer arithmetic (whole-number releases; no floating-point
artefact of the kind Mironov (2012) showed in the textbook Laplace sampler). The mean uses the
analytic Gaussian mechanism of Balle and Wang (2018), valid for every epsilon; the classical
bound, valid only for epsilon <= 1, is gone. All three mechanisms use one neighbouring relation
(add or remove one record): the mean is a noised sum over a noised count. New
morie_dp_budget() / morie_dp_spent() charge releases against a budget (basic composition)
and refuse one that would exceed it. Examples no longer call set.seed().
Parquet reader. Every byte access is bounds-checked (an out-of-range raw index had silently
read zeros); varints, lists, maps and nesting are capped by the bytes that remain, so the 21-byte
file that looped forever is refused at once; the decoded row count must equal the row group's
and the footer's exactly (no padding with NA, no truncation); a column chunk's type must match
the schema's; PLAIN values must fill their page exactly; dictionary indices are range-checked;
page sizes and footer row counts are capped (options(rmoriedata.parquet_max_page_bytes, rmoriedata.parquet_max_rows)); PageHeader.crc is verified when present and the writer now
writes it. tests/testthat/test-parquet-hostile.R fuzzes all three codecs.
Downloads. .rmd_dl(), which carries the MORIE key as a bearer header, goes through
rmoriebricklayer::bricklayer_download() (https to public hosts only, no file://, size cap)
instead of base url(); requires rmoriebricklayer 0.5.9. load_cihi_data_tables() checks the
catalogue against the signed manifest before reading it. morie_data_verify() reports files
added to the store (listed = FALSE), and morie_data_checksums() compares each digest with the
manifest (in_manifest).
Workflows. The SIU manifest refresh runs only on demand and uploads an artifact instead of
committing: the manifest is covered by the XMSS-signed _checksums.csv, whose key never leaves
the maintainer's machine, so it is re-signed locally with data-raw/sign_store.R. A new
ASAN/UBSAN job builds the C and runs the suite, including test-codec-hostile.R.
Packaging. utils is declared in Imports; the stale RoxygenNote is dropped
(Config/roxygen2/version is the roxygen 8 field); NOTICE moves to inst/NOTICE so it
installs, and its data attribution now names the publishers instead of per-file headers that did
not exist. The licence text itself stays out of the tarball, as CRAN asks for a standard licence.
DESCRIPTION describes the disclosure-control helpers by what they do.
The curated-data service address comes from the signed services document at
rmorie.com (through rmoriebricklayer::bricklayer_services() when the installed
bricklayer has it; an older bricklayer keeps the default address), so the endpoint can
move or be paused without a release. Keys are personal and issued on request at
www.rmorie.com/access/; every hint says so, and morie_data_hosted_login(token = )
stores one (the GitHub and emailed-code sign-ins keep working).
ask() follows rmoriebricklayer::bricklayer_llm_ask()'s route order: an endpoint of
your own, a local Ollama server, then the hosted MORIE tier as a last resort;
backend = "hosted" insists on the hosted tier. The setup hint when nothing answers
names all three.
The Parquet store is compressed: the writer's Snappy stream was literal-only (valid, but it
compressed nothing), so every table's Parquet copy was as large as its raw pages. Pages are now
written with GZIP (zlib, level 9) by default, and Snappy, still available, has a real
match-finder; both codecs and the reader are C (src/gzip.c, src/snappy.c). The Parquet copies
shrink from 16.1 MB to 3.3 MB (the SIU corpus from 10.0 MB to 2.1 MB) and the installed package
from about 27 MB to 14.5 MB; pandas/pyarrow read them as before.
SIU corpus: narrative_summary holds each report's own account of the incident on 4,605 of 4,613
reports (it was empty but for three values that described the SIU's investigation). The text is
the opening of the report's "Incident Narrative" / "Description de l'incident" section, or of the
"Notification of the SIU" / "Avis à l'UES" section in reports that have no narrative section (2005-2016
and re-opened files), about 900 characters cut at a sentence end. It was read from the SIU's pages,
re-crawled on 2026-10-05 at one request every 3 seconds; data-raw/add_siu_narratives.R rebuilds it.
SIU corpus, round 8 (4,613 rows; every change is a row of data-raw/siu_round8_review.csv, with
the reason read from the page, or a rule of data-raw/siu_round8_rules.R):
corrigenda, not counted as
a report (drids 385, 620/621 and 5065 correct 328, 521/522 and 4561); six second copies of one
report (identical page text) are removed;charges_recommended is "FALSE", "TRUE" or empty (it had 39 spellings), and logical in the
typed table (morie_data_load("siu_directors_reports"), CSV and Parquet), as panel_reviewed is;
sex_gender_affected is male or female; one name for Kawartha Lakes and for Cornwall (the
Cornwall Community Police Service is now the Cornwall Police Service);narrative_summary held the SIU's mandate paragraph or the page title, not a narrative, and is
empty except on the three rows that held report text; supplemental_materials lists the
legislation and case law a report cites (it held the page's share and privacy links).ask() sends the bundled catalogue (slug, kind, rows) with the question, so the model names tables
(it was told only to "prefer the bundled catalog", which it could not see). Through the rmorie binary it
calls rmorie agent [--model NAME] QUESTION, the agent's own options: it passed --backend, which
the agent refuses, and -m for the model; backend is "auto", "hosted" or "cli". The examples that need a stored key or the binary are \dontrun; the one that runs
switches the hosted tier off (MORIE_HOSTED_BASE_URL = "off"), so it reaches no network
whatever key the machine holds.
morie_dp_laplace_count() and morie_dp_laplace_histogram() refuse Inf in words (they warned
about integer coercion, then failed in R's internals) and refuse counts above 2^53: a double that
large has no room for Laplace noise of scale 1/epsilon, so morie_dp_laplace_count(1e300, 1)
returned exactly 1e300, a "private" release equal to the true count. A whole count beyond R's
integer range but below 2^53 is still a count. morie_core_mean() refuses text instead of
averaging NA.
Help: ?rmoriedata describes the package's functions (it said there were none); ?load_siu_reports
lost "% of them" to an Rd escape; "more than 160" databases at data.rmorie.com; a garbled sentence
in ?load_chicago_data; the README's Chicago example is bounded (limit = 200).
The SIU director's-report corpus: police_service is the service of the subject officials
on every row (it was the notifying force on many French rows, and on 24 English rows a
custody, requesting or neighbouring service -- each corrected after reading the report);
one spelling per service; 59 English reports published since the last build, reviewed by
the same model panel, and their 60 French reports added; 54 French reports filed under
/en/ relabelled; 29 rows that held no director's report removed.
ask() answers through the hosted MORIE tier with the key
morie_data_hosted_login() stores (ollama.com cloud models and additional AI
models such as kimi-k2.6:cf or gpt-oss-120b:cf), and only falls back to the
optional rmorie command-line agent when no key is stored.
Hosted tables download with a live progress bar (percent, size, rate) in
an interactive session; options(morie.quiet = TRUE) silences it.
Requires rmoriebricklayer 0.5.2 or newer (the login flow).
morie_data_hosted_login() signs in to the hosted MORIE tier (GitHub device
flow, email code, or a key you hold) and stores the key in the credentials
file every MORIE package reads, so rmoriedata alone is enough to use
data.rmorie.com.
morie_data_hosted_catalog() and morie_data_hosted_load("db/table"): the
160 databases the MORIE project materialises from BigQuery public datasets,
served from the edge at data.rmorie.com and opened by the MORIE key that
rmorie, morie or rmoriebricklayer store. Cached under the package cache
directory (tempdir() unless options(rmoriedata.cache_dir = )).
Nothing is written outside tempdir() unless the user asks. The
Chicago full-dataset cache lives under tempdir() by default and moves
to a persistent directory only through
options(rmoriedata.cache_dir = ); clear_chicago_cache() empties it.
load_chicago_data(as = "parquet_path") also writes under tempdir():
it wrote the bundled sample into the user's R_user_dir() cache, and a
bounded full = TRUE fetch could land on the full-dataset cache path.
The schema names the language column of siu_directors_reports and
siu_drid_manifest _language, as the files do; it carried the
X_language that check.names had made of it, so the typed store and
load_siu_reports() disagreed on the name. load_siu_reports() documents
that it returns the corpus as text ("" for empty cells) while
morie_data_load("siu_directors_reports") returns the typed frame.
Every bundled table ships as Parquet again, next to its CSV
(inst/extdata/parquet/<slug>.parquet, plus _catalog.parquet and
siu_directors_reports_corpus.parquet). morie_data_load(format = "parquet") and
load_siu_reports(format = "parquet") read those copies; the new
morie_data_path(slug, format) returns the verified path of either
copy, which is the bridge to pandas.read_parquet(). The catalog gains
a parquet_path column and the signed manifest covers the new files.
The Parquet writer no longer mangles non-ASCII text in a C locale
(enc2utf8() on an unmarked string turned each byte into its <c3><a9>
display form, and load_chicago_data(full = TRUE) wrote that into the
cross-session cache). load_siu_reports() reads its CSV as UTF-8 like
the store does, so both copies agree in every locale; a data page v2
file is reported as such before any decompression; and CI runs the
check in a C locale. Every CSV reader in the package now declares
UTF-8, so a live Chicago fetch and its cached copy are identical()
in any locale. A string that is not valid UTF-8 and carries no
encoding mark is refused by column name (a Parquet string is UTF-8;
escaping lost the data, passing the bytes through made a file no other
reader opens), and duplicate column names are refused on write and kept
apart on read. The cross-session Chicago cache carries a SHA-256
sidecar checked on every read. load_cihi_data_tables() rejects an
archived_only that is not TRUE or FALSE instead of ignoring it.
morie_data_checksums() gains a path column relative to the extdata
root; the bare file basenames it returned were not unique and 20 of
them did not resolve from the root (morie_data_verify() already
returned path).
ask() quotes every argument it hands to the rmorie CLI. The
preamble contains a parenthesis, so with the CLI installed every call
had died in the shell before the binary ran; a ; in the question
would have run as a command. model and backend are validated, and a
blank or NA question is rejected like "" and NULL. A stub binary
on PATH now exercises the branch the tests never reached.
morie_data_load() on a dictionary slug says to use
morie_data_dictionary() instead of pointing back at the catalog.
load_chicago_data(full = TRUE) no longer caches a response that is not
the dataset. A 200 carrying an outage page, a header-only body or an
export shorter than the row count the service reports is refused with
the reason; before, an outage was written to the cross-session cache and
every later session read the empty frame back. The cache is written to
a temporary file and renamed, so a concurrent reader never sees a
half-written file; a cache that cannot be read is discarded and
refetched; and a new refresh = TRUE argument refetches on demand. When
the service's count endpoint is down the data is returned but not
cached across sessions.
fetch_cihi_table() checks the downloaded file's size and signature
against the catalogue's declared format and validates which (a whole
number within the catalogue, or a non-empty title substring) and
timeout before any request. A live URL that answers 200 with an
outage page now falls through to the Wayback copy.
The verifier pins the signing key's XMSS root and public seed in package code. A store re-signed with another key, even with a consistent manifest and shipped public key, no longer verifies; the signature now means more than a checksum against local tampering.
CSV tables are read with encoding = "UTF-8" instead of
fileEncoding = "UTF-8-BOM": in a C locale the latter truncated 22 of
the 99 tables (5 to zero rows); the bytes are now left alone and
marked as UTF-8 in every locale.
The data store is signed. _checksums.csv lists every shipped file
with its SHA-256, and the manifest's own SHA-256 carries an XMSS
(RFC 8391, SHA-256) signature whose public key ships as
_signing_key.json. The signature is checked once per session and each
file when first read, so a modified or corrupted install errors instead
of returning altered data; morie_data_verify() checks the whole store
at once. Needs rmoriebricklayer 0.4.0 or newer for the verification.
load_siu_reports(format = "parquet") pointed at the parquet copy this
release removed; the option now errors with a note that the gzip CSV is
the same corpus. rmorie reads the corpus through the default "csv".
morie_dp_gaussian_mean() now rejects infinite lower or upper with
the error its message already promised, instead of returning NaN with
a warning from rnorm().
The data store ships every table once. morie_data_load() reads the
table's CSV (the file rmorie reads by name) and applies a bundled column
schema, so each table comes back with exactly the names and classes the
retired parquet copies had (verified identical for all 99 tables), and
keeps it in a session cache: repeated loads are instant. The parquet
directory, a duplicate samples/ directory and a second copy of the SIU
reports are gone, which takes the source tarball from 6.8 MB to under the
5 MB CRAN cap. The ten Victoria tables that existed only as parquet now
ship as CSV under vic/. morie_data_catalog() has n_rows and
n_cols for every table; morie_data_dictionary() returns the shipped
JSON.
Documentation is generated with markdown roxygen; backticks in the help pages are now \code{} (they rendered as stray opening quotes in the PDF manual).
load_chicago_data(full = TRUE) gains limit (exact row cap) and
fraction (share of the live dataset, total looked up first); bounded
fetches never touch the full-dataset cache. Network examples are now
\donttest{} with try() so they degrade gracefully offline.
SIU corpus also bundled as Parquet; load_siu_reports(format = "parquet") reads it via nanoparquet (columnar / SQL-friendly). CSV remains the default (no extra dependency).
panel_reviewed flag column (65 cols).inst/extdata/rmoriedata.sqlite
(retired by the sqlite-to-parquet migration; no shipped code read it —
the loader API is Parquet-only via nanoparquet, and the file remains
backed up outside the package). Tarball drops from 5.5 MB to 4.25 MB.AGPL (>= 3) (the LICENSE file was a
verbatim copy of the standard license, which CRAN flags).CITATION.cff, NOTICE, and LICENSE are excluded from the built
package (.Rbuildignore), clearing the top-level-files NOTE.LinkingTo: rmoriebricklayer (>= 0.2.0) and links
the ecosystem's shared compiled core instead of duplicating any C code.morie_core_sha256() and morie_core_mean() call the shared
kernels directly (fast data-integrity hashing + summaries for the integrated
fixtures, with no dependency on rmorie). Tests assert they are
byte-identical to rmoriebricklayer's own core_sha256() / core_mean().Six small, base-R-only helpers for analysts releasing aggregate statistics from the integrated fixtures (or any other dataset) without re-identification risk:
morie_dp_laplace_count() — (epsilon, 0)-DP count via the Laplace
mechanism (sensitivity 1).morie_dp_gaussian_mean() — approximate (epsilon, delta)-DP mean of a
bounded numeric vector via the analytic Gaussian mechanism.morie_dp_laplace_histogram() — per-bin Laplace noise on a histogram.morie_k_anonymity_verify() — checks k-anonymity over a set of
quasi-identifiers and returns the offending equivalence classes.morie_l_diversity_verify() — per-class distinct-count check on a
sensitive attribute.morie_cell_suppress() — small-cell suppression with optional
complementary suppression so primary-suppressed cells can't be
reconstructed from row/column marginals (StatCan-style).All helpers use only base R + the existing stats import; no new
runtime dependencies. Inputs are validated and edge cases (empty
vectors, NAs, out-of-bounds inputs, invalid privacy parameters) throw
informative errors.
tests/testthat/test-dp.R — Monte-Carlo convergence + variance-scaling
sanity tests for the DP mechanisms; input-validation coverage.tests/testthat/test-k-anonymity.R — hand-built fixtures with known
equivalence-class structure; complementary-suppression correctness.