write_data()/read_data() were
shifted by the time-zone offset (e.g. 2 or 3 hours for Europe/Bucharest).
The instant is now preserved with every 'writexl' version and with
'openxlsx'. ?write_data documents how each format stores date-times.inst/CITATION and CITATION.cff give the Zenodo DOI.?fit_synthesizer.ts selected rows with
EuStockMarkets[1:500, ], which returns a matrix rather than a ts, so the
stationary bootstrap was never used and a warning was printed; it now uses
window(). Passing time-series arguments (method, order, ...) for a
non-ts input now gives a warning that explains the cause.synthesize(file = ) example now writes a
file, shows its first lines and reads it back with read_data().Statistical validation, and the bias it uncovered.
x and the
missingness of y was reproduced at about 60% of its strength (0.30
instead of 0.51). Such pairs are now estimated by maximum likelihood under
the Gaussian copula: polyserial (discrete-continuous) and polychoric
(discrete-discrete) correlations, with thresholds from the marginal
probabilities (Olsson 1979; Olsson, Drasgow and Dorans 1982) and the
bivariate normal CDF of Genz (2004). The estimates agree with polycor's
two-step estimators to about 1e-5. The t copula uses them for these pairs.y depends on
x, the rows where both are observed have a restricted range; the new
estimator reproduces the relation among the observed rows (e.g.
cor(x, y) among observed rows 0.568 synthetic vs 0.572 real, where the
old estimator, combined with the first fix, would give 0.646). With
complete data nothing changes.iris now keeps the species separation much better (a
petal-based rule classifies 82% of synthetic records correctly vs 66%;
real data 96%).latent:
"normal_scores" or "polyserial_polychoric").tests/testthat/test-validation.R: data simulated from known
distributions, dependence structures, missingness mechanisms (MCAR, MAR),
strata and VAR processes; synthetic samples are compared with the truth
or the training data within Monte Carlo tolerances (marginals, latent
correlations, rank association with discrete margins, 2 x 2 tables, MAR
missingness probabilities, observed-row relations, stratum means and
correlations, t-copula df and tail dependence, VAR auto- and
cross-correlations). Run against 0.2.6, it fails 11 expectations in
exactly the areas affected by the bias above. The large designs are
skipped on CRAN; one fast MAR check always runs.fit_synthesizer.ts() warns when an argument does not apply to the chosen
method (block_length outside the bootstraps, order/burn_in outside
"copula_var", max_lag when order is given).disclosure_risk() gains abs_tolerance for numeric targets: a
prediction is correct if within max(tolerance * |y|, abs_tolerance), so
targets with meaningful zeros can be assessed.missing = "drop" leaves columns without any observed value
entirely missing; missingness that depends on the unobserved value itself
(MNAR) is not reproduced; the time-series methods assume approximately
stationary series (?fit_synthesizer.ts, time-series vignette).airquality; its own output showed
otherwise (also in 0.2.6). A single copula only represents monotone
relations; the section now shows a monotone MAR example and that
by = "Month" reproduces the non-monotone pattern.testthat::local_mocked_bindings() (testthat >=
3.1.7) or data.frame() on an integer64 column (needs bit64), which
failed on R 4.2 with an older testthat.Bugs fixed.
seed argument in fit_synthesizer() (data frame, matrix, vector and
ts methods). Fitting is stochastic (random tie-breaking, the randomised
distributional transform, Kendall subsampling for the t copula), so the
fitted model can now be made reproducible when fitting and generation are
separate steps; the global RNG state is restored afterwards.k could be read as a total.n_pair alone does
not flag all of them (?copula_correlation, ?fit_synthesizer).
summary() labels its count as referring to the pooled model.Bugs fixed.
n_pair) and the identification of
pairwise correlations were computed on the random 2000-row subsample used
for Kendall's tau (bug introduced in 0.2.4). With more than 2000 rows,
copula_correlation(fit, what = "n_pair") under-reported the counts, and
a pair with sparse overlap (e.g. 10 joint rows out of 10,000) could be
declared unidentified, or estimated from only a few of its rows, depending
on the seed. Counts and identification now always use the full data, and
any pair that the subsample leaves with fewer than min(n_pair, 500) joint
rows is re-estimated from up to 2000 of its own joint rows.p(p - 1)/2 separate O(n^2) calls of cor(method = "kendall"). Estimates
for complete data are unchanged.completion: "none", "maxdet" or "zero_fallback") and the largest
change made by the subsequent positive-definiteness repair
(pd_adjustment); summary() reports both. The fit-time warning now
states the method actually used and that the repair may follow, instead of
always claiming maximum-determinant completion.Bugs fixed.
fit_synthesizer() warns when this happens, the
fitted copula stores the pairwise counts (n_pair) and the number of
unidentified pairs, copula_correlation(fit, what = "n_pair") returns
the counts, and summary() reports them. Fully observed data are
unaffected.r_mvskewnorm() requires a positive-definite omega, as in the standard
definition of the skew-normal density; singular sigma remains allowed
for r_mvnorm(), r_mvt() and r_mvlnorm() (degenerate distributions),
and this is now documented..check_prob() (used for closeness, close_quantile, ...) rejects
non-finite values explicitly, consistent with the other validators.Bugs fixed.
r_mvnorm(), r_mvt(),
r_mvlnorm(), r_mvskewnorm() and r_mvmixture() are validated by one
shared check: non-empty, square, numeric, finite, symmetric and positive
semi-definite, each with a clear error message (previously some invalid
inputs failed with internal errors such as "infinite or missing values in 'x'").mean, meanlog, xi, skew-normal alpha,
mixture means) must be finite; Inf/NA used to propagate silently into
the output.r_mvskewnorm() requires a strictly positive diagonal of omega, as its
parameterisation standardises by sqrt(diag(omega)).r_dirichlet() now draws the gamma variates on the log scale (with the
Y * U^(1/a) boost for shapes below 1) and normalises with log-sum-exp, so
very small concentration parameters no longer give NaN rows. Draws for
alpha < 1 therefore differ from 0.2.2 for the same seed.make_corr(d = 1) returns the 1 x 1 identity without consulting rho.df is not
estimated but fixed at 10; this now triggers a warning, is recorded as
df_estimated = FALSE in the fitted copula and is shown by summary().?fit_synthesizer gains an "Estimation details" section documenting the
pairwise-complete estimation of latent correlations, the subsequent
projection to a valid correlation matrix, and the grid/profile-likelihood
estimate of the t-copula df.Bugs fixed.
r_dirichlet() requires finite positive alpha;
make_corr() requires finite numeric rho; disclosure_risk() requires a
finite non-negative tolerance; seed arguments must be finite and in
the integer range.r_mvmixture() validates every component (finite means, finite and
positive semi-definite covariance matrices) before sampling, so an
invalid component is reported even when its weight is zero.NaN like NA (both are missing values).dcr(): exact_match_rate now counts only records identical to a real
record on all variables. New zero_distance_rate (distance zero on the
variables compared) and compared_share (share of variables on which the
nearest real record was compared) make the meaning explicit under
na = "exclude". real_duplicate_rate is defined in the same way.compare_synthetic() flags a discriminator AUC well below 0.5, which
indicates synthetic records that copy real ones. The AUC is deliberately
not converted to a symmetric measure such as max(AUC, 1 - AUC), which
would report copies of the real data as easy to distinguish.Bugs fixed.
r_copula() and r_mvdist() now require corr to be a correlation
matrix (square, symmetric, unit diagonal, entries in [-1, 1], positive
semi-definite). Previously a covariance matrix was accepted and produced
margins that were not uniform. The t copula and r_mvt() validate
df (positive; Inf gives the Gaussian copula); Archimedean theta
must be finite.disclosure_risk() and exact-copy detection
are now built from integer codes: a missing value can no longer collide
with the string "<NA>", and values containing the old separator can no
longer make different rows look identical. Strata are listed in order of
first appearance.dcr() gains na = c("category", "exclude"). The default treats
missingness as a category (as before, now documented); "exclude" gives
the standard Gower coefficient, ignoring variables missing in either
record.discriminator_auc() learns all preprocessing (imputation, scaling,
retained categories) within the training folds, so the cross-validated
AUC has no leakage from the held-out fold. Its documentation explains
that values clearly below 0.5 indicate copies of real records.r_mvmixture() rejects all-zero or non-finite weights and components of
inconsistent dimension with clear messages; margin_categorical() rejects
duplicated levels and non-finite probabilities; the quantile functions it
and margin_empirical() return check that probabilities lie in [0, 1].match_types() reads date-time strings without zone information as
local times in the template's time zone (strings ending in Z are UTC,
explicit offsets such as +02:00 are honoured), and accepts mixed
formats within a column.noise perturbation preserves mean and
variance approximately rather than exactly.Complete rewrite. the API changed and is not backward compatible with 0.1.0.
fit_synthesizer() / generate() / synthesize() with Gaussian, Student-t
and independence copulas; categorical, ordinal and logical variables take
part in the dependence structure through the distributional transform.by) with pooled fallback,
identifier columns, compressed quantile functions (knots), dependence
strength control and simulate() for multiple synthetic data sets.closeness
argument of generate(), synthesize() and augment_data();
perturbation() for record-level (noise, swap) and distribution-level
(shift, scale, temperature, dependence up to 2) settings; and
calibrate_closeness() to reach a target discriminator AUC, DCR ratio,
pMSE ratio or marginal distance.discriminator_auc() now uses squares and pairwise interactions of the
numeric variables by default (model = "quadratic", ridge-penalised), so
that differences in spread and correlation are detected; pmse() gains the
same option.disclosure_risk() for statistical disclosure control: replicated uniques
and correct attribution probability (with no-data baseline) for chosen key
and sensitive variables. dcr() gains close_share, the share of
near-copies of real records. calibrate_closeness() accepts any custom
metric, e.g. a risk threshold.census_sim and vignette "Releasing
confidential microdata as synthetic data".augment_data() and ts_windows() for enlarging small training sets of
tabular data and time series for machine learning.r_mvdist() for arbitrary margins.compare_synthetic() bundles them.read_data() / write_data() for CSV, TSV, RDS, RData, Parquet, Feather,
HDF5, JSON, NDJSON, Excel, SPSS, Stata, SAS and fst; generate_to_file()
for chunked generation of very large data sets.closeness given to fit_synthesizer()) now trigger a warning instead of
being silently ignored.closeness, perturb and dependence are not available for time-series
synthesizers; passing them now warns instead of being silently ignored.discrete_threshold values and these are whole numbers or repeat.noise, shift, scale) also apply to numeric
variables modelled as discrete, e.g. within strata; whole-number variables
are rounded stochastically, so shifts are not biased by rounding.difftime, keep their class and
units; 64-bit integers (integer64) are rejected with a clear message.disclosure_risk() gains ignore to leave out direct identifiers, which
otherwise mask exact copies.augment_data() accepts plain vectors.pd_method = "none".rankcor < 1 changed the marginal distributions.make_synthesizer.POSIXct() returned a character vector.seed = left a global .Random.seed behind.synthesize() failed for series longer than 1000 observations.