Every score this package reports is written out below. Nothing is a black box, and where a choice is arguable the argument is given rather than hidden.
All four dimension scores are ratios in \([0, 1]\), the form ISO/IEC 25024
measures take. as_pct() converts once, at the point of
display, to 0–100. Holding one scale internally is deliberate: the
implementation this package replaces kept completeness on 0–10 and
consistency on 0–1 and multiplied both by ten, which is why its
published consistency figure for iris reads 2.5% where 25%
was meant.
\[ \text{completeness} = \left(1 - \frac{1}{p}\sum_{j=1}^{p} \frac{m_j}{n}\right) \]
where \(m_j\) is the number of missing values in column \(j\), \(n\) the row count and \(p\) the column count.
This is the mean of the per-column missing rates,
not the overall share of missing cells. The two differ whenever columns
are unequally incomplete, and the column-averaged form dilutes a badly
damaged column: one column that is 30% empty out of four costs 0.075,
not 0.3. That is a property of the published formula, kept here for
continuity with the dissertation; if you want the cell ratio, compute it
from report$missing$columns.
\[ \text{accuracy} = \left(1 - \frac{\text{outliers flagged}}{\text{numeric values tested}}\right) \]
Outliers are flagged by one of two rules.
Z-score. A value is flagged when \(\left|\frac{x_i - \mu}{\sigma}\right| > 3\).
IQR. A value is flagged when it falls outside \([Q_1 - 1.5\,\text{IQR},\; Q_3 + 1.5\,\text{IQR}]\).
Two implementation choices are worth stating plainly.
Outliers are counted on observed values, not imputed ones. Mean imputation adds mass at the centre of the distribution and shrinks \(\sigma\), which mechanically suppresses the very z-scores this step exists to find. The dissertation ran this step after imputation; this package does not.
A zero-variance column contributes zero outliers, not
NA. scale() returns NaN when
\(\sigma = 0\), and summing that
propagated NA through the entire report, so a single
constant column made the whole outlier count unreportable.
\[ \text{consistency} = \frac{\left|\{j : \mathrm{VIF}_j \le \tau\}\right|}{p_{\text{num}}} \]
with \(\tau\) =
vif_threshold, default 5.
The variance inflation factor is defined as \(\mathrm{VIF}_j = \frac{1}{1 - R_j^2}\), where \(R_j^2\) comes from regressing column \(j\) on all other numeric columns. This package does not fit those regressions. It uses the identity that the VIFs are the diagonal of the inverted correlation matrix:
\[ \mathrm{VIF} = \operatorname{diag}\left(\mathbf{R}^{-1}\right) \]
which gives the same numbers in \(O(p^3)\) time, independent of the row
count. The equivalence is asserted in the test suite against explicit
lm() fits:
set.seed(11)
d <- data.frame(a = rnorm(500), b = rnorm(500))
d$c <- 0.6 * d$a + 0.3 * d$b + rnorm(500)
dg_vif(d)
#> a b c
#> 1.311907 1.090128 1.410281
vapply(names(d), function(col) {
r2 <- summary(lm(d[[col]] ~ ., data = d[setdiff(names(d), col)]))$r.squared
1 / (1 - r2)
}, numeric(1))
#> a b c
#> 1.311907 1.090128 1.410281Perfectly collinear columns are reported as Inf rather
than dropped, so they stay visible and still count against the
score.
\[ \text{timeliness} = \left(1 - \frac{\text{future dates} + \text{stale dates}}{\text{dates present}}\right) \]
A date is future if it is after Sys.Date(),
stale if it is more than stale_years (default 10)
old.
This dimension is only measurable if date columns exist. Character
columns are therefore parsed as dates before any text
normalisation runs – otherwise punctuation stripping turns
"2024-01-15" into "20240115" and no date
column ever exists.
Skewness and kurtosis use the sample moment forms:
\[ g_1 = \frac{1}{n}\sum \left(\frac{x_i-\mu}{\sigma}\right)^3 \qquad g_2 = \frac{1}{n}\sum \left(\frac{x_i-\mu}{\sigma}\right)^4 - 3 \]
The \(-3\) makes \(g_2\) excess kurtosis, so
a normal distribution scores 0. This matters: the dissertation’s formula
subtracts 3 but its code called moments::kurtosis(), which
does not, so the published formula and the published code disagreed by
exactly 3. The formula is the contract here.
Normality uses the Shapiro-Wilk test, which is undefined above 5,000
observations. Larger columns are tested on a deterministic subsample and
the size actually used is reported in shapiro_n, so the
result is never presented as though the whole column were tested.
\[ \text{overall} = \frac{\sum_{d \in D} w_d \, s_d}{\sum_{d \in D} w_d} \]
where \(D\) is the set of dimensions that apply to the data. A data set with no date columns cannot be scored on timeliness, so timeliness is dropped and the remaining weights are renormalised rather than the data being penalised for a dimension it cannot have.
Everything above is a statistic. This section is how those statistics become scores against the standard’s model.
ISO/IEC 25012 defines five inherent characteristics of data quality – accuracy, completeness, consistency, credibility and currentness – and beneath them fifteen quality properties. ISO/IEC 25024 gives each property a measurement function of the form
\[ X = \frac{A}{B} \]
on a ratio scale over \([0.0, 1.0]\), where \(A\) and \(B\) are counts of quality measure elements: \(A\) the items satisfying the property, \(B\) the items for which the property is defined at all.
That second count is the whole difficulty. For the accuracy-range property, \(B\) is “the number of data items for which can be defined a required interval of values” – and a required interval is not in the data. It is declared by whoever knows what the data means. This is exactly why every other R package in this space demands a data dictionary before it will run.
datagrade splits the difference. Six properties have a
\(B\) countable from the table itself
and are always measured; nine have a \(B\) that only a declaration can supply, and
dg_spec() is where it goes.
report <- dg_assess(iris, verbose = FALSE)
report$measures[, c("code", "characteristic", "value", "a", "b")]
#> code characteristic value a b
#> 5 EXAC_SINT accuracy NA NA NA
#> 6 EXAC_SEMAN accuracy NA NA NA
#> 7 RAN_EXAC accuracy NA NA NA
#> 1 COMP_REG completeness 1.00 150 150
#> 2 COMP_VAL_ESP completeness 1.00 750 750
#> 3 COMP_FICH completeness NA NA NA
#> 4 FAL_COMP_FICH completeness 1.00 150 150
#> 8 INT_REF consistency NA NA NA
#> 9 RIES_INCO consistency 0.25 1 4
#> 10 CONS_SEMAN consistency NA NA NA
#> 11 CONS_FORM consistency 1.00 150 150
#> 12 CRED_VAL_DAT credibility NA NA NA
#> 13 CRED_FUEN credibility NA NA NA
#> 14 FREC_ACT currentness NA NA NA
#> 15 CONV_ACT currentness NA NA NACOMP_REG, record completeness. \(A\) is records with every required field
populated, \(B\) is all records. With
no required in the spec, every column counts as
required.
COMP_VAL_ESP, data value completeness.
\(A\) is populated cells in the
required columns, \(B\) is all cells in
them. This differs from the dissertation’s completeness, which averaged
the per-column missing rates and so diluted a badly damaged column; the
cell ratio is the form the standard’s \(A/B\) takes.
FAL_COMP_FICH, false completeness of
file. A cell can be occupied and still empty. \(A\) is text values that are not
placeholders, \(B\) is all non-null
text values in the required columns. The placeholder list is short and
conservative on purpose – every entry on it ("none",
"unknown", "-999") is a legitimate value in
some data set, and a false positive silently lowers the score.
Matching is case-insensitive after trimming. Replace the list with
null_equivalents, or pass character(0) to
disable the check.
RIES_INCO, risk of inconsistency. \(A\) is numeric columns with \(\mathrm{VIF} \le \tau\), \(B\) is numeric columns with a computable
VIF. The reasoning: information held more than once can contradict
itself, and a column whose variance is largely explained by the others
carries no independent content. This is a proxy, not a
contradiction count – the standard’s property concerns data items at
risk of inconsistency, and collinearity is the only signal for that
available without declared rules. Declared rules go to
CONS_SEMAN instead.
CONS_FORM, format consistency. Each
value is reduced to a signature: runs of digits become
9, runs of letters A, everything else is kept.
So "2024-01-15" and "1999-12-31" share the
signature 9-9-9. \(A\) is
values matching their column’s dominant signature, \(B\) is non-null text values.
The trap is free text. Every sentence is its own signature, so a
prose column would score near zero for failing to conform to a format it
never had. A column is therefore only measured when its values collapse
to a handful of shapes – at most \(\max(3,
\lceil 0.05 \, d \rceil)\) signatures for \(d\) distinct values. Columns that fail this
test are dropped from \(B\) and named
in the measure’s note, not scored.
d <- data.frame(ref = c(rep("AB-1234", 8), "1234-AB", "XX/99"),
stringsAsFactors = FALSE)
m <- dg_assess(d, verbose = FALSE)$measures
m[m$code == "CONS_FORM", c("value", "a", "b")]
#> value a b
#> 11 0.8 8 10Declaring formats in the spec replaces the inferred
signature with a required one, which changes the question from “is this
column internally consistent?” to “does it match the rule?”.
CONV_ACT, timeliness of update. \(A\) is dates neither in the future nor
older than max_age (ten years by default), \(B\) is all dates present. Future dates are
counted against the score rather than ignored: they are impossible, and
treating them as fresh would let a corrupt timestamp raise
currentness.
Each is NA until the corresponding
dg_spec() argument is given, and each carries a
note saying which argument that is. NA here is
not a bad score – it is the statement that \(B\) could not be counted. Aggregation drops
these rather than scoring them zero, so an undocumented data set is
never punished for being undocumented.
Two details are worth stating because they change what the denominators mean.
EXAC_SEMAN joins on the declared key first. A
record with no counterpart in the reference is unverifiable, not
inaccurate, so it never enters \(B\).
CONS_SEMAN counts rule evaluations, not
records. \(B\) is records times
evaluable rules. A rule that errors – because it names a column the data
does not have – is a specification error rather than a data failure: it
is excluded from \(B\) and named in the
note.
The dissertation’s accuracy score flagged outliers against an interval derived from the data – \(\mu \pm 3\sigma\), or the IQR fence. All three ISO accuracy properties require an interval, pattern or reference that was declared. Substituting a derived one measures something real but different: an outlier need not be wrong, and a wrong value can sit exactly on the mean.
So the measure is kept, renamed plausibility, reported
beside the five characteristics and excluded from overall.
It is the most useful number in the report for data that arrives with no
documentation, and it is not the standard’s measure.
Properties combine into a characteristic by the unweighted mean over
those that apply; characteristics combine into overall by
the weighted mean over those that apply. ISO/IEC 25024 does not
address how measures should be aggregated, so both steps are
this package’s choice and are stated as such rather than presented as
conformance.
\[ \text{characteristic} = \frac{1}{|P|}\sum_{p \in P} X_p \qquad \text{overall} = \frac{\sum_{c \in C} w_c \, s_c}{\sum_{c \in C} w_c} \]
where \(P\) is the applicable properties of that characteristic and \(C\) the applicable characteristics.
The model implemented here – five characteristics, fifteen
properties, their identifiers, the \(A/B\) form – is sourced below. What is
not verified against the standard is the exact
definition of each property’s quality measure elements: precisely which
items ISO counts into \(A\) and \(B\). Those live in Annexes A–E of ISO/IEC
25024, which are not in any public source. Where this package had to
choose, the choice is written out above and visible in the
a and b columns of
report$measures, so it can be checked and disagreed
with.
Note also that ISO’s word is currentness;
timeliness comes from a different tradition and was the
dissertation’s word. It is kept as report$timeliness and
accepted in weights.
ISO/IEC 25012 and 25024 are paywalled. Every statement above about their structure is cited below. The standard’s own measurement functions are deliberately not reproduced, because they could not be verified without the documents themselves.
ISO/IEC 25024:2015. Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Measurement of data quality. https://www.iso.org/standard/35749.html
ISO/IEC 25012:2008. Software engineering — SQuaRE — Data quality model. https://www.iso.org/standard/35736.html
Gualo, F., Rodriguez, M., Verdugo, J., Caballero, I. and Piattini, M. (2021). ‘Data Quality Certification using ISO/IEC 25012: Industrial Experiences’, Journal of Systems and Software. https://arxiv.org/abs/2102.11527
The source for three specific claims: that ISO/IEC 25024 measures take the form \(X = A/B\) on a ratio scale over \([0.0, 1.0]\); that the Accuracy Range measure (RAN_EXAC) defines its element \(B\) as “the number of data items for which can be defined a required interval of values”; and that “ISO/IEC 25024 does not specifically address how the measures corresponding to the data quality properties should be aggregated to compute the overall quality level for each data quality characteristic.” The authors are from AQCLab, an accredited ISO/IEC 25012 data quality evaluation laboratory.
Calabrese, J., Esponda, S. and Pesado, P. (2020). ‘Framework for Data Quality Evaluation Based on ISO/IEC 25012 and ISO/IEC 25024’, 8th Conference on Cloud Computing, Big Data & Emerging Topics, Universidad Nacional de La Plata. https://sedici.unlp.edu.ar/handle/10915/104778
The source for the five inherent characteristics being Accuracy, Completeness, Consistency, Credibility and Currentness — which is how the naming mismatch with timeliness was established — and for the decomposition of each characteristic into the fifteen quality properties and their identifiers implemented in this package.
Demirhan, O. H. (2024). Assessing Data Quality Management Issues in Open-Sourced Data Sets with Statistical Methodology. MSc dissertation, University of Greenwich. Cites ISO/IEC 25024:2015 eight times, all within Chapter 2; the citation establishes what the dimensions mean and is not a claim of measure conformance.
The standard’s Annexes A–E — the synoptic table of quality measure elements, the measures indexed by element and target entity, and the alphabetic measure list — are what would settle the remaining question of exactly which items belong in each \(A\) and \(B\). They are not reproduced in any public source.