This package implements the framework described in:
Demirhan, O. H. (2024). Assessing Data Quality Management Issues in Open-Sourced Data Sets with Statistical Methodology. MSc dissertation, University of Greenwich.
The dissertation is a fixed, submitted document. This package is not, and where the two disagree the disagreement is recorded here rather than quietly resolved. Anyone comparing a number in the dissertation against a number this package produces should read this page first.
irisThe dissertation reports 2.5%. The corresponding
measure here – risk of inconsistency, RIES_INCO – reports
0.250, which is 25%.
The scoring function returned a proportion on 0–1 while completeness returned a score on 0–10, and the caller multiplied both by ten as though they were on the same scale. The underlying quantity was never wrong; its presentation was off by a factor of ten. Scores are now held as ratios in \([0, 1]\) everywhere and converted to percentages in exactly one place.
report <- dg_assess(iris, verbose = FALSE)
m <- report$measures
m[m$code == "RIES_INCO", c("value", "a", "b")]
#> value a b
#> 9 0.25 1 4Note that report$scores[["consistency"]] is no longer
this number. Consistency is now a characteristic averaging every
applicable property beneath it, of which RIES_INCO is one;
see the section on the scoring model below.
The dissertation reports
"Petal.Length", "Petal.Width", "Sepal.Length", which is
correct. The code that produced it also appended the literal strings
"row" and "col", because it took
colnames() of the index matrix returned by
which(..., arr.ind = TRUE) – whose columns are named “row”
and “col” – rather than the names of the correlation matrix. At a high
enough threshold the function returned those two strings and nothing
else, reporting redundancy in data that had none.
The dissertation’s formula subtracts 3, so that a normal distribution
scores 0. Its code called moments::kurtosis(), which does
not subtract 3 and returns 3 for a normal distribution. Formula and code
disagreed by exactly that constant. This package follows the
formula.
The plotting code was written as
ggplot(...) + labs(...) + theme_minimal() %>% print().
Because %>% binds more tightly than +, that
parses as
ggplot(...) + labs(...) + (theme_minimal() %>% print()):
the theme object was printed and the plot was discarded. Inside
a for loop, where auto-printing does not happen, no figure
was ever displayed. The boxplots, heatmaps and histograms shown in the
dissertation cannot be produced by the code that accompanies it.
The dissertation documents a “Returns” section listing completeness,
outlier count, consistency and so on. The implementation printed all of
it with cat() and returned NULL.
dg_assess() returns a dg_report object;
printing is a side effect.
data argument existsdata = my_data_frame is documented in three places in
the dissertation and implemented in none of them – only
path existed. It now works.
Text cleaning applied gsub("[^a-zA-Z0-9 ]", "")
unconditionally, deleting every accented and non-Latin character, every
date separator and every @. An assessment tool should
diagnose, not operate, so whitespace normalisation is the only default
and everything else is opt-in and Unicode-aware.
df <- data.frame(city = c(" Ağrı ", "Çorum"), email = c("[email protected]", "[email protected]"),
stringsAsFactors = FALSE)
clean_text_columns(df)
#> city email
#> 1 Ağrı [email protected]
#> 2 Çorum [email protected]
clean_text_columns(df, strip_punctuation = TRUE)
#> city email
#> 1 Ağrı abcom
#> 2 Çorum cdorgBecause a CSV delivers dates as character and text cleaning stripped
the separators, no Date column ever existed and the date
validation step silently checked nothing. Date parsing now runs
first.
Appendix A.3 tests uniqueness after the storage-type tests,
so the "ID" branch was unreachable: a character key matched
is.character() and returned "Text". The
uniqueness test now runs first.
It also runs with an extra condition that the dissertation did not need but this implementation does: a numeric column qualifies as a key only if its values are whole numbers. Without it, every continuous measurement column is distinct in every row, gets classified as an identifier, and is excluded from the statistics – which silently inflated the consistency score to a perfect 1.
Mean imputation adds mass at the centre and shrinks the standard deviation, suppressing the z-scores the outlier step is looking for. The dissertation ran outlier detection on the imputed frame; this package uses observed values.
assign("imported_data", data, envir = .GlobalEnv)
clobbered the user’s workspace and would fail R CMD check
outright.
| Input | Old behaviour |
|---|---|
| A constant column | scale() returns NaN, whole outlier count
becomes NA |
| Exactly one numeric column | data[, sapply(data, is.numeric)] drops to a vector,
cor() fails |
| One redundant pair | Index matrix drops to a vector, rownames() is
NULL |
An all-NA column |
Mean is NaN, written into every row by imputation |
| Perfect collinearity | vif() aborts on aliased coefficients |
| Zero rows | Several checks error |
Each has a named regression test in
tests/testthat/test-edge-cases.R.
visualise_outliers in the dissertation’s examples
versus visualize_outliers in its code; the argument is now
visualize_outliers throughout.correlation_threshold defaulted to 0.5 in the text, 0.9
in the helper and 0.1 in Appendix A.8. It is 0.5.weights, scoring_ranges and
return_imputed_data were accepted and never used.
weights now works; the other two are gone.flag_anomalies_in_relationships) that Appendix A does not
cover. It is retained, documented, and off by default.This is the largest deviation, and it is structural rather than numerical.
The dissertation scores four dimensions – completeness, accuracy, consistency, timeliness – with measures of its own devising. It cites ISO/IEC 25024:2015 eight times, all in the literature review, none in the methodology or implementation chapters: the citation establishes what the dimensions mean and was never a claim of measure conformance.
This package scores the five inherent characteristics of ISO/IEC 25012 through the fifteen quality properties beneath them, each as a ratio \(A/B\) over counted data items. Four consequences follow.
timeliness became
currentness, which is ISO’s word for the same
characteristic. report$timeliness still resolves, and
weights still accepts the old name.credibility is measured, where the
dissertation did not measure it at all. Both its properties need a
declaration, so it reports NA without a
dg_spec().plausibility. The
dissertation’s accuracy score compares values against an interval
derived from the data; all three ISO accuracy properties need
one that was declared. The measure is unchanged and still
reported – under an honest name, and outside the overall score.COMP_VAL_ESP is the cell ratio, which is
the form the standard’s \(A/B\) takes.
The old quantity is still derivable from
report$missing$columns.vignette("methodology") derives every measure. What
remains unverified against the standard itself is which items ISO counts
into each \(A\) and \(B\); that is stated there too.
Every statistic. The choice of z-score and IQR methods, the VIF computation, the correlation-based redundancy check, the Shapiro-Wilk normality test, the mean and median imputation options, and the names of all thirteen helper functions. What changed is how those statistics are organised into scores – the dissertation’s framework, repaired and then mapped onto the standard, not a redesign of the statistics themselves.