The justifier package
offers a flexible compromise that enables thorough documentation that is
both human- and machine-readable. This allows learning from the
decisions taken in one or multiple projects, and making clear where
decisions were based on strong versus weak justifications. What exactly
comprises a strong or weak justification is of course greatly dependent
on domain and context, and justifier provides for this.
Justification frameworks for two specific cases have been provided in
the vignettes ‘justifier in behavior change intervention development’
and ‘justifier in study design’. This vignette provides a general
introduction to justifier.
In justifier, four types of information are
distinguished:
source: A source is a document or other
resource the contains more information about something. Commonly,
sources will be journal articles, books, or reports, but
they can be anything really.assertion: An assertion is a statement
that something is the case. An assertion will, if possible,
be based on one or more sources. In justifier,
an assertion is very similar to assertions in
the nanopublication
ontology, in which case one could compare the
assertion’s description plus the referenced
sources or other justifer elements as its
provenance.justification: A justification is a
justification of something or other. Generally,
justifications will be used to justify
decisions, but sometimes, you may want to justify something
without it being tied to a specific decision.decision: A decision is an explicit choice
between one or more implicit or explicitly listed options. A
decision should have one or more
justifications which are each ideally based on one or more
assertions that each reference one or more
sources.These justifier elements are specified in YAML fragments
in plain-text files, that as convention have the extension
jmd (‘justifier markdown’). This enables including
justifications right in one’s minutes or meeting notes without having to
use dedicated software - the only constraint this introduces is that one
has to use Markdown for those minutes or notes. Markdowns combination of
using a plain text format while allowing embedding of basic markup
metadata (and the ability to extend that using HTML), and its popularity
to produce reproducible reports and literature prorgamming documentation
using the R Markdown framework, make Markdown an obvous choice for
documentation if one desires to adhere to Open Science principles.
The justifier package offers functions to extract and
load these YAML fragments. This, of course, in itself doesn’t have any
added value. This added value is introduced by parsing the
justifier elements in a specific context where, for
example, the types of decisions can be organised hierarchically, and
types of evidence can be ranked, to provide insight into the decisions
in different phases or a process or in different projects in an
organisation.
To this end, justifier work with the concept of
justification frameworks. These frameworks provide the
context for the decisions and justifications. The
justification frameworks, therefore, enable one to extend
the justifier format to attach context to decisions, for
example in terms of which part of a process they pertain to, or in terms
of what part of the world they relate to. These contexts or domains are
hierarchically structured within the
justification framework. In addition to their position in
the wider framework, each context can specify constraints for decisions
in that context. When justifier processes a set of
justifications in this framework, it will notice violations of this
specification. Such values can achieve additional attributes, such as
scores, ranks, or strings with comments. These attributes can then be
aggregated for all justifications in a domain, or collapsed to lower
levels of the context tree. This makes
justification frameworks ‘judgemental’, where
justifications in themselves are not judgemental; they simply document.
However, when placed in a context, such justifications can be
categorized as better or worse using the metadata provided by the
justification framework.
For example, imagine a social activity committee at a faculty at a
university somewhere in the world. Since social cohesion is an important
factor in staff happiness, it would make sense that such a vital organ
as the social activity committee would decide to document their minutes
using justifier, after all. In a meeting, they could decide
to organise the next annual faculty outing as a bowling match, and to
not include dinner, but include lunch. The underying reasoning could be
that last year, there was a non-sporty activity, and the dinner got
quite out of hand. These choices therefore cater to people’s perceived
need for variation and hopefully avoid embarrassing (but fun) scenarios
like those witnessed last year.
By imposing a justification framework, one could, for
example, attach a ‘decision quality’ value to these decisions. Imagine a
framework for a faculty that values democracy very highly. In this
framework, decisions’ justifications pertaining to the context of
‘activity’ have to have one of three values: ‘survey under staff’,
‘discussion at staff meeting’, or ‘internally generated’. It is then
possible to easily generate overview of types of justifications in
different meetings. For example, it could, hypothetically, turn out to
be the case that decision bodies at higher levels make more internally
generated decisions, whereas decisions at lower levels are more often
based on discussions with staff members.
Two examples of such frameworks are provided in the ‘justifier in behavior change intervention development’ and ‘justifier in study design’ vignettes.
For example, when designing an empirical study, some form of data will often be gathered. In the design phase, therefore, decisions are taken as to the amount of data that will be required. These decisions share a role in the study design process, and once such context is clear, further specification of the decisions becomes possible. For example, for quantitative studies in psychology, a decision could pertain to whether sample size calculations are based on a frequentist approach, such as on null hpothesis significance testing, accuracy in parameter estimation, or a Bayesian approach.
For example, a framework could contain a context called
study type that specifies that decisions in that context
can only have one of two values: “confirmatory” or “exploratory”.
In addition, the relative contextual information enables collapsing the justifi
Justification frameworks can be specified in three ways. First, using YAML stored in a local file; second, using a spreadsheet stored in a local file; and third, using a spreadsheet loaded from Google Sheets. Of these,
In justification frameworks, common metadata specifications are the following:
These specifications can pertain either to justifier
specifications within a given domain, but they can also hold for all
justifier specifications that are loaded in a project. For
example, it is possible to specify that all sources must
have a type, and that that type must always be
team belief, expert consensus,
target group consultation, qualitative study,
quantitative study, or research synthesis.
justifier will then flag deviations.
When specifying something for multiple justifications, the eponymous
justifier element can be used, making that the fifth
justifier element. Contents of a justifier
element pertain to all other elements in a file
(scope: local), or in a project directory
(scope: global), or all files that are read by the
justifier package in a project
(scope: universal).
For example, to specify that the justification framework
is stored in file justifier-framework-specification.jmd,
and that this framework should be applied to all justifications in this
directory that do not have another framework attached:
---
justifier:
framework: justifier-framework-specification.jmd
scope: global
---To specify that all justifications in this file are produced at date the sixth of March 2019:
---
justifier:
date: 2019-03-06
scope: local
---The type metadata can be used to impose more constraints
by providing conditions. For example, to specify that all decisions in
the project of the type “global_design” must have either “experimental”
or “observational” as value, one would use:
---
justifier:
scope: universal
condition:
element: decision
type: global_method
values: ["experimental", "observational"]
---justifier will then check the type of all
decisions in the project. If a decision has the type “global_design”,
justifier will check whether whether it has a value and
whether the value is “experimental” or “observational”; if not,
justifier will indicate this.
It is also possible to attach additional attributes to these values, such as scores or weights, by including another vector:
---
justifier:
scope: universal
condition:
element: decision
type: global_method
values: ["experimental", "observational"]
scores: [1, 2]
---