R packages are an effective way to distribute research data. A
package carries a version, documentation for each dataset, and a defined
installation location, and R already provides mature ways to reach the
contents: data(), lazy loading of package data, and
namespace-qualified access such as pkg::object. For a
single data package these tools are usually all that is needed, and
projectData is designed to work alongside them, not to replace them.
Difficulties tend to appear when an analysis depends on several
independently distributed collections. Data may be divided between a
primary project package and supplementary or reference packages, and
different collections may define objects with the same name. Attaching
packages or calling data() for each one leaves the choice
of source to attachment order or to repeated manual qualification, which
is hard to keep consistent across a project. Some resources are also
large or stored in formats that ordinary deserialization cannot restore,
such as disk-backed matrices, so they need a format-specific restoration
step and should stay unopened until they are used. Finally, it is often
useful to ask what a project has available, and which installed package
version and library each object came from, without reading any data.
This matters most when several versions or libraries are present. These
are workflow and coordination problems. They do not reflect a limitation
of base R, and many projects never meet them.
projectData is a small coordination layer over installed data
packages. Calling data_attach() on a package registers it
as a named collection and records an inventory of its
resources, along with the package name, version, and the installation
root that was resolved. Registering reads package metadata only; no data
values are read. Objects are then read on first access, and
data_get() can return an object from a named collection or
package without attaching anything. When collections share a name,
projectData does not pick a winner by attachment order. You can
designate a primary collection, state a preference for individual
objects, or add suffix aliases. An unresolved conflict produces an error
that lists the candidates. Attachment is optional and happens only when
you request it, so objects can also be reached without adding names to
the search path. data_catalog() and
data_info() report provenance from the recorded metadata
alone. Ordinary data packages need no changes to work this way. An
optional, validated manifest can describe special files or object
mappings, and a versioned loader interface lets an adapter
supply the discovery and opening steps for a specialized storage
format.
This division keeps the core package light. The core handles
registration, resolution, caching, and provenance, and it reads ordinary
datasets and explicitly described RDS files. Format-specific code lives
in separate adapter packages that you choose to install. For example,
the separate projectData.monocle3 package restores Monocle
3 cell_data_set objects backed by BPCells matrices, so that
projectData does not depend on monocle3 or BPCells. See the adapter vignette for how adapters work. Missing
or unavailable adapters affect only the resources that need them.
projectData is not a package manager. It never installs or updates
data packages, and it does not replace distribution through CRAN,
Bioconductor, Zenodo, or similar channels. It cannot make compatible
software versions or external backing files available, and it does not
by itself make an analysis reproducible. It is also not a
data-processing framework. If data() and
namespace-qualified access already meet your needs, you can keep using
them. projectData is intended for projects that need consistent access
to several collections, deferred opening of large or specialized
resources, and a record of where each object came from.
The normal workflow has three steps:
Need another data package later? Attach it too. If the packages define the same name, projectData asks you to decide which one you mean (see Conflicts).
Installing is ordinary R package installation and is not done by projectData. A data package distributed as a tarball (for example from Zenodo) is installed like any source package:
To keep this vignette offline, the next chunk builds two tiny
throwaway data packages in a temporary library. In real use, skip this
chunk and omit the lib.loc argument below.
library(projectData)
source(system.file("examples", "fixtures.R", package = "projectData"))
lib <- fixture_library()
fixture_package(lib, "rnaseqData", "1.0.0", lazy = TRUE, datasets = list(
counts_rna = list(counts_rna = matrix(1:6, 2)),
meta_rna = list(meta_rna = data.frame(sample = c("a", "b")))
))
fixture_package(lib, "proteinData", "2.1.0", lazy = FALSE, datasets = list(
counts_prot = list(counts_prot = matrix(7:12, 2)),
meta_prot = list(meta_prot = data.frame(sample = c("a", "b")))
))data_attach() takes the name of an installed data
package. It records what the package contains and puts its objects on
the search path under their own names. This reads package metadata only:
no data values are read, and nothing is installed. Nothing is attached
unless you call this function.
Use the objects by their ordinary names. The first use of an object reads it.
data_config(messages = TRUE)
counts_rna
#> projectData: opening resource 'counts_rna' of collection 'rnaseqData' (rnaseqData 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 1 3 5
#> [2,] 2 4 6The message names the resource and says it is opened from the
installed package; projectData never installs packages. The same code
works in a script after the data_attach() call.
Attach another package whenever you need it. Objects already in use are not affected.
data_attach("proteinData", lib.loc = lib)
counts_prot
#> projectData: opening resource 'counts_prot' of collection 'proteinData' (proteinData 2.1.0) with object(s) counts_prot [109 bytes on disk: data/counts_prot.rda]. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 7 9 11
#> [2,] 8 10 12These read only recorded metadata. They never open any data.
data_catalog()[, c("collection", "exposed_name", "resource", "state")]
#> <projectData catalog>: 4 row(s); metadata only, no data opened.
#> collection exposed_name resource state
#> rnaseqData counts_rna counts_rna ready
#> rnaseqData meta_rna meta_rna unopened
#> proteinData counts_prot counts_prot ready
#> proteinData meta_prot meta_prot unopenedstate is unopened until the object has been
used, then ready. The resource column is the
loadable unit, which can differ from the object name. A resource that
produces several objects is opened once for all of them.
data_info("counts_rna") returns the full provenance row
for a name: package, version, installed location, source file, where it
was discovered (origin), and whether its object names have
been checked (mapping).
data_help("counts_rna") opens the help page of the original
topic in the source package. It is separate from inspection and never
runs implicitly.
Bare names are convenient, but data_get() returns an
object without relying on what is on the search path, so it is the more
robust choice inside functions and scripts. (Attached bindings can also
be evaluated by IDE completion or previews; see
system.file("ide-repro", package = "projectData").)
data_get("meta_prot")
#> projectData: opening resource 'meta_prot' of collection 'proteinData' (proteinData 2.1.0) with object(s) meta_prot [145 bytes on disk: data/meta_prot.rda]. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 b
data_get("meta_rna", collection = "rnaseqData")
#> projectData: opening resource 'meta_rna' of collection 'rnaseqData' (rnaseqData 1.0.0) with object(s) meta_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 bGive several names to open several objects in one call. This is a good way to open large objects at the start of a batch job: the first-access messages appear together, before the rest of the analysis runs.
Sometimes you want a single object and no new names on the search
path, much as you would write package::object for an
exported object. Give data_get() the
package:
fixture_package(lib, "imagingData", "0.3.0", lazy = TRUE, datasets = list(
counts_img = list(counts_img = matrix(13:18, 2))
))
counts_img <- data_get("counts_img", package = "imagingData", lib.loc = lib)
#> projectData: opening resource 'counts_img' of collection 'imagingData' (imagingData 0.3.0) with object(s) counts_img. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
counts_img
#> [,1] [,2] [,3]
#> [1,] 13 15 17
#> [2,] 14 16 18
"imagingData" %in% data_collections()$package[data_collections()$attached]
#> [1] FALSEThere is nothing to register first. data_get()
inventories the package if necessary (metadata only), then returns the
object. It does not attach anything and does not change collections that
are already attached. The name is looked up in that package itself, so
preferences, primary collections and suffixes you set for attached
access do not affect it.
data_get() still opens the object you
ask for; it is not a metadata query. Use data_catalog() or
data_query() to look without opening. If you attach the
package later, the object is not read again.
Before each resource is read for the first time, projectData prints a
message naming the collection, the objects, the size of the source file
when it is a single file, and the loader. For large data this is the
moment to notice what is about to load. Turn the messages off with
data_config(messages = FALSE); the setting lasts for the
session.
data_query() shows what a package contains, and which
names would collide with the packages you list or with what is already
attached, without attaching anything or remembering
anything. Use it to look before you attach.
data_detach()
fixture_package(lib, "rnaseqBatch2", "1.0.0", lazy = TRUE, datasets = list(
counts_rna = list(counts_rna = matrix(101:106, 2)),
meta_rna = list(meta_rna = data.frame(sample = c("c", "d")))
))
data_query(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib,
id = c("batch1", "batch2"))[, c("collection", "exposed_name", "binding")]
#> <projectData catalog>: 4 row(s); metadata only, no data opened.
#> collection exposed_name binding
#> batch1 counts_rna ambiguous
#> batch1 meta_rna ambiguous
#> batch2 counts_rna ambiguous
#> batch2 meta_rna ambiguous
nrow(data_collections())
#> [1] 0The query already shows the problem: counts_rna and
meta_rna exist in both packages, so their
binding is ambiguous. The query changed
nothing: no collections are attached.
When two attached collections define the same name, projectData never
resolves it by attachment order. Attaching still succeeds, and the
catalog flags the conflict. Using the bare name fails with an error that
lists the candidates. Use id to give each collection a name
of your own:
data_attach(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib, id = c("batch1", "batch2"))
counts_rna
#> Error:
#> ! 'counts_rna' is ambiguous: it is provided by collections 'batch1', 'batch2'. Resolve with data_prefer("counts_rna", "batch1"), designate a primary collection with data_primary(), use a suffix alias if one was registered, or access it explicitly with data_get("counts_rna", collection = "batch1").There are three ways to resolve conflicts, and they can be combined.
Per-object preference. Choose the collection for a particular name. This is the right tool when an object from one package is always preferred over an object with the same name in another package.
data_prefer("counts_rna", "batch1")
counts_rna
#> projectData: opening resource 'counts_rna' of collection 'batch1' (rnaseqData 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 1 3 5
#> [2,] 2 4 6Other contested names stay unresolved until you decide them:
meta_rna
#> Error:
#> ! 'meta_rna' is ambiguous: it is provided by collections 'batch1', 'batch2'. Resolve with data_prefer("meta_rna", "batch1"), designate a primary collection with data_primary(), use a suffix alias if one was registered, or access it explicitly with data_get("meta_rna", collection = "batch1").A primary collection. If one collection should win
all its contested names, designate it. A per-object preference
for a non-primary collection overrides the primary designation for that
name. A primary collection can also be set when attaching:
data_attach("rnaseqData", id = "batch1", primary = "batch1").
data_primary("batch1")
meta_rna
#> projectData: opening resource 'meta_rna' of collection 'batch1' (rnaseqData 1.0.0) with object(s) meta_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 bSuffix aliases and qualified access. Qualified
access, data_get(name, collection = ), is the most robust
way to access objects within scripts or functions. It returns the named
object and also reaches the copy that lost a conflict.
data_get("counts_rna", collection = "batch2")
#> projectData: opening resource 'counts_rna' of collection 'batch2' (rnaseqBatch2 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 101 103 105
#> [2,] 102 104 106A suffix exposes every object in a collection as an alias of the form
<name><suffix>. Set it when you attach.
data_attach("rnaseqBatch2", lib.loc = lib, id = "batch2", suffix = ".batch2",
replace = TRUE)
counts_rna.batch2
#> projectData: opening resource 'counts_rna' of collection 'batch2' (rnaseqBatch2 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 101 103 105
#> [2,] 102 104 106Suffix aliases must be unique. A clash is reported when you attach, before anything is changed.
To clear decisions, use data_prefer("counts_rna", NULL)
and data_primary(NULL).
A package that ships non-lazy data scripts (data/*.R) is
opened by running the script, only on first access. The script sees the
functions and data of the standard startup packages (stats, graphics,
grDevices, utils, datasets, methods and base, in the order of an
ordinary R session), so head() or
model.matrix() work unqualified. That set is fixed; it does
not follow your defaultPackages option or your search path.
Your workspace and any other attached package are not visible, so
qualify calls to other packages (otherpkg::fn()). This
isolates names; it is not a security sandbox. See the “Script datasets”
section of ?data_attach.
data_detach() removes the attached collections and the
environment projectData created, and forgets them. It does not delete
files or modify objects you already hold, so an object kept in a
variable can keep reading its backing files after detachment. That is
not a promise about replacing or removing the installed package: a
disk-backed object that references installed files may stop working
then. See ?data_detach. data_detach("batch2")
removes just one collection. Packages you only reached through
data_get(package = ) are forgotten the same way (they were
never on the search path); a later data_get() simply opens
them again.
conflicted resolves conflicts between
package exports and library(include.only = ) selects
exports. They work with exported data but do not provide collection
inventory, deferred access beyond lazy data, format-specific
restoration, or provenance.