| Title: | Query Curated 'DuckDB' Databases Built from Public Data |
|---|---|
| Description: | Connects to the 'datapond' registry of curated 'DuckDB' databases built from public government and research data (immigration courts, campaign finance, clinical trials, Medicare, and more). Databases are attached remotely over HTTP so only the byte ranges a query touches are transferred, or downloaded once for local use. Returns standard 'DBI' connections that work with 'dbplyr'. |
| Authors: | Ian Nason [aut, cre] |
| Maintainer: | Ian Nason <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-09-27 18:40:43 UTC |
| Source: | https://github.com/cran/datapond |
Opens an in-memory DuckDB connection and attaches each requested database
read-only, either remotely from its attach_url (the default; DuckDB's
httpfs extension fetches only the byte ranges a query touches, so there is
no full download) or from a local copy made with dp_download().
dp_connect(id, local = FALSE, quiet = FALSE)dp_connect(id, local = FALSE, quiet = FALSE)
id |
One or more database ids (see |
local |
Attach local copies under |
quiet |
Suppress progress messages? |
With a single id the database is made the default catalog (USE), so
tables can be referenced unqualified. With several ids, qualify tables with
the database id, double-quoting ids that contain a hyphen:
"cms-medicare".physician_summary.
A duckdb_connection (a DBIConnection). Close it with
dp_disconnect().
## Not run: con <- dp_connect("eoir") DBI::dbGetQuery(con, "SELECT * FROM proceedings LIMIT 5") dp_disconnect(con) con <- dp_connect(c("cms-medicare", "openpayments")) DBI::dbGetQuery(con, 'SELECT * FROM "cms-medicare".physician_summary LIMIT 5') dp_disconnect(con) ## End(Not run)## Not run: con <- dp_connect("eoir") DBI::dbGetQuery(con, "SELECT * FROM proceedings LIMIT 5") dp_disconnect(con) con <- dp_connect(c("cms-medicare", "openpayments")) DBI::dbGetQuery(con, 'SELECT * FROM "cms-medicare".physician_summary LIMIT 5') dp_disconnect(con) ## End(Not run)
The registry is cached under tools::R_user_dir("datapond", "cache") and
downloaded databases live under tools::R_user_dir("datapond", "data").
Override either with options(datapond.cache_dir = ...) /
options(datapond.data_dir = ...). Setting
options(datapond.data_dir = "~/.datapond") shares downloads with the
Python package, which names local files the same way (<id>.duckdb).
dp_data_dir() dp_cache_dir()dp_data_dir() dp_cache_dir()
A path (character scalar).
dp_data_dir()dp_data_dir()
Every datapond database ships _metadata (one row per table) and
_columns (types, null rates, example values, join hints). dp_describe()
reads them and falls back to information_schema when they are absent.
dp_describe(x, table = NULL, search = NULL)dp_describe(x, table = NULL, search = NULL)
x |
A database id, or an open connection from |
table |
Describe the columns of this table instead of listing tables. |
search |
Find columns whose name contains this text (case-insensitive). |
A data frame (a tibble when the tibble package is installed).
## Not run: dp_describe("eoir") dp_describe("eoir", table = "proceedings") dp_describe("eoir", search = "judge") ## End(Not run)## Not run: dp_describe("eoir") dp_describe("eoir", table = "proceedings") dp_describe("eoir", search = "judge") ## End(Not run)
Close a datapond connection
dp_disconnect(con)dp_disconnect(con)
con |
A connection from |
TRUE, invisibly.
Downloads the full .duckdb file so later queries run at disk speed.
Files are saved as <id>.duckdb under dp_data_dir() (or path), which is
where dp_connect(id, local = TRUE) looks for them.
dp_download(id, path = NULL, quiet = FALSE, resume = TRUE)dp_download(id, path = NULL, quiet = FALSE, resume = TRUE)
id |
Database id. |
path |
Destination file or directory. Defaults to |
quiet |
Suppress the progress bar? |
resume |
Resume a partial download if one exists? |
The local path, invisibly.
## Not run: dp_download("dol-visas") con <- dp_connect("dol-visas", local = TRUE) ## End(Not run)## Not run: dp_download("dol-visas") con <- dp_connect("dol-visas", local = TRUE) ## End(Not run)
Information about one database
dp_info(id)dp_info(id)
id |
Database id. |
The registry entry (a list of class datapond_db), printed in a
readable layout.
## Not run: dp_info("eoir") ## End(Not run)## Not run: dp_info("eoir") ## End(Not run)
List available databases
dp_list() dp_databases()dp_list() dp_databases()
dp_list() returns a character vector of database ids;
dp_databases() returns one row per database with its registry fields.
## Not run: dp_list() dp_databases()[, c("id", "rows", "size_gb")] ## End(Not run)## Not run: dp_list() dp_databases()[, c("id", "rows", "size_gb")] ## End(Not run)
Path of a locally downloaded database
dp_local_path(id, must_exist = FALSE)dp_local_path(id, must_exist = FALSE)
id |
Database id (see |
must_exist |
Error if the file is not present? |
A path (character scalar).
dp_local_path("eoir")dp_local_path("eoir")
Fetches registry.json from GitHub (cached for an hour under
dp_cache_dir(); a stale cache is used if the network is unavailable).
dp_registry(refresh = FALSE)dp_registry(refresh = FALSE)
refresh |
Ignore the cache and fetch again? |
The parsed registry: a list with a databases element, one list
per database.
## Not run: reg <- dp_registry() length(reg$databases) ## End(Not run)## Not run: reg <- dp_registry() length(reg$databases) ## End(Not run)
Compares the registry's updated date with the local file's modification
time and re-downloads when the registry is newer.
dp_update(id, quiet = FALSE)dp_update(id, quiet = FALSE)
id |
Database id. |
quiet |
Suppress the progress bar? |
The local path, invisibly.