Possible Table Structures

Suppose you have two patients (i.e., “1” and “2”) with three intake variables (i.e., height, weight, and bmi). If you record this on a piece of paper, it would probably look like Table 1. The table’s grain is “patient”, because each row represents a distinct patient. Understanding the grain of each structure below will help you understand how the structures are re-expressions of the same set of observations.

Terminology

One REDCapR Call for Each Table

The tidy datasets represented in Tables 1, 3a, and 3b can be obtained by calling REDCapR three times –one call per table. Using the forms parameter, pass “intake” to get Table 1, “blood_pressure” to get Table 3a, and “laboratory” to get Table 3b.

Although it is not required, we recommend specifying a readr::cols() object to ensure the desired variable data types.

col_types_intake <-
  readr::cols_only(
    record_id                 = readr::col_integer(),
    height                    = readr::col_double(),
    weight                    = readr::col_double(),
    bmi                       = readr::col_double()
  )

ds_intake <-
  REDCapR::redcap_read(
    redcap_uri  = credential$redcap_uri, # From the previous code snippet.
    token       = credential$token,
    forms       = c("intake"),
    col_types   = col_types_intake,
    verbose     = FALSE,
  )$data

ds_intake

col_types_blood_pressure <-
  readr::cols(
    record_id                 = readr::col_integer(),
    redcap_repeat_instrument  = readr::col_character(),
    redcap_repeat_instance    = readr::col_integer(),
    sbp                       = readr::col_double(),
    dbp                       = readr::col_double(),
    blood_pressure_complete   = readr::col_integer()
  )

ds_blood_pressure <-
  REDCapR::redcap_read(
    redcap_uri  = credential$redcap_uri,
    token       = credential$token,
    forms       = c("blood_pressure"),
    col_types   = col_types_blood_pressure,
    verbose     = FALSE
  )$data

ds_blood_pressure %>%
  tidyr::drop_na(redcap_repeat_instrument)

col_types_laboratory  <-
  readr::cols(
    record_id                 = readr::col_integer(),
    redcap_repeat_instrument  = readr::col_character(),
    redcap_repeat_instance    = readr::col_integer(),
    lab                       = readr::col_character(),
    conc                      = readr::col_character(),
    laboratory_complete       = readr::col_integer()
  )

ds_laboratory  <-
  REDCapR::redcap_read(
    redcap_uri  = credential$redcap_uri,
    token       = credential$token,
    forms       = c("laboratory"),
    col_types   = col_types_laboratory,
    verbose     = FALSE
  )$data

ds_laboratory %>%
  tidyr::drop_na(redcap_repeat_instrument)

ds_block <-
  REDCapR::redcap_read(
    redcap_uri  = credential$redcap_uri,
    token       = credential$token,
    col_types   = readr::cols(.default = readr::col_character()),
    verbose     = FALSE,
  )$data

ds_block

One REDCapTidieR Call for All Tables

REDCapTidieR’s initial motivation is to facilitate longitudinal analyses and promote tidy data hygiene.

{Stephan Kadauke & Richard Hanna, please represent your package as you wish. Tell me if I’ve positioned it differently than you would have.}

Choosing between the Approaches

When retrieving data from REDCap, we recommend calling REDCapTidieR in many scenarios, such as:

• you are new to managing or analyzing data with R, or
• your analyses will require most of the dataset’s rows or columns, or
• you’d benefit from some of the auxiliary information in REDCapTidieR’s supertibble, such as the instrument’s structure.

However we recommend calling REDCapR in other scenarios. It could be worth calling REDCapR multiple times if:

• you are performing some operation other than retrieving/reading data from REDCap,
• you are comfortable with managing and analyzing data with R, or
• your analyses require only a fraction of the data (such as (a) you need only the first event, or (b) the analyses don’t involve most of the instruments), or
• you want to specify the variables’ data types with readr::cols().

If in doubt, start with REDCapTidieR. Escalate to REDCapR if your download time is too long and might be decreased by reducing the information retrieved from the server and transported across the network.

And of course many scenarios are solved best with a combination of both packages, such as (a) REDCapR populates the initial demographics in REDCap, (b) research staff enter measures collected from patients over time, (c) REDCapTidieR retrieves the complete longitudinal dataset, (d) dplyr joins the tibbles, and finally (e) lme4 tests hypotheses involving patient trajectories over time.

Escalating to REDCapR

Even if you think you’ll need REDCapR’s low-level control, consider starting with REDCapTidieR anyway. …particularly if you are unsure how to specify the grain of each table. The structure of REDCapTidieR’s tables easily compatible with conventional analyses. If you need the performance of REDCapR but are unsure how the tables should look, simply execute something like REDCapTidieR::redcap_read_tidy(url, project_token) and study its output. Then try to mimic it exactly with REDCapR::redcap_read() calls.

Finally, cull unwanted cells using the parameters of REDCapR::redcap_read(). These data points will not even leave the REDCap instance, which will improve performance. Some possible strategies include passing arguments to

• forms: this will retrieve only the specified instruments/forms. Beginners should start here. It is easy to conceptualize and usually has a big increase in speed for just a little development work.
• events: this will retrieve only the desired events within a longitudinal project. For instance if the analyses involve only the “intake” and “follow up #1” events, leave follow ups #2, #3, and #4 on the server.
• fields: this is more granular than forms. It can be combined with calls to forms, such as passing "bmi" to fields and c("blood_pressure", "laboratory") to forms.
• records: in the example scenario, this will pluck individual patients and their associated events and repeating instances. To be useful in research, it’s usually combined with other REDCapR calls. See the Read a subset of records, conditioned on the values in some variables section of REDCapR’s Basic REDCapR Operations vignette.
• filter_logic: this will leverage the observation values to limit the rows returned. Because the underlying table does not index the obs values, this will be less computationally efficient than the options above.
• datetime_range_begin & datetime_range_end: this will return only the records that have been created or modified during the specified window.

Note that the efficiency gain from moving from the block dataset to REDCapTidieR is different than the gain from moving from REDCapTidieR to REDCapR. When moving to from Table 5 to a REDCapTidieR Supertibble, you are eliminating empty cells that will never contain worthwhile data. When moving from a REDCapTidieR Supertibble call to a collection of REDCapR calls, you are eliminating cells that contain data, but may not be relevant to your analysis (such as a patient’s name or the time a lab specimen was collected). {This paragraph needs work.}

Longitudinal

{TODO: Generalize the argument to longitudinal events}

Caching Results to Improve Performance

If escalating to REDCapR didn’t decrease the duration enough, consider the strategy of calling REDCap only once per day (with either package) and saving the results to a secured disk. This will be efficient when the analyses require a large dataset, but not a real-time dataset.

In many retrospective projects, multiple analyses during the day can reuse the same dataset retrieved the night before (with a function like readr::write_rds()). This has helped our complicated investigations when multiple statisticians frantically tackle overlapping aspects before a funder’s deadline.

Essentially you are transferring security responsibilities from REDCap to the file server. To balance its advantages, the approach’s drawbacks include:

• extra effort is required to create, secure, and maintain a networked drive that is accessible to only authorized analysts. This will take 15-60 minutes, depending on your institution.
• extra effort is required to create an automated program to retrieve and cache the dataset every night. See [CDS] for a head start.
• REDCap’s logs will not support a meaningful audit because no human is accessing the API. User access audits are now the responsibility of the file system.

Caching Results for other Languages

Packages for other programming languages have been developed that access REDCap’s API, such PyCap and PhpCap. Please see a more complete list at https://redcap-tools.github.io/projects/. The caching strategy described above may also benefit your investigation if:

• it uses a language like SAS that does not support packages. We strongly discourage deploying loose scripts that call the API. When something is fixed in the original script, it must be distributed by the developer to the ~50 client institutions, then it must by copied to the ~100 projects within each institution. In contrast, when a developer updates a package distributed through a repository like CRAN, the client needs to run only base::update.packages() once per machine to update all its packages –not just REDCap packages. Data scientists already do this regularly.

• it uses a language like Julia that lacks a stable or updated package. The REDCap developers are always releasing useful features (e.g., repeated measures) that improve the quality and efficiency of your research. Leverage them.

• your investigation incorporates multiple programming languages and you would like the analysis team to use a consistent dataset. In this scenario, we recommend the code save the same R dataset in multiple file formats, such as a csv file and a feather file.