--- title: "medExtractR Vignette" date: '`r Sys.Date()`' output: pdf_document --- ```{r setup, include = FALSE} knitr::opts_chunk$set(echo = TRUE) library(ggplot2) ``` ## Introduction The `medExtractR` package uses a natural language processing (NLP) system called *medExtractR*.$^{1}$ This system is a medication extraction system that uses regular expressions and rule-based approaches to identify key dosing information including drug name, strength, dose amount, frequency or intake time, dose change, and last dose time. Function arguments can be specified to allow the user to tailor the `medExtractR` system to the particular drug or dataset of interest, improving the quality of extracted information. The `medExtractR` system forms the basis of the *Extract-Med* module in Choi *et al.*'s$^{2}$ pipeline approach for performing pharmacokinetic/pharmacodynamic (PK/PD) analyses using electronic health records (EHRs). This approach and corresponding R package, `EHR`,$^{3}$ convert raw output from `medExtractR` into a format that is usable for PK/PD analyses. Since `medExtractR` is integral to the *Extract-Med* module in `EHR`, parts of this vignette are taken and adapted from the `EHR` package vignette. ## Basic `medExtractR` The function `medExtractR` is primarily responsible for identifying and creating search windows for all mentions of the drug of interest within a note. This function then calls the `extract_entities` subfunction, which identifies and extracts entities within the search window. The entities that can be identified with the basic version of `medExtractR` include: drug name (entity name in output: "DrugName"), strength ("Strength"), dose amount ("DoseAmt"), dose given intake ("DoseStrength"), frequency ("Frequency"), intake time ("IntakeTime"), keywords indicating an increase or decrease in dose ("DoseChange"), route of administration ("Route"), duration of dosing regimen ("Duration"), and time of last dose ("LastDose"). In order to run `medExtractR`, certain function arguments must be specified, including: - `note`: A character string containing the note on which you want to run `medExtractR`. - `drug_names`: Names of the drugs for which we want to extract medication dosing information. This can include any way in which the drug name might be represented in the clinical note, such as generic name (e.g., `"lamotrigine"`), brand name (e.g., `"Lamictal"`), or an abbreviation (e.g., `"LTG"`). - `unit`: The unit of the drug(s) listed in `drug_names`, for example `"mg"`. - `window_length`: Length of the search window around each found drug name in which to search for dosing information. There is no default for this argument, requiring the user to carefully consider its value through tuning (see tuning section below). - `max_dist`: The maximum edit distance allowed when identifying `drug_names`. Maximum edit distance determines the difference between two strings, and is defined as the number of insertions, deletions, or substitutions required to change one string into the other. This allows us to capture misspellings in the drug names we are searching for, and its value should be carefully considered through tuning (see tuning section below). - The default value is '0', or exact spelling matches to `drug_names`. A value of 0 is always used for drug names with less than 5 characters regardless of the value set by `max_dist`. - A value of 1 would capture mistakes such as a single missing or extra letter, e.g., "tacrlimus" or "tacroolimus" instead of "tacrolimus" - A value of 2 would capture these mistakes or a single transposition, e.g., "tcarolimus" instead of "tacrolimus" - Higher values (3 or above) would capture increasingly more severe mistakes, though setting the value too high can cause similar words to be mistaken as the drug name, likely increasing the false positive rate. Generally, the function call to `medExtractR` is ```{r, eval = FALSE} note <- paste(scan(filename, '', sep = '\n', quiet = TRUE), collapse = '\n') medExtractR(note, drug_names, unit, window_length, max_dist, ...) ``` where `...` refers to additional arguments to `medExtractR`. Examples of additional arguments include: - `drug_list`, a list of other drug names (besides the drug names of interest). This list is used to shorten the search window in which `medExtractR` looks for dosing entities by truncating at the nearest mentions of a competing drug name. By default, this calls `rxnorm_druglist`, a partially cleaned and processed list of brand name and ingredient drug names in the RxNorm database.$^{4}$ This list could also incorporate other competing information besides drug names, such as drug abbreviations, symptoms, procedures, or names of laboratory measurements. - `strength_sep`, where users can specify special characters to separate doses administered at different times of day. For example, consider the drug name *"lamotrigine"* and the phrase *"Patient is on lamotrigine 200-300"*, indicating that the patient takes 200 mg of the drug in the morning and 300 mg in the evening. Setting `strength_sep = c('-')` would allow `medExtractR` to identify the expression *200-300* as "DoseStrength" (i.e., dose given intake) since they are separated by the special character "-". The default value is `NULL`. - `lastdose`, a logical input specifying whether or not the last dose time entity should be extracted. Default value is `FALSE`. - `_dict` and `_fun`, where `` is a dictionary-based entity (e.g., frequency, intake time, route, duration). These optional arguments allow for user-customized dictionaries and extraction functions. Default dictionaries are provided within `medExtractR`, as is a default extraction function (`extract_generic`). As mentioned above, some arguments to `medExtractR` should be specified through a tuning process. In a later section, we briefly describe the process by which a user could tune the `medExtractR` system using a validated gold standard dataset. ## Running `medExtractR` Below, we demonstrate how to run `medExtractR` using sample notes for two drugs: tacrolimus (simpler prescription patterns, used to prevent rejection after organ transplant) and lamotrigine (more complex prescription patterns, used to treat epilepsy). The arguments specified for each drug here were determined based on training sets of 60 notes for each drug.$^{1}$ We specify `lastdose=TRUE` for tacrolimus to extract information about time of last dose, and `strength_sep="-"` for lamotrigine which can have varying doses depending on the time of day. ```{r mxr, message = FALSE} library(medExtractR) # tacrolimus note file names tac_fn <- list( system.file("examples", "tacpid1_2008-06-26_note1_1.txt", package = "medExtractR"), system.file("examples", "tacpid1_2008-06-26_note2_1.txt", package = "medExtractR"), system.file("examples", "tacpid1_2008-12-16_note3_1.txt", package = "medExtractR") ) # execute medExtractR tac_mxr <- do.call(rbind, lapply(tac_fn, function(filename){ tac_note <- paste(scan(filename, '', sep = '\n', quiet = TRUE), collapse = '\n') fn <- sub(".+/", "", filename) cbind("filename" = fn, medExtractR(note = tac_note, drug_names = c("tacrolimus", "prograf", "tac", "tacro", "fk", "fk506"), unit = "mg", window_length = 60, max_dist = 2, lastdose=TRUE)) })) # lamotrigine note file name lam_fn <- c( system.file("examples", "lampid1_2016-02-05_note4_1.txt", package = "medExtractR"), system.file("examples", "lampid1_2016-02-05_note5_1.txt", package = "medExtractR"), system.file("examples", "lampid2_2008-07-20_note6_1.txt", package = "medExtractR"), system.file("examples", "lampid2_2012-04-15_note7_1.txt", package = "medExtractR") ) # execute medExtractR lam_mxr <- do.call(rbind, lapply(lam_fn, function(filename){ lam_note <- paste(scan(filename, '', sep = '\n', quiet = TRUE), collapse = '\n') fn <- sub(".+/", "", filename) cbind("filename" = fn, medExtractR(note = lam_note, drug_names = c("lamotrigine", "lamotrigine XR", "lamictal", "lamictal XR", "LTG", "LTG XR"), unit = "mg", window_length = 130, max_dist = 1, strength_sep="-")) })) ``` The format of raw output from the `medExtractR` function is a `data.frame` with 3 columns: - `entity`: The label of the entity for the extracted expression. - `expr`: Expression extracted from the clinical note. - `pos`: Position of the extracted expression in the note, in the format `startPosition:stopPosition`. Note that we slightly modify the stop position by adding one to avoid output for single-character entities appearing to have zero length (for example, `entity expr pos` output of `DoseAmt 2 33:33`) In the output presented below, we manually attached the corresponding file name to each note's output before combining results across notes. ```{r, echo = FALSE} # Print output message("tacrolimus `medExtractR` output:\n") tac_mxr message("lamotrigine `medExtractR` output:\n") lam_mxr ``` For the tacrolimus output, we chose to also extract the last dose time entity by specifying `lastdose=TRUE`. The last dose time entity is extracted as raw character expressions from the clinical note, and must first be converted to a standardized datetime format. The `EHR`$^{3}$ package provides for parsing and standardizing raw `medExtractR` last dose times when laboratory measurements are available with its `processLastDose` function. ## Tuning the `medExtractR` system In a previous section, we mentioned that parameters within the `medExtractR` should be tuned in order to ensure higher quality of extracted drug information. This section provides recommendations for how to implement this tuning procedure. In order to tune `medExtractR`, we recommend selecting a small set of tuning notes, from which the parameter values can be selected. Below, we describe this process with a set of three notes (note that these notes were chosen for the purpose of demonstration, and we recommend using tuning sets of at least 10 notes). Once a set of tuning notes has been curated, they must be manually annotated by reviewers to identify the information that should be extracted. This process produces a gold standard set of annotations, which identify the correct drug information of interest. This includes entities like the drug name, strength, and frequency. For example, in the phrase $$\text{Patient is taking } \textbf{lamotrigine} \text{ } \textit{300 mg} \text{ in the } \underline{\text{morning}} \text{ and } \textit{200 mg} \text{ in the }\underline{\text{evening}}$$ bolded, italicized, and underlined phrases represent annotated drug names, dose strength (i.e., dose given intake), and intake times, respectively. These annotations are stored as a dataset. First, we read in the annotation files for three example tuning notes, which can be generated using an annotation tool, such as the Brat Rapid Annotation Tool (BRAT) software.$^{5}$ By default, the output file from BRAT is tab delimited with 3 columns: an annotation identifier, a column with labeling information in the format "label startPosition stopPosition", and the annotation itself, as shown in the example below: ```{r, echo = FALSE} ann <- read.delim(system.file("mxr_tune", "ann_example.ann", package = "medExtractR"), header = FALSE, sep = "\t", stringsAsFactors = FALSE, col.names = c("id", "entity", "annotation")) head(ann) ``` In order to compare with the `medExtractR` output, the format of the annotation dataset should be four columns with: 1. The file name of the corresponding clinical note 2. The entity label of the annotated expression 3. The annotated expression 4. The start and stop position of the annotated expression in the format "start:stop" The exact formatting performed below is specific to the format of the annotation files, and may vary if an annotation software other than BRAT is used. ```{r} # Read in the annotations - might be specific to annotation method/software ann_filenames <- list(system.file("mxr_tune", "tune_note1.ann", package = "medExtractR"), system.file("mxr_tune", "tune_note2.ann", package = "medExtractR"), system.file("mxr_tune", "tune_note3.ann", package = "medExtractR")) tune_ann <- do.call(rbind, lapply(ann_filenames, function(fn){ annotations <- read.delim(fn, header = FALSE, sep = "\t", stringsAsFactors = FALSE, col.names = c("id", "entity", "annotation")) # Label with file name annotations$filename <- sub(".ann", ".txt", sub(".+/", "", fn), fixed=TRUE) # Separate entity information into entity label and start:stop position # Format is "entity start stop" ent_info <- strsplit(as.character(annotations$entity), split="\\s") annotations$entity <- unlist(lapply(ent_info, '[[', 1)) annotations$pos <- paste(lapply(ent_info, '[[', 2), lapply(ent_info, '[[', 3), sep=":") annotations <- annotations[,c("filename", "entity", "annotation", "pos")] return(annotations) })) head(tune_ann) ``` To select appropriate tuning parameters, we identify a range of possible values for each of the `window_length` and `max_dist` parameters. Here, we allow `window_length` to vary from 30 to 120 characters in increments of 30, and `max_dist` to take a value of 0, 1, or 2. We then obtain the `medExtractR` results for each combination. ```{r run_mxr, cache = TRUE} wind_len <- seq(30, 120, 30) max_edit <- seq(0, 2, 1) tune_pick <- expand.grid("window_length" = wind_len, "max_edit_distance" = max_edit) # Run the Extract-Med module on the tuning notes note_filenames <- list(system.file("mxr_tune", "tune_note1.txt", package = "medExtractR"), system.file("mxr_tune", "tune_note2.txt", package = "medExtractR"), system.file("mxr_tune", "tune_note3.txt", package = "medExtractR")) # List to store output for each parameter combination mxr_tune <- vector(mode="list", length=nrow(tune_pick)) for(i in 1:nrow(tune_pick)){ mxr_tune[[i]] <- do.call(rbind, lapply(note_filenames, function(filename){ tune_note <- paste(scan(filename, '', sep = '\n', quiet = TRUE), collapse = '\n') fn <- sub(".+/", "", filename) cbind("filename" = fn, medExtractR(note = tune_note, drug_names = c("tacrolimus", "prograf", "tac", "tacro", "fk", "fk506"), unit = "mg", window_length = tune_pick$window_length[i], max_dist = tune_pick$max_edit_distance[i])) })) } ``` Finally, we determine which parameter combination yielded the highest performance, quantified by some metric. For our purpose, we used the F1-measure (F1), the harmonic mean of precision $\left(\frac{\text{true positives}}{\text{true positives + false positives}}\right)$ and recall $\left(\frac{\text{true positives}}{\text{true positives + false negatives}}\right)$. Tuning parameters were selected based on which combination maximized F1 performance within the tuning set. The code below determines true positives as well as false positives and negatives, used to compute precision, recall, and F1. ```{r, echo=FALSE} # Functions to compute true positive, false positive, and false negatives # number of true positives - how many annotations were correctly identified by medExtractR Tpos <- function(df){ sum(df$annotation == df$expr, na.rm=TRUE) } # number of false positive (identified by medExtractR but not annotated) Fpos <- function(df){ sum(is.na(df$annotation)) } # number of false negatives (annotated but not identified by medExtractR) Fneg <- function(df){ # keep only rows with annotation df_ann <- subset(df, !is.na(annotation)) sum(is.na(df$expr)) } prf <- function(df){ tp <- Tpos(df) fp <- Fpos(df) fn <- Fneg(df) precision <- tp/(tp + fp) recall <- tp/(tp + fn) f1 <- (2*precision*recall)/(precision + recall) return(f1) } tune_pick$F1 <- sapply(mxr_tune, function(x) { y <- merge(x, tune_ann, by = c("filename", "entity", "pos"), all = TRUE) compare <- y[order(as.numeric(gsub(":.+", "", y[,'pos']))),] prf(compare) }) ggplot(tune_pick) + geom_point(aes(max_edit_distance, window_length, size = F1)) + scale_y_continuous(breaks=seq(30,120,30)) + annotate("text", x = tune_pick$max_edit_distance+.2, y = tune_pick$window_length, label = round(tune_pick$F1, 2)) + ggtitle("F1 for tuning parameter values") ``` The plot shows that the highest F1 achieved was 1, and occurred for three different combinations of parameter values: a maximum edit distance of 2 and a window length of 60, 90, or 120 characters. The relatively small number of unique F1 values is likely the result of only using 3 tuning notes. In this case, we would typically err on the side of allowing a larger search window and decide to use a maximum edit distance of 2 and a window length of 120 characters. In a real-world tuning scenario and with a larger tuning set, we would also want to test longer window lengths since the best case scenario occurred at the longest window length we used. Additional information for the tuning process of `medExtractR` can be found in Weeks *et al.*$^{1}$ ## References 1. Weeks HL, Beck C, McNeer E, Williams ML, Bejan CA, Denny JC, Choi L. medExtractR: A targeted, customizable approach to medication extraction from electronic health records. Journal of the American Medical Informatics Association. 2020 Mar;27(3):407-18. doi: 10.1093/jamia/ocz207. 2. Choi L, Beck C, McNeer E, Weeks HL, Williams ML, James NT, Niu X, Abou-Khalil BW, Birdwell KA, Roden DM, Stein CM. Development of a System for Post-marketing Population Pharmacokinetic and Pharmacodynamic Studies using Real-World Data from Electronic Health Records. Clinical Pharmacology & Therapeutics. 2020 Apr;107(4):934-43. doi: 10.1002/cpt.1787. 3. Choi L, Beck C, Weeks HL, and McNeer E (2020). EHR: Electronic Health Record (EHR) Data Processing and Analysis Tool. R package version 0.3-1. https://CRAN.R-project.org/package=EHR 4. Nelson SJ, Zeng K, Kilbourne J, Powell T, Moore R. Normalized names for clinical drugs: RxNorm at 6 years. Journal of the American Medical Informatics Association. 2011 Jul-Aug;18(4)441-8. doi: 10.1136/amiajnl-2011-000116. Epub 2011 Apr 21. PubMed PMID: 21515544; PubMed Central PMCID: PMC3128404. 5. Stenetorp P, Pyysalo S, Topić G, Ohta T, Ananiadou S, Tsujii JI. BRAT: a web-based tool for NLP-assisted text annotation. InProceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics 2012 Apr 23 (pp. 102-107). Association for Computational Linguistics.