--- title: "Converting a package to use doctest" output: html_document: toc: true toc_depth: 3 toc_float: true theme: flatly vignette: > %\VignetteIndexEntry{Converting a package to use doctest} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` [Doctest](https://hughjonesd.github.io/doctest/) is an R package to let you write doctests. Doctests are chunks of code which act as both examples for your users, and tests of your package's functionality. The doctest package does this by letting you add [testthat](https://testthat.r-lib.org) tests to your [roxygen](https://roxygen2.r-lib.org) documentation. This article gives a real world example of how to convert an R package to use doctest. I'll use [onetime](https://hughjonesd.github.io/onetime/dev/), a small package which lets you run code only once. Onetime 0.1.0 is on CRAN, so we will be dogfooding real production code. To follow along, you'll need to be familiar with both testthat and roxygen. Both packages have documentation and tutorials elsewhere. # Setting up doctest My first step, obviously, was to install the doctest package: ```r remotes::install_github("hughjonesd/doctest") ``` Doctest is not on CRAN yet, but don't worry, because you don't need to add it as a package dependency. You can just use it on your own machine when you build your package. Next, I added the doctest roclet `dt_roclet` to onetime'S DESCRIPTION FILE: ``` Roxygen: list(roclets = c("collate", "rd", "namespace", "doctest::dt_roclet")) ``` Now, whenever I run `devtools::document()`, it will create doctests in the `tests/testthat` directory, as well doing the usual roxygen tasks like writing .Rd files. I already had `tests/testthat` set up so I didn't need to do anything more in this direction. One caveat: if you hit `Ctrl+Shift+D` in RStudio, it won't run the doctest roclet. You need to type `devtools::document()` or `roxygen2::roxygenize()` on the command line. Otherwise your doctest tags won't be recognized. # Converting `@examples` to `@doctest` sections My next step was to "Find in files" all my `@examples` tags and change them to `@doctest` tags. `@doctest` sections create examples in Rd files, just like `@examples` sections. So I expected this to make no difference to the output from `document()`. ### Before ```r #' @examples #' oo <- options(onetime.dir = tempdir(check = TRUE)) #' id <- sample(10000L, 1) #' ... ``` ### After ```r #' @doctest #' oo <- options(onetime.dir = tempdir(check = TRUE)) #' id <- sample(10000L, 1) #' ... ``` Indeed, after I ran `devtools::document()`, my .Rd files were unchanged, apart from a few deleted empty lines in the examples. I judged that these were not important, so I made my first commit. At this stage, you might want to create a new branch for your commits, using `git branch` on the command line, or clicking the "new branch" button in RStudio. I was gung ho, so I just put [my commit](https://github.com/hughjonesd/onetime/commit/1c6400cac3528c27ffd2d397250da527fa2dbf7d?diff=split) on the master branch. I made one exception: I left the `@examples` tag unchanged in the `set_ok_to_store()` function. This is a function with side effects on the user's installation; testing it needs to be done carefully. I thought that a doctest would need too much complex setup, so I left it as is. There is an existing test for `set_ok_to_store()` anyway. # Creating doctests by adding expectations So far, nothing has actually changed. To generate doctests from my examples, I needed to add some *expectations* to my code. If you know testthat, doctest expectations will look very familiar. In testthat, you write: ```r expect_equal(exp(1), 2.71828183) ``` In a `@doctest` roxygen section, this becomes: ```r #' @expect equal(2.71828183) #' exp(1) ``` where `exp(1)` is part of your example code. Similarly, in testthat you might write: ```r expect_warning(mean("foo"), "not numeric") ``` In a `@doctest` section you might write: ```r #' @expect warning("not numeric") #' mean("foo") ``` In other words: * Use the `@expect` tag to create an expectation. * After `@expect`, write a testthat expectation, without the `expect_` prefix. * The next R expression after the `@expect` line becomes the first argument to the expectation. # Doctests for messaging functions I started with onetime's messaging functions, which print a message or warning only once. For example, `onetime_message_confirm()` had the following `@doctest` section: ```r #' @doctest #' oo <- options(onetime.dir = tempdir(check = TRUE)) #' id <- sample(10000L, 1L) #' #' onetime_message_confirm("A message to show one or more times", id = id) #' #' onetime_reset(id = id) #' options(oo) ``` I want to check that when this example code runs, the user indeed sees a message. So I added an expectation: ```r #' @doctest #' oo <- options(onetime.dir = tempdir(check = TRUE)) #' id <- sample(10000L, 1L) #' #' @expect message("A message") #' onetime_message_confirm("A message to show one or more times", id = id) #' #' onetime_reset(id = id) ``` That was simple. To check everything was working, I ran `devtools::document()` again. It produced a new file under `tests/testthat`, called `test-doctest-onetime_message_confirm.R`: ```r # Generated by doctest: do not edit by hand # Please edit file in R/messages.R test_that("Doctest: onetime_message_confirm", { # Created from @doctest for `onetime_message_confirm` # Source file: R/messages.R # Source line: 110 oo <- options(onetime.dir = tempdir(check = TRUE)) id <- sample(10000L, 1L) expect_message(onetime_message_confirm("A message to show one or more times", id = id), "A message") onetime_reset(id = id) options(oo) }) ``` This looked fine, so I ran my tests using `Ctrl+Shift+T`, and the new test passed. Notice the warning in the comment at the top of the test file. If you edit this file, it will be overwritten next time you run the doctest roclet. If you want to edit it manually, you should change its name to something without `doctest`, and remove the `Generated by doctest` stamp. Then you'll just have a normal testthat test which you can edit as you please. If you don't want to regenerate the automated test file again, then remember to edit the relevant `@doctest` section, removing expectations or replacing `@doctest` back with `@examples`. My next doctest was more complex. The roxygen looked like this: ```r #' @doctest ... #' #' for (n in 1:3) { #' onetime_warning("will be shown once", id = id) #' } #' ... ``` It's fine to use expectations inside a for loop, but my problem was that I expected different things each time. `onetime_warning()` shows its warning only the first time it is called. So on the first time round the loop, I would expect a warning. Afterwards I would expect no output. I could have unrolled the loop, like this: ```r #' @expect warning() #' onetime_warning("will be shown once", id = id) #' @expect silent() #' onetime_warning("will be shown once", id = id) #' @expect silent() #' onetime_warning("will be shown once", id = id) ``` But I liked the loop because it made it very clear how `onetime_warning()` worked. I wanted to follow the philosophy "write great documentation, then add tests where appropriate" rather than "turn your documentation into a test suite". So, I bit the bullet and wrote a more complex expectation: ```r #' for (n in 1:3) { #' @expect warning(regexp = if (n == 1L) "once" else NA) #' onetime_warning("will be shown once", id = id) #' } ``` This is a bit ugly. It uses the fact that `expect_warning(regexp = NA)` is equivalent to *not* expecting a warning. So, on the first time round the loop, the expectation checks for a warning matching the string `"once"`; afterwards, it checks that there is no warning. Notice that the `@expect` tag isn't indented. Roxygen tags have to come straight after the starting `#' ` characters, with at most one space. Again, I ran `devtools::document()` and checked the new test: ```r for (n in 1:3) { expect_warning(onetime_warning("will be shown once", id = id), regexp = if (n == 1L) "once" else NA) } ``` Fine. I ran the test and again, it passed. I added some more similar tests and then made a [commit](https://github.com/hughjonesd/onetime/commit/a870ede984f6523b32c0da8a54ca114e9a1bba66?diff=split). I had set up Github actions to run `R CMD check`, so I knew that the tests would also be checked on different platforms. Happily, they all passed. # Adding doctests for utility functions Next I added some doctests for utility functions, which manipulate various aspects of onetime's on-disk records. Mostly these don't print output, so instead I tested their return value. For example, `onetime_been_done()`, which checks if a particular onetime call has already been made, got a doctest like this: ```r #' @expect false() #' onetime_been_done(id = id) #' onetime_message("Creating an ID", id = id) #' @expect true() #' onetime_been_done(id = id) ``` The function `onetime_dir()` is very simple and just returns a file path. Its example was simple too: ```r #' @doctest #' #' onetime_dir("my-folder") #' #' oo <- options(onetime.dir = tempdir(check = TRUE)) #' onetime_dir("my-folder") #' options(oo) ``` I decided to just test the first call to `onetime_dir()`, confirming that the result ended with the subfolder I passed in. The second call would return a temporary directory, which would be different between different R sessions, so I wasn't sure how to test it usefully. In fact, to skip unnecessary code from the test, I used the `@omit` tag: ```r #' @expect match("my-folder$") #' onetime_dir("my-folder") #' #' @omit #' oo <- options(onetime.dir = tempdir(check = TRUE)) ... ``` `@omit` omits everything after it from the generated test. This code created a very simple test file in `test-doctest-onetime_dir.R`: ```r # Generated by doctest: do not edit by hand # Please edit file in R/utils.R test_that("Doctest: onetime_dir", { # Created from @doctest for `onetime_dir` # Source file: R/utils.R # Source line: 138 expect_match(onetime_dir("my-folder"), "my-folder$") }) ``` I ran these tests and [committed](https://github.com/hughjonesd/onetime/commit/cec1949a0129f00310dec23a12b411092dfbe3fe?diff=split) them. Lastly, I added tests for my final functions. There's nothing new here. You can see the [commit](https://github.com/hughjonesd/onetime/commit/d3be66faabee09929deceafc9598488efd0ae8ee?diff=split) on GitHub. # Adding doctest to Suggests: Now that my doctests were working, I decided to make it easy for other developers to work on my package too. In the onetime DESCRIPTION file, I added doctest to `Suggests:`, and added a `Remotes:` field pointing to the [github repository](https://github.com/hughjonesd/doctest). This is a oneliner with the usethis package: ```r usethis::use_dev_package("doctest", type = "Suggests", remote = "hughjonesd/doctest") ``` There is a tradeoff here: adding the doctest dependency will help other developers, but CRAN doesn't allow `Remotes:` fields in packages. So when I submit the next version, I'll have to remove the dependency again. # Conclusion This was encouragingly easy. All my tests passed the first time. Now I can develop more securely, knowing that if my changes stop my examples working, doctest will help to catch that. I followed some principles when making these changes to my package: * Start small, by changing `@examples` tags to `@doctest` tags. Doctest is a new package, so you want to make sure it doesn't do anything bad. (If it does, please [file a bug report](https://github.com/hughjonesd/doctest/issues)!) Obviously, you should make sure your code is checked in to version control before using doctest. * Keep doctests simple. Focus example code on its key role, which is to teach the user about your package. If you need to make big changes, or if your expectations are becoming complex, consider splitting them out into a "proper" testthat test. There are many features of doctest that I didn't need to use for this small package, including the `@expectRaw` and `@snap` tags to generate other expectations, and the `@testRaw` tag to add code to your tests. You can read about those in the package documentation or in the main vignette: `vignette("doctest")`. If you are using doctest in your package, I'd love to hear about it. There is a [github issue](https://github.com/hughjonesd/doctest/issues/8) where end users can add their package. And of course, I welcome bug reports, enhancement requests and feedback. Happy testing!