Data Science Workflows
I’m obsessed with how to structure a data science project. The time I spend worrying about project structure would be better spent on actually writing code. Here’s my preferred R workflow, and a few notes on Python as well.
The R package workflow
In R, the package is “the fundamental unit of shareable code”.
At rstudio::conf 2020, Hadley gave a rule of thumb for when to create a package, which I’ll paraphrase: “When you copy and paste a block of code three times, make a function. When you do that three times, make a package.” My rule of thumb is stricter: if I might come back to this project tomorrow, it should be a package. I’m a big fan of this workflow, and I use it for just about everything.
People have told me that making a package seems like a massive learning curve. That’s fair. Packages look strange from the outside, with a whole bunch of bizarre files. Code is lumped together into a single directory, except for the stuff in inst/, and it’s hard to tell what’s going on with inst/. Imagine being an R novice, and trying to find the code for your favourite function somewhere in a git repository. This stuff is hard, but there are tools out there that make the rewards greater than the costs.
Packages are just a way to organise code
First thing’s first: R packages exist independently of CRAN. The packages I make are specific to a project, and often specific to a single data set. They’re never going to be submitted to CRAN. Most don’t even make it onto GitHub. R packages can be just a way to organise code, nothing more. By following a certain structure and a few rules we get to benefit from a whole bunch of tools designed just for packages.
Package structure is thoroughly explained in Hadley’s book, but here are the simplified requirements:
- All function code goes in R files in the
R/directory. The functions in here will be available when the package is loaded. - Unit tests go in the
tests/directory. - Package metadata goes in the
DESCRIPTIONfile. - There’s a
NAMESPACEfile and aman/directory, but these can be ignored; they’re generated automatically from Roxygen strings. - There’s a LICENSE file for the licence under which the package is released (if it is released).
- Everything else — exploratory analysis, stray bits of code, R markdown, or anything that isn’t mentioned above — goes into the
inst/directory.
The tricky parts of package development are automated
Most of the intricacies of package development can be outsourced to devtools and usethis. I never have to think about how to set up the structure of a package: I type usethis::create_package("packagename") and it’s all set up for me. I never have to think about where to stick my test files: usethis::create_test("file-name") does it for me. The LICENSE requirement is a bit annoying, but usethis::use_mit_license() can immediately set me up with a permissive licence.1
Maybe the end result of my project is an R Markdown report. Instead of filling the R Markdown file with helper functions, I put all of those functions in files in the R/ directory. I put my R Markdown file in the inst/ directory, and in it I load the functions of the package. I don’t even need to install the package: I can use devtools::load_all() to make all of those functions available to me.
I’ll repeat that: a package workflow can be used without ever installing the package. For almost all personal packages, devtools::load_all() is good enough.
The benefits of a package workflow outweigh the extra effort
A package structure does require a bit more thought, but what you get in return makes it worthwhile. In exchange for following the structure and the rules, I get:
- A very clear set of dependencies for my project. I have to declare which packages I’m using in my package (shortcut:
usethis::use_package()) but in exchange I can see them listed all in one file, rather than being a bunch oflibrarycalls scattered throughout multiple files. This is also one of the ways thatrenvcan pick up on dependencies, to make reproducibility easier. - Powerful function documentation through
Roxygen2. By putting some special comments above my functions, I can generate proper R documentation. It’s satisfying to type?my_custom_functionand see help material. Even if I’m the only one who will use this code, that documentation make my life easier. Hint: use RStudio to generate Roxygen skeletons! - A convenient testing framework through
testthat. Unit tests save time, and this is the hill I will die on. There’s a bit of set-up involved, but it’s all taken care of the first time a test is created withusethis::use_test(). - An all-in-one package check.
rcmdcheck::rcmdcheck()(also available as a button in RStudio) not only runs my unit tests, but also looks for issues with portability, missing dependency declarations, misplaced characters, and much more. I often run this when I’m developing my functions, and I also set it up to run automatically every time I push code to GitHub. This is a big deal — if I break a unit test or do something else wrong, I want to know about it!
Package development shortcuts
I took note of every function and shortcut I use when following a package workflow. Every function below links to its documentation:
| Operation | Function | Notes |
|---|---|---|
| Create a new package | usethis::create_package | Also creates an RStudio project if you’re using RStudio |
| Create a new function | usethis::use_r | If the file exists, will open it |
| Create a new test file | usethis::use_test | File names are prefixed with “test-”. Will also set up test infrastructure if it doesn’t already exist. |
| Declare a dependency | usethis::use_package | Use the type = "Suggests" argument for suggested packages |
| Source all functions | devtools::load_all | RStudio default keyboard shortcut: Ctrl/⌘ + Shift + L |
| Build documentation | devtools::document | RStudio default keyboard shortcut: Ctrl/⌘ + Shift + D |
| Set up environment to pin package dependencies | renv::init | renv is extremely important for any piece of work that needs to be reproducible |
| Capture or update package dependencies | renv::snapshot | This is in addition to the dependencies in the DESCRIPTION file, since it captures all required packages along with their installed versions |
| Run tests | devtools::test | UI option: In RStudio, look for the “Test package” option below the “More” button in the “Build” tab. RStudio default keyboard shortcut: Ctrl/⌘ + Shift + T |
| Run tests in a specific file | devtools::test_file | By default, will test the active file, or a file path can be provided. UI option: he test file in RStudio and look for the “Run Tests” button above the editor. |
| Run a full package check, including tests | rcmdcheck::rcmdcheck | UI option: In Rstudio, look for the “Check” button in the “Build” tab |
Package workflows and APIs
R code can be served as an API with the plumber package, and this is compatible with a package workflow. James Blair gave a great talk on Practical Plumber Patterns at rstudio::conf 2020 where he presented an R package combined with plumber. One of the functions in the package actually starts the API. The source code is available on GitHub
The other language
I use Python almost every day, but I hadn’t given a lot of thought on how to structure a Python project. I reached out to Twitter to see if there was a standard.
Thank you to everyone who responded to this! I learnt a lot.
The responses are worth reading. There’s a range of answers, but the general theme is about “promoting” code from an informal notebook to a formal module. Some people start with a module structure, which they install into their system, and then use notebooks for exploration. Some people start with notebooks, and eventually transition their work into something with more structure.
Handy Python workflow tools
There are tools in Python that make projects a bit easier:
- Cookiecutter is a tool for creating projects in Python from templates. The package includes a few official templates (including a package template) but there are over 4000 templates supplied by members of the Python community.
- Poetry is a build tool for Python packages with a heavy emphasis on dependency management.
pip install -einstall an “editable” version of a local module. This means that changes to the local code are immediately reflected.- nbdev makes it easier to develop in notebooks, by allowing easy promotion of code to a module with tags.
I’m particularly keen to check out Cookiecutter. It seems like it fills a purpose in Python similar to that of the usethis package in R.
I struggle with notebooks
Personally, though, I still can’t quite grok notebooks. I remember using them way back in the day when I was using SageMath, but now they just don’t click with me. The idea that I have to switch a cell to markdown still throws me off, and I find myself reaching for a repl that doesn’t exist. Mentally I tell myself, “I’ll create a cell which I’ll use just for quick calculations”, and my experience is that this is a recipe for a notebook filled with junk code.
I still use notebooks when I’m working in Python — Python users tend to be wedded to them, and I don’t want to be the one going against the grain instead of collaborating. In the past I’ve used Spyder or VSCode for Python development. Lately I’ve even be experimenting with RStudio as a Python IDE, and it does pretty well.
devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.0.0 (2020-04-24)
#> os Ubuntu 20.04 LTS
#> system x86_64, linux-gnu
#> ui X11
#> language en_AU:en
#> collate en_AU.UTF-8
#> ctype en_AU.UTF-8
#> tz Australia/Melbourne
#> date 2020-07-19
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date lib source
#> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.0)
#> backports 1.1.8 2020-06-17 [1] CRAN (R 4.0.0)
#> callr 3.4.3 2020-03-28 [1] CRAN (R 4.0.0)
#> chromote 0.0.0.9002 2020-07-13 [1] Github (rstudio/chromote@b706e21)
#> cli 2.0.2 2020-02-28 [1] CRAN (R 4.0.0)
#> crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.0)
#> curl 4.3 2019-12-02 [1] CRAN (R 4.0.0)
#> desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.0)
#> devtools 2.3.0 2020-04-10 [1] CRAN (R 4.0.0)
#> digest 0.6.25 2020-02-23 [1] CRAN (R 4.0.0)
#> ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.0)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.0)
#> fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.0)
#> fastmap 1.0.1 2019-10-08 [1] CRAN (R 4.0.0)
#> fs 1.4.1 2020-04-04 [1] CRAN (R 4.0.0)
#> glue 1.4.1 2020-05-13 [1] CRAN (R 4.0.0)
#> htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.0)
#> httr 1.4.1 2019-08-05 [1] CRAN (R 4.0.0)
#> hugodown 0.0.0.9000 2020-06-20 [1] Github (r-lib/hugodown@f7df565)
#> jsonlite 1.7.0 2020-06-25 [1] CRAN (R 4.0.0)
#> knitr 1.29 2020-06-23 [1] CRAN (R 4.0.0)
#> later 1.1.0.1 2020-06-05 [1] CRAN (R 4.0.0)
#> magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.0)
#> memoise 1.1.0.9000 2020-05-09 [1] Github (hadley/memoise@4aefd9f)
#> pkgbuild 1.0.8 2020-05-07 [1] CRAN (R 4.0.0)
#> pkgload 1.1.0 2020-05-29 [1] CRAN (R 4.0.0)
#> prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.0)
#> processx 3.4.3 2020-07-05 [1] CRAN (R 4.0.0)
#> promises 1.1.1 2020-06-09 [1] CRAN (R 4.0.0)
#> ps 1.3.3 2020-05-08 [1] CRAN (R 4.0.0)
#> R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.0)
#> Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.0)
#> remotes 2.1.1 2020-02-15 [1] CRAN (R 4.0.0)
#> rlang 0.4.7 2020-07-09 [1] CRAN (R 4.0.0)
#> rmarkdown 2.3.2 2020-07-12 [1] Github (rstudio/rmarkdown@ff1b279)
#> rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.0)
#> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.0)
#> stringi 1.4.6 2020-02-17 [1] CRAN (R 4.0.0)
#> stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.0)
#> testthat 2.3.2 2020-03-02 [1] CRAN (R 4.0.0)
#> tweetrmd 0.0.8 2020-07-12 [1] Github (gadenbuie/tweetrmd@50bcdf8)
#> usethis 1.6.1 2020-04-29 [1] CRAN (R 4.0.0)
#> webshot2 0.0.0.9000 2020-07-13 [1] Github (rstudio/webshot2@f62e743)
#> websocket 1.3.0 2020-07-05 [1] CRAN (R 4.0.0)
#> withr 2.2.0 2020-04-20 [1] CRAN (R 4.0.0)
#> xfun 0.15 2020-06-21 [1] CRAN (R 4.0.0)
#> yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.0)
#>
#> [1] /home/mdneuzerling/R/x86_64-pc-linux-gnu-library/4.0
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/libraryThe image at the top of this page is in the public domain.
-
For closed-source packages, create a file that says “Proprietary. Property of
.” Then put “License: file LICENSE” in the package DESCRIPTION. ↩︎
