The LASER Toolkit & Reproducible Research

LASER Orientation Module: Conceptual Overview

Overview

Reproducible Research

• Definition

• Benefits

• Pain Points

• Best Practices

• Discussion

The LASER Toolkit

• GitHub

• Posit Cloud

• RStudio

• Quarto

• R & Python

Speaker Notes:

Each module begins with a 30-minute conceptual overview of key terminology, techniques, and applications.

In this module we’ll discuss the importance of reproducible research including benefits, pain points and best practices.

We’ll also breifly introduce you to the tools we’ve adopted for LASER and tools for reproducible research. These tools include GitHub and Posit Cloud for hosting our website and curriculum materials, RStudio for managing research projects, Quarto for documenting and reporting research activities, and R & Python for wrangling, exploring, and modeling data.

Everything we introduce in this presentation you will have an opportunity to explore in much greater depth through the essential readings, code-along, and coding case study.

What is Reproducible Research?

In computational sciences like LA:

• Ideally… researchers can replicate your findings by following the procedures used to gather the data and run the computer code.

• Realistically… the data and code used to make a finding are available and sufficient to recreate the finding.

Reproducible Research with R (Gandrud 2021)

Speaker Notes:

The information from this conceptual overview on reproducible research is drawn largely from the excellent book by Christopher Gandrud, Reproducible research with R and R studio. Even if you’re a python user, this is a valuable resource for essential concepts and practices for ensuring that your research is reproducible.

The distinction between these two definitions is that we recognize it is sometimes difficult to replicate the original data set in a study because of issues such as limited resources to gather new data, or because the original study may have already sampled the full universe of cases.

The Benefits of Reproducible Research

For Science:

• Standard to judge scientific claims

• Enhances replicability

• Avoiding effort duplication

• Cumulative knowledge development

For Yourself:

• Better work habits

• Better teamwork

• Changes are easier

• Higher impact research

Speaker Notes:

For Science:

• Standard to judge scientific claims. Reproducibility opens up your findings and work to scrutiny, and is the minimum standard for judging scientific claims. However, “a study can be reproducible but still be wrong” (Peng, 2014).

• Reproducibility enhances replicability. If other researchers are able to clearly understand how a finding was originally made, then they will be better able to conduct comparable research, in meaningful attempts to replicate the original findings. However, a finding in one study may be statistically significant when reproduced using the original data/code, but when researchers try to replicate it using new data and even methods, they are unable to find a similar result. The original finding could have been noise, even though it is fully reproducible.

• Avoiding effort duplication. Reproducible research, in theory, should cut down on the amount of time others have to spend gathering data or developing procedures that have already been collected or figured out.

• Cumulative knowledge development. They can instead use that time to more quickly build on established findings and develop new knowledge.

For Yourself:

• Better work habits. It can spur you to more effectively plan and organize your research. It should push you to bring your data and source code up to a higher level of quality than you might if you thought no one was looking.

• Better teamwork. RR makes it easier for your collaborators figure out what you have done also to understand your work and up to speed on a research project.

• Changes are easier. No actual research process is completely linear. You almost never gather data, run analyses, and present your results without going backwards to add variables, make changes to your statistical models, create new graphs, alter results tables in light of new findings. RR makes it easier to reproduce parts or even all of your own research.

• Higher impact research. Reproducible research is more likely to be useful for other researchers than nonreproducible research. Useful research is cited more frequently!

Reproducible Research?

Speaker Notes:

In practice, your research should be easy for independent researchers to reproduce! If a study is difficult to reproduce, it’s more likely that no one will reproduce it. If someone does attempt to reproduce it, it will be difficult for them to tell if any errors they find were in the original research or problems introduced during the reproduction.

So next we’re going to talk about some general best practices for reproducible research, and then give an overview of the research tools we’ll be working with throughout the LASER institute to help ensure it’s relatively easy to reproduce our research.

Discussion

Think about the following questions and then discuss at your table:

• What have your experiences been with reproducible research?

• What tools have you used to ensure the reproducibility of your work?

• What questions do you have about reproducible research?

Best Practices for Reproducible Research

1. Document everything!

2. Everything is a (text) file.

3. All files should be “human readable.”

4. Explicitly tie your files together.

5. Have a plan to organize, store, and make your files available.

To learn more: Reproducible research with R and R studio (3rd Edition). CRC Press.

Speaker Notes:

1. Document everything! Obviously, in order to reproduce your research, others must be able to know what you did. Documentation of includes, but is not limited to the data itself, analysis code, the write-up of results, and explanations of these files (e.g. data set codebooks, software information, and so on).

2. Everything is a (text) file. Ideally, your documentation will be stored in the simplest file format possible to store this information. Usually the best file format is as simple but versatile text file like a comma-separated values (CSV) file for storing data, R and python scripts for documenting code, and markdown files like Quarto for articles, webpages, books, and more.

3. All files should be human readable. Treat all of your research files as if someone who has not worked on the project will, in the future, try to understand them. Computer code like R and Python is ‘machine readable’ in that the computer is able to use it to understand what you want to do. However, there is a very good chance that other people (or you six months in the future) will not understand what you were telling the computer to do. So, you need to make all of your files ‘human readable’ by commenting on your code so others can understand its design and purpose.

4. Explicitly tie your files together. Researchers often do not explicitly document the relationships between files that they used in their research. For example, the results of an analysis, for example a table or figure, might be copied and pasted into a presentation document. However, it can be very difficult for future researchers to trace the table or figure back to a particular statistical model and a particular data set without clear documentation. Therefore, you should make an effort to document the links between your files.

5. Have a plan to organize, store, and make your files available. Finally, in order for independent researchers to reproduce your work, they need to be able access the files that instruct them how to do this. Files also need to be organized so that independent researchers can figure out how they fit together. So, from the beginning of your research process, you should have a plan for organizing your files and a way to make them accessible.

Learn More

We’ve only scratched the surface Reproducible Research practices and highlight recommend the book Reproducible research with R and R studio to learn more.

The LASER Toolkit

GitHub, Posit, RStudio, Quarto, and R & Python

Speaker Notes:

One point we want to emphasis is that the tools we use for LASER and that you’ll become familiar with over the next year, is that the LASER way of doing things is just one approach. We hope that you find these set of tools useful for your own research and teaching, but we also encourage you to adapt our curriculum to tools that you and the learners you’ll be working with are comfortable with.

In fact, we’d love to be able over the course of the year to include in our curriculum alternate formats for our case studies and presentations, such as Jupyter notebooks or versions using Google Collab.

Tool Types

Reproducible research involves two broad sets of tools:

• A Reproducible Research Environment, that includes the statistical tools you need to run your analyses; automatically track the provenance of data, analyses, and results; and to package them for redistribution”.

• A Reproducible Research Publisher that prepares dynamic documents for presenting results and is easily linked to the reproducible research environment.

GitHub

The LASER Institute GitHub site houses repositories for all curriculum materials and GitHub Pages is used to publish our site to the web.

GitHub is a web-based platform used for version control, collaboration, and sharing of a project’s code, documents, and other related files.

            go.ncsu.edu/laser-github

Speaker Notes:

GitHub is an interface and cloud hosting service built on top of the Git version control system. GitHub was originally designed to for programmers to work together to build computer programs, but has become widely used by researcher and educators to host statistical software packages, course websites and books, and even online tutorials.

We won’t be using GitHub directly, but I did want to note that if you click the link, it will take you to our GitHub organization site where you will find our of the course materials stored as Repositories, or repos, including the laser-orientation repository you just opened.

Later, you’ll be learning how to use this GitHub site to adapt these materials for your own instruction.

[Prompt scholars to click the link and then give them a very brief tour of the LASER GitHub site including the repositories for each method.]

Posit Cloud

Posit Cloud lets you access Posit’s powerful set of data science tools like RStudio IDE and Jupyter Notebooks right in your browser.

Interactive components for each module (e.g., Case Studies and Code-Alongs) are accessed and complete in our LASER Learners workspace.

            go.ncsu.edu/laser-learners

Speaker Notes:

Posit Cloud (formerly RStudio Cloud) provides a powerful set of data science tools right in your browser with no installation or complex configuration required. All of our Module materials will be accessed through Posit Cloud this year so we encourage you bookmark it in your browser.

[Prompt scholars navigate to our LASER Learners workspace using the link provided and share a brief overview of the the Posit Cloud features like workspaces, recipes, cheatsheets, etc.

Direct them click on the LASER Orientation Module RStudio “assignment” and have them open.]

Note that when you clicked on this assignment, you are actually creating a copy of the original project and that only you and LASER Instructors can access these personal copies.

It’s important that instructors have access so we can assist you with assignments when you get stuck.

RStudio

RStudio is an integrated development environment (IDE) for R and Python and includes:

• a Console for running R code directly,

• syntax-highlighting editor that supports direct code execution in the Source pane,

• tools for plotting, history, debugging, and management of research projects in the Environment and Files panes.

Speaker Notes:

What you are looking at now is RStudio, which is an integrated development environment (IDE) for R and Python and includes a console for running R code directly, syntax-highlighting editor that supports direct code execution in the source pane, and tools for plotting, history, debugging, and management of research projects in the Environment and Files panes.

RStudio is basically your home for all of the files, images, reports, and code that are used in a research project given project. When you create a “project” with RStudio, it’s straightforward to divide your work into multiple contexts, each with their own working directory, workspace, history, and source documents.

[Provide a brief walkthrough of the different panes and discuss what they are used for including how they might make their own R Project is they so desired.]

Quarto

Quarto is used with Python and R to create reproducible, production quality:

• websites

• presentations

• dashboards

• coding tutorials

• books

• and much more!

Speaker Notes:

Virtually all of our curriculum materials, including our website, our proudly built with Quarto. Quarto is is an open-source scientific and technical publishing system that enables the creation of dynamic documents, dashboards, presentations, websites and more.

Quarto also integrates with programming languages like R and Python, allowing for the inclusion of interactive elements and reproducible research workflows, making it a great tool for data analysis, visualization, and reporting.

[Click through each link to show them examples of different LASER related products that can be made with Quarto. Highlight the case study answer key and dashboard which we’ll be working with shortly.

Prompt learners to open up the orientation-data-product.qmd file as an example of a simple data dashboard and show them how to render a Quarto document using the dashboard as an example.

Then have them open up the orientation-cass-study-R.qmd or orientation-cass-study-python.qmd case study file and ask them to render and discuss what happens when it doesn’t and why.]

R & Python

The LASER Institute uses R and Python, two of the most popular programming languages for data science, statistical analysis, and machine learning.

Both are freely available have large and active communities and a vast number of libraries and frameworks for learning analytics.

Speaker Notes:

Underlying all of this is R (or Python), two of the most popular programming languages used in data science, statistical analysis, and machine learning. Each provide a rich platform that can be used to create a wide range of static and dynamic outputs for communicating your research.

Let’s create a new R (or Python) script so we can begin working with R to help illustrate a simple research workflow and analyze some data.

[Direct learners to create a new R or Python script as illustrated in the screenshot. Show them how to name the file and save to their main directory.

Transition to code-along slides.]

Essential Readings

Chapters 1 and 2 of Reproducible Research with R and RStudio by Gandrud (2021) lay the groundwork for understanding and implementing reproducible research:

• Chapter 1: Introducing Reproducible Research

• Chapter 2: Getting Started with Reproducible Research

Questions for reflection and discussion are also included in our Essential Readings document. Responses can be posted to our laser-orientation channelon Slack.

Speaker Notes:

Each LASER Module comes with a set of essential readings and reflection questions to dig deeper into concepts shared in our conceptual overviews and provide opportunities for checking your own understanding or the understanding of colleagues and students you work with.

For this module, we recommend this week taking a look at Chapters 1 and 2 of Reproducible Research with R and RStudio.

To help guide your reflection on the readings, a set of questions are provided in our essential readings document. You are welecome to contribute to our learning community by creating a new post to our laser-orientation channel on Slack. Your post might contain a response to one or more of the guiding questions, questions you still have about the topics addressed, or insights gained into your own research.

Acknowledgements

This work was supported by the National Science Foundation grants DRL-2025090 and DRL-2321128 (ECR:BCSER). Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

Gandrud, Christopher. 2021. Reproducible Research with r and r Studio (3rd Edition). CRC Press. http://github.com/christophergandrud/Rep-Res-Book.