Prepare and Wrangle
Text Mining Module 1: A Code-a-long
Welcome to the LAW Code-a-long for Module 1
The Text Mining course is designed for those seeking an introductory understanding of quantifying the text in documents to better understand their properties
The following code-a-long is a companion to the Module 1 case study’s Prepare and Wrangle stages
Figure 2.2 Steps of Data-Intensive Research Workflow
[@krumm2018]
Load Libraries
- Load the
tidyverseandtidytextpackages usinglibrary()
Reading Data, Cont.
To get started, we need to import, or “read”, our data into R
The RttT Online PD survey data is stored in a CSV file named opd_survey.csv which is located in the data folder of this project
Use
read_csv()to saveopd_survey.csvas a new objectopd_surveyThis file can be found in the local data folder
If you have questions about the above function, don’t be afraid to look it up in console using
?read_csv()
- There is also an Import Dataset feature under the Environment tab in the upper right pane of RStudio
Tidy Text, Cont.
In this section, our goal is to transform our opd_teacher text data from this:
Tidy Text, Cont.
to this:
Remove Stop Words
One final step in tidying our text is to remove words that don’t add much value to our analysis (at least when using this approach) such as “and”, “the”, “of”, “to” etc. The tidytext package contains a stop_words dataset derived from three different lexicons that we’ll use to remove rows that match words in this dataset.
Let’s take a look at the first 20 stop words so we know what we’re getting rid of from our opd_tidy dataset.
# A tibble: 20 × 2
word lexicon
<chr> <chr>
1 a SMART
2 a's SMART
3 able SMART
4 about SMART
5 above SMART
6 according SMART
7 accordingly SMART
8 across SMART
9 actually SMART
10 after SMART
11 afterwards SMART
12 again SMART
13 against SMART
14 ain't SMART
15 all SMART
16 allow SMART
17 allows SMART
18 almost SMART
19 alone SMART
20 along SMART Now run the following code to view all stop words in each lexicon:
In order to remove these stop words, we will use function called anti_join() that looks for matching values in a specific column from two datasets and returns rows from the original dataset that have no matches. For a good overview of the different dplyr joins see here: https://medium.com/the-codehub/beginners-guide-to-using-joins-in-r-682fc9b1f119
Let’s remove rows from our opd_tidy data frame that contain matches in the word column with those in the stop_words dataset and save it as opd_clean since we were done cleaning our data at this point.
# A tibble: 78,675 × 3
Role Resource word
<chr> <chr> <chr>
1 Teacher Live Webinar levels
2 Teacher Live Webinar ofquestioning
3 Teacher Live Webinar revised
4 Teacher Live Webinar blooms
5 Teacher Online Learning Module modules
6 Teacher Online Learning Module teacher
7 Teacher Online Learning Module shown
8 Teacher Online Learning Module action
9 Teacher Online Learning Module classroom
10 Teacher Online Learning Module modeling
# ℹ 78,665 more rows👉 Your Turn ⤵
In the code chunk below, add the anti_join() function to our chain of data wrangling functions as our final text preprocessing step. Don’t forget to include in the function the stop words you are trying to remove.
Also, add a short comment to each line of code explaining each data wrangling step. The first had been done for you.
# A tibble: 78,675 × 3
Role Resource word
<chr> <chr> <chr>
1 Teacher Live Webinar levels
2 Teacher Live Webinar ofquestioning
3 Teacher Live Webinar revised
4 Teacher Live Webinar blooms
5 Teacher Online Learning Module modules
6 Teacher Online Learning Module teacher
7 Teacher Online Learning Module shown
8 Teacher Online Learning Module action
9 Teacher Online Learning Module classroom
10 Teacher Online Learning Module modeling
# ℹ 78,665 more rowsIf you did this correctly, you should now see a wrangled_teacher data frame in the Environment pane that has 78,675 observations and 3 variables.
👉 Your Turn ⤵
Finally, let’s wrangle a data set that includes tokens not just for teachers, but for all educators who particiated in the online professional development offerings.
Complete and run the following code to wrangle and tidy your opd_survey data for all survey respondents.
# A tibble: 92,722 × 3
Role Resource word
<chr> <chr> <chr>
1 Central Office Staff (e.g. Superintendents, Tech Director, Cu… Online … glob…
2 Central Office Staff (e.g. Superintendents, Tech Director, Cu… Online … view
3 School Support Staff (e.g. Counselors, Technology Facilitator… Calendar comm…
4 Teacher Live We… leve…
5 Teacher Live We… ofqu…
6 Teacher Live We… revi…
7 Teacher Live We… bloo…
8 School Support Staff (e.g. Counselors, Technology Facilitator… Live We… awar…
9 Central Office Staff (e.g. Superintendents, Tech Director, Cu… Wiki info…
10 Central Office Staff (e.g. Superintendents, Tech Director, Cu… Wiki rele…
# ℹ 92,712 more rows❓Questions
- How many observations are in the data frame with all educators?
- YOUR RESPONSE HERE
- Why do you think the console provided the message “Joining, by = ‘word’”?
- YOUR RESPONSE HERE
Congratulations, we’re now ready to begin exploring our wrangled data!
