Adding Metrics to a Workflow

Code Along

# load all the relevant packages
pacman::p_load(pacman, knitr, tidyverse, readxl, tidymodels)

Getting started

Process

• Again, create a .R file in /module-3
• Then, run copy and paste the code in this presentation as we talk through each step

Quick discussion

• What are the benefits of using metrics beyond “Accuracy”?
• Why is feature engineering a useful step?

Code-along: R

• 0.
• 1.
• 2.
• 3.
• 4.
• 4.
• 5.

Loading, setting up: create a .R file in /module-3 and run this code

#| eval: false
#| echo: true
library(tidyverse)
library(tidymodels)

pokemon <- read_csv("data/pokemon-data.csv")

pokemon %>%
    glimpse()

pokemon %>% 
    count(early_gen) # distribution of early vs later generations

#| eval: false
#| echo: true

train_test_split <- initial_split(pokemon, prop = .70)

data_train <- training(train_test_split)

#| eval: false
#| echo: true

my_rec <- recipe(early_gen ~ height_m + weight_kg + hp, data = data_train) %>% 
    step_mutate(early_gen = as.factor(early_gen))

#| eval: false
#| echo: true

my_mod <-
    logistic_reg() %>% 
    set_engine("glm") %>%
    set_mode("classification")

my_wf <-
    workflow() %>%
    add_model(my_mod) %>% 
    add_recipe(my_rec)

Model building with training data

#| eval: false
#| echo: true

class_metrics <- metric_set(accuracy, sensitivity, specificity, ppv, npv, kap) # this is new
final_fit <- last_fit(my_wf, train_test_split, metrics = class_metrics)

Model evaluating with testing data

#| eval: false
#| echo: true

fit_model <- last_fit(my_wf, train_test_split)

Only run this once you’re done training/messing with your model!; this way, these estimates will be unbiased

#| eval: false
#| echo: true

final_fit %>%
    collect_metrics()

Code-along: python

• 0.
• 1.
• 2.
• 3.
• 4.
• 5.

Loading, setting up: create a .py file in /module-3 and run this code

import pandas as pd 

pokemon_df = pd.read_csv("../module-2/data/pokemon-data.csv")
pokemon_df['early_gen'].value_counts()

Train-test split the dataset

from sklearn.model_selection import train_test_split
train_df, test_df = train_test_split(pokemon_df, test_size=0.3)

Specify the model formula and fit it to the training dataset

# predicting early generation status based on height, weight, and HP
import statsmodels.formula.api as smf

model = smf.logit('early_gen ~ height_m + weight_kg + hp', data=train_df).fit()

Make predictions on test dataset

y_pred_prob = model.predict(test_df)

# Convert the predictions from a probability to a binary outcome
y_preds = (y_pred_prob >= 0.5).astype(int)

y_actual = test_df['early_gen'].astype(int)

Calculate model metrics

from sklearn.metrics import accuracy_score, recall_score, confusion_matrix, cohen_kappa_score

# Metrics we'se seen: 
accuracy = accuracy_score(y_actual, y_preds)

# New metrics
precision = precision_score(y_actual, y_preds, pos_label=1)
kappa = cohen_kappa_score(y_actual, y_preds)

# We'll learn more about the confusion matrix in the case study
true_negative, false_positive, false_negative, true_positive = confusion_matrix(y_actual, y_preds).ravel()
sensitivity = true_positive / (true_positive + false_negative)  # or the sklearn.metrics recall_score function
specificity = true_negative / (true_negative + false_positive)
negative_predictive_value = true_negative / (true_negative + false_negative)

Evaluate metrics

metrics = {
    "accuracy": accuracy,
    "sensitivity": sensitivity,
    "specificity": specificity,
    "precision": precision,
    "negative_predictive_value": negative_predictive_value,
    "cohen_kappa": kappa
}

for key, value in metrics.items(): 
    print(f'{key}: {round(value, 3)}')

Discussion

• What are things you consider when choosing which metric(s) to interpret for a particular analysis?