Adding Metrics to a Workflow

Code Along

Getting started

Process

• Again, create a .R file in /module-3
• Then, run copy and paste the code in this presentation as we talk through each step

Quick discussion

• What are the benefits of using metrics beyond “Accuracy”?
• Why is feature engineering a useful step?

Code-along: R

• 0. Setup
• 1. Split data
• 2. Engineer features
• 3. Specify model
• 4. Fit model (training)
• 4. Fit model (testing)
• 5. Evaluate accuracy
• 6. Aside: Joins!

Loading, setting up: create a .R file in /lab-1 and run this code

library(tidyverse)
library(tidymodels)

starwars_recoded <- starwars %>% # built-in data available just by typing
    mutate(species_human = ifelse(species == "Human", "human", "not human"))

starwars_recoded %>% 
    count(species_human) # how many humans are there?

train_test_split <- initial_split(starwars_recoded, prop = .70)

data_train <- training(train_test_split)

# predicting humans based on the variables we used in LL1 + birth_year and homeworld
my_rec <- recipe(species_human ~ height + mass + birth_year + eye_color, data = data_train) %>% 
    step_dummy(eye_color) # need to dummy code

my_mod <-
    logistic_reg() %>% 
    set_engine("glm") %>%
    set_mode("classification")

my_wf <-
    workflow() %>%
    add_model(my_mod) %>% 
    add_recipe(my_rec)

Model building with training data

class_metrics <- metric_set(accuracy, sensitivity, specificity, ppv, npv, kap) # this is new
final_fit <- last_fit(my_wf, train_test_split, metrics = class_metrics)

Model evaluating with testing data

fit_model <- fit(my_wf, data_train)

predictions <- predict(fit_model, data_train) %>% 
    bind_cols(data_train) %>% 
    mutate(species_human = as.factor(species_human))

predictions %>%
    metrics(species_human, .pred_class) %>%
    filter(.metric == "accuracy")

Only run this once you’re done training/messing with your model!; this way, these estimates will be unbiased

final_fit %>%
    collect_metrics()

band_members %>% left_join(band_instruments)
band_members %>% inner_join(band_instruments)

Code-along: python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Assuming starwars data is loaded into a pandas DataFrame called 'starwars'
starwars_recoded = starwars.copy()
starwars_recoded['species_human'] = starwars_recoded['species'].map(lambda x: 'human' if x == 'Human' else 'not human')

# Split data
X = starwars_recoded[['height', 'mass', 'birth_year', 'eye_color']]
y = starwars_recoded['species_human']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Engineer features and specify model
preprocessor = ColumnTransformer(
    transformers=[
        ('num', 'passthrough', ['height', 'mass', 'birth_year']),
        ('cat', OneHotEncoder(drop='first'), ['eye_color'])
    ])

model = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression())
])

# Fit model
model.fit(X_train, y_train)

# Evaluate accuracy
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, pos_label='human')
recall = recall_score(y_test, y_pred, pos_label='human')
f1 = f1_score(y_test, y_pred, pos_label='human')

print(f"Accuracy: {accuracy:.2f}")
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

# Aside: Joins (using pandas)
merged_inner = pd.merge(band_members, band_instruments, how='inner')
merged_left = pd.merge(band_members, band_instruments, how='left')