KNN (K-Nearest Neighbours)
Hannah Pahama
February 2025
1. Setup
wine <- readRDS(gzcon(url("https://github.com/cd-public/D505/raw/master/dat/pinot.rds")))
# Peeking at variable types first before anything
variable_classes <- data.frame(
Variable = colnames(wine),
Class = sapply(wine, class)
)
print(variable_classes)## Variable Class
## id id integer
## province province character
## price price numeric
## points points numeric
## year year numeric
## description description character2. KNN Concepts
TODO: Explain how the choice of K affects the quality of your prediction when using a Nearest Neighbors algorithm.
Explanation:
- A lower K value (e.g., K=1) can lead to overfitting because
predictions are too sensitive to noise. - A higher K value smooths
predictions but may cause underfitting. - Choosing an optimal K using
cross-validation helps balance bias and variance.
3. Feature Engineering
Before everything else…
wine <- wine %>%
mutate(
year_factor = factor(year),
description = tolower(description),
has_cherry = str_detect(description,
"cherry"),
has_chocolate = str_detect(description,
"chocolate"),
has_earth = str_detect(description,
"earth"),
cherry_year = year * has_cherry,
chocolate_year = year * has_chocolate,
earth_year = year * has_earth
) %>%
select(-description)
head(wine)## id province price points year year_factor has_cherry has_chocolate
## 1 1 Oregon 65 87 2012 2012 FALSE FALSE
## 2 2 Oregon 20 87 2013 2013 FALSE TRUE
## 3 3 California 69 87 2011 2011 FALSE FALSE
## 4 4 Oregon 50 86 2010 2010 FALSE FALSE
## 5 5 Oregon 22 86 2009 2009 FALSE FALSE
## 6 6 Oregon 25 86 2015 2015 FALSE FALSE
## has_earth cherry_year chocolate_year earth_year
## 1 TRUE 0 0 2012
## 2 FALSE 0 2013 0
## 3 TRUE 0 0 2011
## 4 FALSE 0 0 0
## 5 TRUE 0 0 2009
## 6 FALSE 0 0 0# Summarize the transformed data
# Summarize the transformed data
summary(wine)## id province price points
## Min. : 1 Length:8380 Min. : 7.00 Min. :80.00
## 1st Qu.:2096 Class :character 1st Qu.: 31.00 1st Qu.:88.00
## Median :4190 Mode :character Median : 45.00 Median :90.00
## Mean :4190 Mean : 52.52 Mean :89.98
## 3rd Qu.:6285 3rd Qu.: 60.00 3rd Qu.:92.00
## Max. :8380 Max. :2500.00 Max. :98.00
##
## year year_factor has_cherry has_chocolate has_earth
## Min. :1996 2014 :2046 Mode :logical Mode :logical Mode :logical
## 1st Qu.:2011 2013 :1819 FALSE:4955 FALSE:7827 FALSE:6907
## Median :2013 2012 :1505 TRUE :3425 TRUE :553 TRUE :1473
## Mean :2012 2015 : 815
## 3rd Qu.:2014 2011 : 582
## Max. :2015 2010 : 502
## (Other):1111
## cherry_year chocolate_year earth_year
## Min. : 0.0 Min. : 0.0 Min. : 0.0
## 1st Qu.: 0.0 1st Qu.: 0.0 1st Qu.: 0.0
## Median : 0.0 Median : 0.0 Median : 0.0
## Mean : 822.5 Mean : 132.7 Mean : 353.7
## 3rd Qu.:2012.0 3rd Qu.: 0.0 3rd Qu.: 0.0
## Max. :2015.0 Max. :2015.0 Max. :2015.0
## # Check the distribution of new binary features
table(wine$has_cherry)##
## FALSE TRUE
## 4955 3425table(wine$has_chocolate)##
## FALSE TRUE
## 7827 553table(wine$has_earth)##
## FALSE TRUE
## 6907 1473# Visualize the interaction features over time
ggplot(wine,
aes(x = as.numeric(year),
y = has_cherry)) +
geom_point() +
labs(title = "Cherry Presence Over Time")
ggplot(wine,
aes(x = as.numeric(year),
y = has_chocolate)) +
geom_point() +
labs(title = "Chocolate Presence Over Time")
ggplot(wine,
aes(x = as.numeric(year),
y = has_earth)) +
geom_point() +
labs(title = "Earth Presence Over Time")
4. Preprocessing
library(caret)
library(fastDummies)
wine <- wine %>%
preProcess(method = c("BoxCox",
"center",
"scale")) %>%
predict(wine) %>%
dummy_cols(select_columns = "year_factor",
remove_most_frequent_dummy = TRUE,
remove_selected_columns = TRUE)
# View the transformed data
head(wine)## id province price points year has_cherry
## 1 -2.206642 Oregon 0.7146905 -1.033841 -0.03425331 FALSE
## 2 -2.202427 Oregon -1.4139991 -1.033841 0.33313680 FALSE
## 3 -2.198830 California 0.8225454 -1.033841 -0.40146088 FALSE
## 4 -2.195581 Oregon 0.2408520 -1.367723 -0.76848588 FALSE
## 5 -2.192571 Oregon -1.2418658 -1.367723 -1.13532834 FALSE
## 6 -2.189736 Oregon -1.0109945 -1.367723 1.06846470 FALSE
## has_chocolate has_earth cherry_year chocolate_year earth_year
## 1 FALSE TRUE -0.8313464 -0.2657899 2.165110
## 2 TRUE FALSE -0.8313464 3.7646929 -0.461775
## 3 FALSE TRUE -0.8313464 -0.2657899 2.163804
## 4 FALSE FALSE -0.8313464 -0.2657899 -0.461775
## 5 FALSE TRUE -0.8313464 -0.2657899 2.161193
## 6 FALSE FALSE -0.8313464 -0.2657899 -0.461775
## year_factor_1996 year_factor_1997 year_factor_1998 year_factor_1999
## 1 0 0 0 0
## 2 0 0 0 0
## 3 0 0 0 0
## 4 0 0 0 0
## 5 0 0 0 0
## 6 0 0 0 0
## year_factor_2000 year_factor_2001 year_factor_2002 year_factor_2003
## 1 0 0 0 0
## 2 0 0 0 0
## 3 0 0 0 0
## 4 0 0 0 0
## 5 0 0 0 0
## 6 0 0 0 0
## year_factor_2004 year_factor_2005 year_factor_2006 year_factor_2007
## 1 0 0 0 0
## 2 0 0 0 0
## 3 0 0 0 0
## 4 0 0 0 0
## 5 0 0 0 0
## 6 0 0 0 0
## year_factor_2008 year_factor_2009 year_factor_2010 year_factor_2011
## 1 0 0 0 0
## 2 0 0 0 0
## 3 0 0 0 1
## 4 0 0 1 0
## 5 0 1 0 0
## 6 0 0 0 0
## year_factor_2012 year_factor_2013 year_factor_2015
## 1 1 0 0
## 2 0 1 0
## 3 0 0 0
## 4 0 0 0
## 5 0 0 0
## 6 0 0 15. Running KNN
# Split the data into 80/20 training and test sets
split <- createDataPartition(wine$province,
p = 0.8,
list = FALSE)
train <- wine[split, ]
test <- wine[-split, ]
# Train the NN model with cross-validation
fit <- train(province ~ .,
data = train,
method = "knn",
tuneLength = 15,
metric = "Kappa",
trControl = trainControl(method = "cv",
number = 5))
# Make predictions and get confusion matrix
conf_matrix <- confusionMatrix(predict(fit, test),
factor(test$province))
# Display key metrics (Accuracy and Kappa)
cat("Accuracy: ", conf_matrix$overall['Accuracy'], "\n")## Accuracy: 0.6294082cat("Kappa: ", conf_matrix$overall['Kappa'], "\n")## Kappa: 0.37950046. Kappa
TODO: Explain how to determine whether a Kappa value represents a good, bad, or other outcome.
Explanation:
- Kappa measures agreement between predicted and actual values,
adjusting for chance.
- Values closer to 1 indicate better agreement, while values near 0
suggest predictions are no better than random guessing.
- General interpretation:
- < 0.2 → Poor
- 0.2 - 0.4 → Fair
- 0.4 - 0.6 → Moderate
- 0.6 - 0.8 → Substantial
- 0.8 - 1.0 → Almost perfect
7. Improvement
TODO: Explain how to interpret the confusion matrix and how to improve predictions. Confusion Matrix Interpretation:
TP (True Positive): Correct positives. TN (True Negative): Correct negatives. FP (False Positive): Incorrect positives. FN (False Negative): Incorrect negatives.
Improvements: - If many misclassifications exist in the confusion matrix, adjusting K might help. - Standardizing and normalizing data can improve accuracy. - Adding or engineering new features (e.g., interactions) could refine predictions. - Using weighted KNN (where closer neighbors have more influence) can help in cases of uneven class distribution.