STA 141A
Fall 2019
Homework 4
Due: December 5 (Thursday), 11.59 pm
Submit the assignment electronically through Canvas. Electronic submission
must be in the form of a zip folder (with extension .zip, .7z, etc.) containing
two files: (i) your answers (.pdf file); (ii) R codes used (.R file). Alternatively,
the assignment can be submitted in the form of a R Markdown file (.Rmd).
Honor Code: “The codes and results derived by using these codes constitute my
own work. I have consulted the following resources regarding this assignment:”
(ADD: names of persons or web resources, if any, excluding the instructor, TAs,
and materials posted on Canvas.)
Problem Statement: The goal is to compare k-means clustering and hierarchical clustering methods,
in a real-data clustering problem.
1. The data-set customers_data.csv contains eight variables measured on 440 instances:
(a) CHANNEL:customer’s Channel - Horeca (Hotel/Restaurant/Cafe’) or Retail channel;
(b) REGION: customer’s Region - Lisbon, Oporto or Other.
(c) FRESH: annual spending (in US dollars) on fresh products;
(d) MILK: annual spending (in US dollars) on milk products;
(e) GROCERY: annual spending (in US dollars) on grocery products;
(f) FROZEN: annual spending (in US dollars) on frozen products;
(g) DETERGENTS_PAPER: annual spending (in US dollars) on detergents and paper products;
(h) DELICATESSEN: annual spending (in US dollars) on delicatessen products.
• Import customers_data.csv as a data frame with header and make a summary
of its variables. (5 points)
• Extract the variables FRESH and FROZEN and store them in a separate data frame,
called customers_2. (5 points)
• From this new data frame, provide a scatter-plot matrix of FRESH and FROZEN via the
function ggpairs from the package GGally. (5 points)
2. Estimate a k-means clustering partition on FRESH and FROZEN. For k = 1, . . . , 10, run 100
times the following procedure: (5 points)
• draw randomly the 80% of observations in customer_2 and use them as a training
data-set. The remaining observations will constitute the test data-set; (5 points)
• run the function kmeans on the training data-set with k centers, 20 random starts and
100 maximum iterations; (5 points)
• use the estimated centers to allocate the observations of the test data-set to a specific
group and derive the relative vector of assignments; (10 points)
• calculate the deviance within estimated groups in the test data-set. (10 points)
Then, for each k, average the deviance within groups over the 100 runs. (5 points)
Finally:
• plot this average over the number of clusters and decide the optimal number of clusters
using the elbow criterion; (5 points)
• re-apply kmeans with the selected number of clusters, 20 random starts and 100 maximum
iterations, and derive the estimated cluster memberships; (5 points)
• provide a scatter-plot matrix of FRESH and FROZEN conditional on the estimated cluster
memberships via the function ggpairs from the package GGally and comment
about the shape of FRESH and FROZEN over groups. (5+5 points)
3. • Estimate a hierachical partition on FRESH and FROZEN by the complete linkage using
the number of clusters selected above. (5 points)
• Derive the estimated cluster memberships. (5 points)
• Provide a scatter-plot matrix of FRESH and FROZEN conditional on the estimated cluster
memberships via the function ggpairs from the package GGally.(5 points)
• Comment about the shape of FRESH and FROZEN over the estimated groups and compare
this outcome to the k-means one. (5+5 points)