Customer Segmentation with Clustering

University of Cambridge PACE - Data Science Career Accelerator Mini-Project

• Unsupervised Learning

• K-Means
• Scikit-learn
• PCA
• t-SNE

Overview

In this project I worked with a transactional dataset that contained almost one million shopping orders placed between 2012 and 2017, and aggregated it to ~66,000 individual customers.

The brief was to segment those customers down using unsupervised learning so that the business could target the groups with different marketing strategies.

Cleaning The Dataset

The raw dataset needed cleaning before it was usable for further analysis. Currency strings needed to be parsed into floats, customer age and order recency had to be derived from birthdate and order dates.

After aggregating by customer, five features were kept: number of orders placed, recency, customer lifetime value (CLV), average unit cost, and age.

Finding Anomalies Before Clustering

Before clustering, I compared three anomaly detection methods: interquartile range (IQR), one-class SVM, and isolation forest. IQR flagged a customer as anomalous once two or more of their features fell outside the normal range, which caught 2.9% of customers, so I matched that same contamination rate for the other two methods.

To pick between them, I measured Jaccard similarity between each pair of methods rather than trusting any one in isolation. Isolation forest had the strongest overall agreement with the other two, so I used it to remove outliers ahead of clustering. A PCA projection of the flagged points showed a reasonable separation, with the two components explaining 65% of total variance between them.

Choosing K and Running K-means

Instead of choosing the number of clusters by eye, I checked against four methods: the elbow method, silhouette score, the Davies-Bouldin index, and hierarchical clustering. All methods suggested k=2, except the Davies-Bouldin index which ranked k=5 the highest, followed by k=2.

Customers were segmented into two groups, primarily split on engagement. Cluster 0 represents customers that have placed a small amount of orders (median=6), have low CLV (median=£954.70), and have older recency (median=3847). On the other hand, cluster 1 represents a high value group of customers: a median of 18 orders, more recent activity, and a median CLV of £3,400.