Market Basket analysis in Retail

Abstract

This Master Thesis memory describes a full end-to-end data science project performed in CleverData, a successful start-up specialized in data mining techniques and analytics tools. This project was performed for one of its clients, which is an important retail company from Spain. The aim of the project was both the analysis of the possibly different selling behaviour of the stores or shops of the client and the analysis of customers’ purchase behaviour, also known as Market Basket Analysis, to confirm the hypotheses from the client regarding the existence of different customer purchasing profiles and different store selling profiles in its company. The project was divided in three tasks. The first one was oriented to the study, detection and validation of different behaviour profiles of the shops/stores of the client. This analysis was done by means of a descriptive process using clustering techniques. In order to guarantee a minimum robustness of the profiles obtained, three clustering algorithms were used: a hierarchical agglomerative clustering technique, a partitional clustering technique with a fixed number of clusters (Kmeans) and a partitional clustering technique with automatic detection of the number of clusters (G-means). For each algorithm, the output clusters were analysed and compared. First, the similarity of the composition of the clusters between algorithms was analysed. Secondly, the resulting clusters (each partition) from each method were structurally validated using four Clusters Validity Indexes (CVIs): Minimum Cluster Separation Index, Maximum Cluster Diameter Index, Dunn Index and Davies-Bouldin Index. Finally, the best partition was found from a technical point of view. After that, the client should be able to interpret and validate the meaning of the clusters obtained. Once chosen the partition more meaningful to the client, the second task was devoted to provide a descriptive analysis of the clusters as meaningful as possible to the client. To that end, some common techniques tools were used, as the computation of the centroids of the clusters, and the characterisation of each one of the clusters through the variables used. However, an important obstacle appeared in this task. The number of variables was so high (around 400) that made impossible that the client was able to analyse and summarise the selling behaviour profile of the different shops. The proposed solution was to apply a feature selection approach, taking advantage from the clustering process done, and to make an aggregation process of variables with temporal relationship. This way, the information about the cluster to which each store belonged, was recorded as a label of a new created class variable. Then, a Random Forest ensemble technique was selected and applied to the new dataset. This discriminant technique, in addition to be able to predict an unlabelled new instance or observation, provides information about the relevant attributes for the discrimination purpose (i.e., the ones being used in the trees of the forest). Then, based on those most important attributes, the descriptive analysis of each cluster was done, and it could be interpreted and fully understood by the client. The third task was focused on the analysis of customers’ purchase behaviour through the analysis of the historic purchase tickets recorded from one year. To identify possible different purchase patterns, it was decided to apply an associative model to find out whether some cooccurrences or associations could be identified. Concretely, the association rules model was used. Because the set of clusters was meaningful to the client, it was decided that the analysis of the purchase behaviour would be done locally to each cluster. Therefore, each cluster was examined to discover associations or co-occurrences of purchase patterns among the customers in each cluster. Hence, some association rules were discovered for the purchase patterns in each store. Two strategies were used to generate the rules: the Lift measure and the Leverage measure. To summarise and conclude the analysis, a web page was created where the results were published to make easier the access of the client to the results. Through the memory, it is gradually explained how the project was developed. Since the first step of defining the objectives, until the last results’ delivery. In the project, both the Python language and machine learning libraries were used, as well as the BigML tool, which uses machine learning as a service. At the end of the project, the results accomplished were analysed, and the aims of the project were compared against the initial goals of the project, with satisfactory results, both from the client practical point of view, and from a technical point of view.