1. Abstract

In the rapidly changing economic environment, bankers are key stakeholders in the country’s economy who should be able to make quick and accurate decisions,especially pertaining to their customers. As churn affects the gross revenue of corporate banks, it is crucial to know who will likely attrite and take action  to prevent  a  higher  churn  rate.  As  part  of our  feature  engineering, we identified 3 customer segments using K-Prototype clustering to further our analysis. By utilising selected features from the credit card churn data set and testing it out on several models, the XGBoost model, overall, performs well in predicting customer churn, with the highest F1 Score of 0.878957. To reduce the churn rate, targeted marketing campaigns towards the identified segments would be pivotal.

2. Introduction

In financial services, customer churn is of particular concern to companies such as credit unions, banks, insurance agencies, and credit card companies. Attrition rates can reach as high as 25-30% for these companies (Kaemingk, 2018). Research has shown that corporate banks lose 10-15% of gross revenues annually  to  customer  churn   (Karthikeyan   et  al.,  2017). With the highly competitive bank scene, consumers are  spoilt  for choices  for their credit card services, alarming corporate banks who wish to increase their profits. Not to mention, the lofty transaction fees and poor exchange rates deter users from using cards (Teope, 2021). Hence, we aim to predict the likelihood of customer cancellation to allow banks to take measures to proactively retain customers and to remain relevant in this competitive industry.

Our analysis is based on a credit card churn dataset from Kaggle (Goyal, 2021). The dataset includes 10, 127 data points and 23 attributes (Appendix, Table A1). The classification problem ofpredicting credit card cancellation takes on a binary variable (Attrition Flag) and four distinct models: Logistic Regression, Random Forest, Gradient Boost, and XGBoost will be explored. The most suitable model will be chosen by analysing the trade-offs between model performance and potential business loss of inaccuracies.

3. Data-Preprocessing

3.1 Removing Data Points

The data is relatively clean as there were no missing values in the data set. However, it contained interim workings from previous projects that we removed for the purpose of this report. The affected columns are ( 1) Naive Bayes classifier for month 1 and (2) Naive Bayes classifier for month 2. Next, we decided to remove the “client number” as it is the index of customers which affected our prediction models. The team also removed the “Unknown” response in the Marital Status and Income Category because, without a clear marital status and income range, the outcome of the prediction will be affected. After the data cleaning, there are 20 features that we used for our predictive model, of which 6 are categorical features and 14 numerical features with 8,348 rows.

3.3 Train-Test Split

We then split the data into training and test sets with an 80/20 split where the target variable (y) is the Attrition Flag and the predictor variable (X) takes on all the other features. The training and test sets contain 6,678 and 1,670 data points respectively.

3.4 Data Scaling & One-Hot Encoding

Next,  we proceeded to normalise the  dataset  after the train/test  split by using the MinMaxScaler in SKlearn which translates the numerical  features  individually between zero and one. The use of MinMaxScaler over StandardScaler was decided due to the non-negative constraint of our data points.

Figure 3. Scatter Plot of Transaction Amount and Transaction Count Based on Attrition Flag

The clear separation of data suggests that there are hidden customer segments within our dataset. Thus, we decided to leverage our knowledge of clustering techniques to identify these clusters and their unique properties which are further elaborated in Section 5.

4.4 Feature Engineering

Based on the observation we initially introduced 5 interaction effects variables into our dataset, for the features with the highest correlation (Appendix, Table A2). Additional necessary feature engineering will be covered in its respective methods.

5. Clustering

5.1 Methodology

The algorithm the group decided on to cluster our data set is K-Prototypes under the ‘kmodes ’ package. Unlike K-means clustering which clusters only numerical data, K-Prototype is able to account for both categorical and numerical variables which is more representative of our data.

The data was clustered on a copy of the main dataset which was scaled using the “MinMaxScaler” . We decided on  scaling  as our dataset contains numerical values of different units. Once the K-Prototype model identified the customer segment, we added a new feature “cluster”  to our main data set which we used for our predictive models.

Our initial model saw the data being clustered based on the target variable “attrition_flag” . 2 of the 3 clusters were either 100% attrited or not attrited, which severely skewed our predictive model results. Hence, we decided to drop the “attrition_flag” column before running the K-Prototype model.

In reference to the elbow plot in Figure A3 of the Appendix, we decided to break the data into 3 clusters. Table 1 shows the breakdown of the more significant variables we used to identify the properties of each cluster and we labelled them as such:

5.2 Customer Segments

( 1) High-Risk  Customers: Lowest credit limit but high accumulated unpaid credit (‘Total_Revolving_Bal’). It would be wise to monitor these groups of customers closely as they have accrued a high credit balance with the bank.

(2) Premium Customers: Healthiest credit score with moderate balance credit. These clients fall into the higher income category and have higher transacting amounts using the credit card.

(3) Low-Risk Customers: Moderate credit score with low usage. This group of customers are punctual in terms of credit billing and has a low threat of defaulting on credit loans.

Table 1. Parameters and Decision Variable

5.3 Data Exploration

6.2 Random Forest

Random Forests are  suited  for capturing complex relationships in data, making them appropriate for analysing churn where factors might interact in a non-linear manner. With the inclusion of diverse control variables, Random Forest can handle the complexity and identify key features contributing to churn. It can also handle the ordinal nature of card types effectively. However, being an ensemble method, Random Forest sacrifices interpretability for improved predictive performance. We fitted the training data using the random forest method, which searches for the best feature among a random subset of features. We fitted 5 folds for each of 162 candidates, totalling 810 fits.  The maximum depth is 15 with 2 splits and 150 estimators.

6.3 Gradient Boost

Gradient Boosting Machine can sequentially correct errors and handle subtle relationships in the data, capturing non-linearities in factors that contribute to churn rates which the logistic regression model is unable to capture. Given that gradient boosting works well with imbalanced datasets that are structured, it is an efficient model for our large-scale data set. We fitted 5-folds for each of 324 candidates, totalling 1620 fits. The maximum depth is 4, with 10 splits and 200 estimators.

6.4 XGBoost

XGBoost is a more regularised form of Gradient Boosting, which makes it perform better than Gradient boosting. We fitted 5-folds for each of 243 candidates, totalling 1215 fits. The maximum depth is 5, with 300 estimators.

7. Model Evaluation

7.1 Loss Matrix

The team decided to include another dimension of model performance by analysing the cost of false predictions. The cost of False Negative (FN) cases is high as card cancellation translates to the loss of an entire customer’s lifetime value. On the other hand, False Positive (FP) cases are lower as false attrition cases would mean inefficient allocation of resources by the bank in an attempt to retain the customer. Table 2 shows the full breakdown of the cost matrix.

Table 2. Loss Matrix in Dollar Terms

7.2 Model Selection

Using the F1 score and Estimated Losses as our performance metric, we selected the XGBoost as our predictive algorithm as it outperformed all other models after cross-validation and in the test set as seen in Table 3. More in-depth analysis of  the individual models can be  found in Appendix B. Logistic Regression performed badly on sensitivity and F1 Score while still having a comparable accuracy. As detecting positives (and therefore sensitivity) is highly important to the bank, this model is not useful for the company. Meanwhile, Random Forest and Gradient Boosting could still be applied as they deliver good performance in those metrics.

Table 3. Summary of Model Performance on Out-Of-Sample Data

7.3 Model Interpretation

Figure 5 highlights  the top 10 significant  factors that determine credit card attrition based on our XGBoost model. We can conclude that Credit Limit and Customer Loyalty are the main factors that will determine if a customer will churn. In Section 8, we will elaborate more on how the bank can leverage this information and employ marketing efforts to address these areas.

Figure 5. Top 10 Feature Importance

7.4 Threshold

As mentioned in Section 7.1, the cost of false predictions is costly for banks. The models above use the default threshold of 0.5 in order to predict classes, which might not be optimal. As a result, we decided to push our model a step further by optimising the threshold value of our XGBoost model.

The nature of recall as a performance metric is such that it is negatively related to the threshold and will only decrease as the threshold increases. Thus,  we decided to use the F1 score as a constraint of sorts, finding the point of intersection between the F1 score and Recall after cross-validation. This equilibrium point is the mathematical derivation of a “sweet spot” between model performance and threshold. The optimal threshold is 0.222 (Appendix, Figure A5).

After tuning the threshold value of our model, we can see improvements in the F1 Score and Estimated Losses of our improved XGBoost model. Understandably, lowering the threshold affected the specificity and accuracy of our model but we felt the slight decrease in these matrices is compensated by a way higher detection of churning customers, which is crucial to the bank.

However, the tuned threshold is relatively low at 0.222, going against the standard that thresholds should be higher for financial institutions due to the high cost of bad debt, emphasising the need to reduce false negative cases. Despite this, we believe our tuning methods are accurate but more data needs to be provided to further refine our model parameters. All in all, the selected tuned XGBoost model delivers great Out-of-Sample performance and can be confidently deployed by the bank to predict churn rates.

Table 4. XGBoost Performance Before & After Threshold Tuning

7.5 Limitations

Our XGBoost model is only useful for banks and not for other financial service companies in general as it considers internal features like the credit card type, which are bank-specific. The latter doesn't state the reasons for specific customer churning. This makes the bank able to predict the churning customers but not necessarily able to take the right corrective actions.