DTS304TC Machine Learning

DTS304TC Machine Learning

Coursework - Assessment Task 1

 You  are  encouraged  to  explore  a  variety  of  advanced  techniques  to  improve  your  model's predictive power.

1.   Utilizing different classifiers, for example, XGBoost.

2.   Implementing methods for outlier detection and treatment.

3.   Creating model ensembles with varied validation splits.

4.   Addressing issues of class imbalance.

5.   Applying feature engineering strategies, such as creating composite attributes.

6.   Implementing  alternative  validation  splitting  strategies,  like  cross-validation  or  stratified sampling, to enhance model tuning.

7.   Additional innovative and valid methods not previously discussed.

You will be awarded 3 marks for successfully applying any one of these methods. Should you incorporate two or more of the aforementioned techniques, a total of 6 marks will be awarded.

Please include code comments that explain how you have implemented these additional techniques. Your code and accompanying commentary should explicitly state the rationale behind the selection of these supplementary strategies, as well as the underlying principles guiding your implementation. Moreover, it should detail any changes in performance, including improvements, if any, resulting from the application of these strategies. An additional 4 marks will be awarded for a clear and comprehensive explanation. To  facilitate a streamlined review and grading process, please ensure that your comments and relevant code are placed within a separate code block in your Jupyter notebook, in a manner that is readily accessible for our evaluation.

Additionally, utilize the entire dataset and the previously determined optimal hyperparameters and classification pipeline to retrain your top-performing classifier. Then, apply this model to the features  in   'dts304tc_a1_heart_disease_dataset_external_test.csv',  which  lacks  true  labels,  to produce a set of predictive probability scores. Save these probabilistic scores in a table with two columns: the first column for patient IDs and the second for the output  classification labels. Export this table to a file named external_test_results_[your_student_id].csv. Submit this file for evaluation. In the external evaluation conducted by us, your scores will be benchmarked against the  performance  of models  developed  by  your  classmates. You will receive four marks for successfully completing the prescribed classifier retraining and submission process. Additionally, your classifier's benchmark ranking—based on its performance relative to models developed by your peers—will be assigned five marks, contingent upon your standing in the ranking.

d) Result Analysis (8 Marks)

•     For your best-performing model, compute critical performance metrics such as precision, recall,  specificity, and the F1 score for each class. Additionally, generate the confusion matrix based on your internal test set. Ensure that the code for calculating these performance metrics, as well as    the resulting statistics, are clearly displayed within your Jupyter notebook. For ease of review, position these elements towards the end of your notebook to provide direct access to your outcomes.

•     Conduct a feature importance analysis by utilizing the feature importance scoring capabilities provided by your chosen classifier. What are the top three most important features for classifying  this medicial condition?  If your best performing model does not offer feature importance scoring, you may utilize an alternative model for this analysis. Present the results of the feature importance analysis within your Jupyter notebook using print statements or code comments.

Question 2:  Ensemble Learning (18 marks):

Students are required not to use AI models, such as ChatGPT, for assistance with this question. You should give clear calculation steps and explain the relevant concepts using your own words.

(a)        Majority Voting with Independent Classifiers (8 Marks)

1.   Given individual classifiers C1, C2, and C3 with statistically independent error rates of 40%,  35%, and 35% respectively, calculate the accuracy of the majority voting ensemble composed of classifiers C1, C2, and C3. Provide the  steps you use to  determine the  ensemble's  accuracy,  assuming the classifiers' decisions are statistically independent. (5 Marks)

(Hints: Calculate the ensemble 's accuracy by summing the probability that all classifiers are correct with the probabilities of exactly two classifiers being correct while the third is incorrect)

2.   Point out the similarities and differences between the majority voting ensemble method and Random Forest, emphasizing the strategies employed by Random Forest to attain higher accuracy. (3 Marks)

(b)        AdaBoost Algorithm (10 Marks)

Consider the process of creating an ensemble of decision stumps, referred to as Gm , through the standard AdaBoost method.

The diagram above shows several two-dimensional labeled points along with the initial decision stump we've chosen. This stump gives out binary values and makes its decisions based on a single variable (the cut-off). In the diagram, there's atiny arrow perpendicular to the classifier's boundary that shows where   the classifier predicts a +1. Initially, every point has the same weight.

1.   Identify all the points in the above diagram that will have their weights increased after adding the initial decision stump (adjustments to AdaBoost sample weights after the initial stump is used) (2 marks)

2.   On the same diagram, draw another decision stump that could be considered in the next round of boosting. Include the boundary where it makes its decision and indicate which side will result in a +1 prediction. (2 marks)

3.   Will the second basic classifier likely get a larger importance score in the group compared to the first one? To put it another way, will α2  >  α1 ?  Just a short explanation is needed (Calculations are not    required). (3 marks)

4.   Suppose you have trained two models on the same set of data: one with AdaBoost and another with a Random Forest approach. The AdaBoost model does better on the training data than the Random Forest model. However, when tested on new, unseen data, the Random Forest model performs better. What could explain this difference in performance? What can be done to make the AdaBoost model   perform better? (3 marks)

Question 3:  K-Means and GMM Clustering (7 marks)

Students are required not to use AI models, such as ChatGPT, for assistance with this question. You should give clear analysis steps and explain the relevant concepts using your own words.

1.   Reflect on the provided data for training and analyze the outcomes of K-Means and GMM techniques. Can we anticipate identical centroids from these clustering methods? Please state your reasoning. (3 marks)

2.   Determine which of the given cluster assignments could be a result of applying K-means clustering, and which could originate from GMM clustering, providing an explanation for your reasoning. (4 marks)

In [ ]: # to student: this is an code snapshot showing how to use ray tune framework to # you are responsible for completing the code, debugging and making sure it work

import sklearn.datasets

from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import accuracy_score

from sklearn import metrics

import os

from ray.tune.schedulers import ASHAScheduler

from sklearn.model_selection import train_test_split

from ray import tune

from ray import train

def train_rf(config: dict, data=None):

# Initialize the RandomForestClassifier with the configuration

classifier = RandomForestClassifier(

n_estimators=config["n_estimators"],

max_depth=config["max_depth"],

min_samples_split=config["min_samples_split"],

min_samples_leaf=config["min_samples_leaf"],

class_weight="balanced",

random_state=42

)

# Fit the RandomForestClassifier on the training data

X_train = data[0]

y_train = data[1]

X_validation = data[2]

y_validation = data[3]

# To Be filled: Train your Random Forest Classifier here

# To Be filled: Evaluate the classifier on the validation set here and get e

# Send the accuracy to Ray Tune

train.report({'accuracy': accuracy})

# note that X_train, y_train, X_validation, y_validation are your training and v

tunable_function = tune.with_parameters(train_rf, data=[X_train, y_train, X_vali

def tune_random_forest(smoke_test=False):

# Define the search space for hyperparameters

search_space = {

"n_estimators": # setup your search space here

"max_depth": # setup your search space here

"min_samples_split" : # setup your search space here

"min_samples_leaf" : # setup your search space here

}

# Define the scheduler for early stopping

scheduler = ASHAScheduler(

max_t= # setup your ASHA parameter here,

grace_period=# setup your ASHA parameter here,

reduction_factor=# setup your ASHA parameter here

)

# Set up the tuner

tuner = tune.Tuner(

tunable_function,

a1_ASHAScheduler_ray_tune_for_student

tune_config=tune.TuneConfig(

metric="accuracy",

mode="max",

scheduler=scheduler,

num_samples=1 if smoke_test else 200,

),

param_space=search_space,

)

# Execute the tuning process

results = tuner.fit()

return results

# Run the tuning function

best_results = tune_random_forest(smoke_test=False)

best_trial = best_results.get_best_result(metric="accuracy", mode="max")

# Get the best trial's hyperparameters

best_params = best_trial.config

# Print the best hyperparameters

print("Best hyperparameters found were: ", best_params)

# Initialize a new RandomForestClassifier with the best hyperparameters

best_rf = RandomForestClassifier(**best_params, random_state=42)

# The remaining part of your code continues ....