EMATM0051 Large Scale Data Engineering

Hello, if you have any need, please feel free to consult us, this is my wechat: wx91due

Coursework - EMATM0051 Large Scale Data Engineering
[FinTech and EwDS cohorts]

Summary

This coursework is divided into two parts:

Part 1: A written task (only) to design architecture of a simple application on AWS cloud, where you are required to have a deep understanding of AWS services and how they work together within an application. The design should demonstrate your knowledge of AWS services covered throughout the entire LSDE course.

Part 2: A combined practical and written activity architecting a scaling application on the Cloud, where you will be required to use knowledge gained and a little further research to implement the scaling infrastructure, followed by a report that will focus on your experience in the practical activity together with knowledge gained in the entire LSDE course.

Weighting: This assessment is worth 100% of your total unit 20 credits.

Due: 13:00. Tue 3rd Dec 2024.

Please note that the Category of generative AI use in assessment for this assessment falls in Category 2: Minimal – for example, using spell and grammar checkers to help identify mistakes but not rewrite chunks of text. More information is available on https://www.bristol.ac.uk/bilt/sharing practice/guides/guidance-on-ai/using-ai-in-assessment/.

Please note that all information shown in the screenshots must be in English. The screenshots will be considered invalid if they include any text in non-English languages

Via the LSDE BlackBoard coursework assessment page, submit one pdf file, named using your UOB username (‘username.pdf), containing:

Unit Director: Wei Wei

Part 1: (25%)

We require you to design the architecture for an application, “FinDatabase”, running on AWS cloud. This is an online database which allows user to search, review and download the financial time-series data. The application needs to meet the following requirements:

You should include your own descriptions of the following requirements, with 500-800 words and no more than 2 A4 pages:

You don’ t need to implement these ideas in your lab account.

Part 2: Scaling the WordFreq Application (75%)

Write a report of no more than 2500 words and 18 A4 pages (there is NO minimum), including: Task A, B, C, D, E.

Overview

WordFreq is a complete, working application, built using the Go programming language.

[NOTE: you are NOT expected to understand or permitted to modify the source code in any way]

The basic functionality of the application is to count words in a text file. It returns the top ten most frequent words found in a text document and can process multiple text files sequentially.

The application uses a number of AWS services:

You will be required to initially set up and test the application, using instructions given with the zip download file. You will then need to implement auto-scaling for the application and improve its architecture based on principles learned in the CF course. Finally, you will write a report covering this process, along with some extra material.

Ensure you have accepted access to your AWS Academy Learner Lab account and have at least $10 credit (you are provided with $50 to start with). If you are running short of credit, please inform your instructor.

Refer to the WordFreq installation instructions (‘README.txt’) in the coursework zip download on the BlackBoard site, to install and configure the application in your Learner Lab account. These instructions do not cover every step – you are assumed to be confident in certain tasks, such as in the use of IAM permissions, launching and connecting via SSH to an EC2 instance, etc.

You will set up the database, storage buckets, queues and worker EC2 instance. Finally, ensure that you can upload a file and can see the results logged from the running worker service, before moving on to the next task.

You will need to give a brief summary of how the application works (without any reference to the code functionality) in this Task.

[NOTE: The application code is in the Go language. You are NOT expected to understand or modify it. Any code changes will be ignored and may lose marks.]

Task B – Design and Implement Auto-scaling

This should initially function as follows:

Using the knowledge gained from the Cloud Foundations course, architect, please implement auto scaling functionality for the WordFreq application and demonstrate how you configure the auto scaling policy. Note that this will not be exactly the same as Lab 6 in Module 10, which is for a web application. You will not need a load balancer, and you will need to identify a different CloudWatch performance metric to use for the ‘scale out’ and ‘scale in’ rules. The 'Average CPU Utilization' metric used in Lab 6 is not necessarily the best choice for this application.

Task C - Perform Load Testing

You can ‘purge’ all files from your processing S3 bucket, then you could copy all the .txt files from you uploading S3 bucket to your processing S3 bucket. Please stop the original instance wordfreq-dev and only use the instances that are created by your auto scaling group. Don’t forget to also ‘purge’ all messages in your SQS job queue.

Based on only AWS services and features learned from the Cloud Foundations course, describe how you could re-design the WordFreq application’s current cloud architecture (i.e. not changing the application’s functionality or code) to improve the architecture in the following areas:

Your description should ideally include diagrams and include the AWS services required together with a high-level explanation of features & configuration for each requirement.

Task E – Further Improvements

You don’ t need to implement these ideas in your lab account.

Combine Part 1 and Part 2 to a single PDF. You will also need to give us Your AWS Academy account credentials (username, password) at the end of your report.

[IMPORTANT: Disable autoscaling at end of each lab session: – Desired capacity = 0 ; Minimum capacity = 0. This saves credit and avoids multiple instances from launching and terminating when starting / stopping a lab session]AWS Academy Learner Lab

You are given an AWS Academy Learner Lab account for this coursework. Each account has $50 assigned to it, which is updated every 24 hours and displayed on the Academy Lab page.

Marking

+80%: Outstanding report and implementation. Extensive exploration, analysis and implementation demonstrating deep understanding and reading outside of the CF course and lectures.

Academic Offences

Academic offences (including submission of work that is not your own, falsification of data/evidence or the use of materials without appropriate referencing) are all taken very seriously by the University. Suspected offences will be dealt with in accordance with the University’s policies and procedures. If an academic offence is suspected in your work, you will be asked to attend an interview with senior members of the school, where you will be given the opportunity to defend your work. The plagiarism panel are able to apply a range of penalties, depending the severity of the offence. These include: requirement to resubmit work, capping of grades and the award of no mark for an element of assessment.

Extensions and Exceptional Circumstances

If the completion of your assignment has been significantly disrupted by serious health conditions (including mental health impairment), personal problems, or other similar issues, you may be able to apply for an extension for assessment submission or consideration of extenuating circumstances (in accordance with the normal university policy and processes).