Hello, if you have any need, please feel free to consult us, this is my wechat: wx91due
COMP3009J – Information Retrieval
Programming Assignment
Due Date: Friday 31th May 2024 at 23:55 (i.e. end of Week 14)
Before you begin, download and extract the files ``small_corpus.zip’’ and ``large_corpus.zip’’ from Brightspace. These contain several files that you will need to complete this assignment. The README.md file in each describes the files contained in the archive and their format1 .
The main objective of the assignment is to create a basic Information Retrieval system that can perform preprocessing, indexing, retrieval (using BM25) and evaluation.
Both corpora are in the same format, except for the relevance judgments. For the small corpus, all documents not included in the relevance judgments have been judged nonrelevant. For the large corpus, documents not included in the relevance judgments have not been judged.
For this assignment, you should write several independent programs, each of which is contained in one file2. The list of programs is below, with descriptions of each. You may choose not to implement all the programs (see the “Grading” section below). However, an A+ grade can only be awarded if all these programs have been written correctly and efficiently.
It is ESSENTIAL that all programs can be run as a standalone command-line program, without requiring an IDE/environment such as IDLE, PyCharm, Jupyter, etc.
Non-standard libraries (other than the Porter stemmer provided) may not be used. Do not use absolute paths (the path to the corpus will always be provided to your program).
What you should submit
Here, “independent programs” means that they should not import anything from one another. If you write a function that is helpful in multiple programs, copy/paste it. This is, of course, not good programming practice in terms of reusability of code. However, it helps with the grading process.
Programs:
index_small_corpus.py
It must be possible to pass the path to the (unzipped) small corpus to this program as a command-line argument named “-p”3:
This program must perform the following tasks:
- Extract the documents contained in the corpus provided. You must divide the documents into terms in an appropriate way (these are contained in the ``documents’’ directory of the corpus. The strategy must be documented in your source code comments.
- Perform stopword removal. A list of stopwords to use can be loaded from the stopwords.txt file that is provided in the ``files’’ directory of the corpus.
- Perform stemming. For this task, you may use the porter.py code in the ``files’’ directory.
- Create an appropriate index so that IR using the BM25 method may be performed. Here, an index is any data structure that is suitable for performing retrieval later.
This will require you to calculate the appropriate weights and do as much pre-calculation as you can. This should be stored in a single external file in some human-readable4 format. Do not use database systems (e.g. MySQL, SQL Server, SQLite, etc.) for this.
The output of this program should be a single index file, stored in the current working directory, named “21888888-small.index” (replacing “21888888” with your UCD student number).
4 Here, “human-readable” means some text-based (i.e. non-binary) format. It should be possible to see the contents and the structure of the index using a standard text editor.
query_small_corpus.py
This program allows a user to submit queries to retrieve from the small corpus, or to run the standard corpus queries so that the system can be evaluated. The BM25 model must be used for retrieval.
Every time this program runs, it should first load the index into memory (named “21888888- small.index” in the current working directory, replacing “21888888” with your UCD student number), so that querying can be as fast as possible.
This program should offer two modes, depending on a command-line argument named “- m”. These are as follows:
1. Interactive mode
Example output is given below.
Interactive mode is activated by running the program in the following way:
./query_small_corpus.py -m interactive -p /path/to/comp3009j-corpus-small
2. Automatic mode
Automatic mode is activated by running the program in the following way:
5 You will need to decide how many results to store for each query.
evaluate_small_corpus.py
The program should calculate the following metrics, based on the relevance judgments contained in the ``qrels.txt’’ file in the ``files’’ directory of the corpus):
- Precision- Recall- R-Precision- P@15- NDCG@15- MAP
./evaluate_small_corpus.py -p /path/to/comp3009j-corpus-small
index_large_corpus.py
query_large_corpus.py
evaluate_large_corpus.py
Otherwise, this program should perform the same tasks as evaluate_small_corpus.py, except that the input results file should be named “21888888-large.results” (replacing “21888888” with your UCD student number).
Sample Run (Interactive)
Loading BM25 index from file, please wait.
Enter query: library information conference
Note: In all of these examples, the results, and similarity scores were generated at random for illustration purposes, so they are not correct scores.
Sample Run (Evaluation)
bpref: 0.345
Grading
- Document reading and preprocessing: 15%- Indexing: 20%- Retrieval with BM25: 20%- Evaluation: 15%- Efficiency: 15% (as evidenced by the performance on the large corpus)- Programming style (comments/organisation): 15%