INFS7410 Project - Part 1

version 1.4

Assignment Preamble

Due Date

30 August 2024, 16:00 Eastern Australia Standard Time

Weight

This assignment (Project - Part 1) constitutes 20% of the overall mark for INFS7410.
(Part 1 + Part 2 = 40% of the total course grade)

Completion Requirements

• You should complete it individually.
• You can check the detailed marking sheet provided alongside this notebook.
  • see INFS7410-project-part-1-marking-sheet.pdf

Prerequisites Checker

You should have already tackled the activities from pracs week1-5, including:

• Indexing corpus (prac-week1)
• Implementing retrieval functions (prac-week3)
• Implementing rank fusion methods (prac-week4)
• Implementing query expansion and reduction from pseudo relevance feedback(prac-week5)
• Performing evaluation, visualisation, and statistical significance test (prac-week2)

Tips

• Start early to allow ample time for completion.
• Proceed step-by-step through the assignment tasks.
• Most of the assignment relies on knowledge and code from your computer practicals. However, be prepared for challenges that may require additional time and effort to solve.

Overview of the IR Workflow

To conduct an experiment for the IR tasks in this project, we generally follow three key stages: Indexing -> Retrieval -> Evaluation. Each stage involves a corresponding portion of data from the collection. A collection typically comprises a corpus, queries, and a qrel. These are illustrated below:

What you need to do:

You are expected to deliver the following:

• Correct implementations and evaluations of the methods required by this project specifications.
• Write-up about the retrieval methods used, including:
  • formula that represent the method you implemented.
  • code that corresponds to the formula.
  • evaluation settings with explanation followed.
  • discussion of the findings.

Both the implementations and write-ups must be documented within this Jupyter notebook.

Required Methods to Implement

• Indexing
  • Pyserini index command: You first need to index the new datasets introduced in this project. Each dataset should be made into a separate index. Check how we use the command to build index for given collections from week1-week3.
• Ranking functions
  • BM25: Implemented by yourself, not using the one from Pyserini. Check week3.
• Query reformulation methods
  • Pseudo-Relevance Feedback using BM25 for Query Expansion: Implemented by yourself. Check week5.
  • IDF-r Query Reduction: Implemented by yourself. Check week5.
• Rank fusion methods
  • Borda: Implemented by yourself. Check week4.
  • CombSUM: Implemented by yourself. Check week4.
  • CombMNZ: Implemented by yourself. Check week4.

Parameter Tuning:

N.B. ONLY TUNE WITH TRAINING QUERIES.

• Tune the parameters in your BM25, Query Expansion, and Query Reduction implementations. Conduct a parameter search over at least 5 carefully selected values for each method, and 5 pairs when the method involves two parameters.

• For Rank fusion methods, focus on fusing the highest-performing tuned run from each of the BM25, Query Expansion, and Query Reduction implementations.

Required Evaluations to Perform

In this project, we provide three datasets with sampled queries the original BEIR version. You first task is to check these datasets and get familiar with the size, content, format, and consider how to process them. Pay particular attention to the differences between these datasets and the MSMARCO collection we used in the pracs.

In Part 1 of the project, you are required to perform the following evaluations:

1. Run your BM25 with k=1.2, b=0.75 with the test_queries of SciFact， NFCorpus, and TREC-COVID as the baselines.

2. Tune the parameters for BM25, Query Expansion, and Query Reduction with the train_queries of SciFact and NFCorpus. Refer to the Parameter Tuning section outlined above.

3. Report the results of each method from tuning in a table. Perform statistical significance analysis across the results of the methods and report them in the table (e.g., comparing Method_A with para-setting_a on dataset_1 with baseline on the same dataset).

4. Select the best parameter setting of the methods from Sci-Fact and NFCorpus seperately, and run with test_queries of TREC-COVID. Report the results in the table, follow the same requirements listed above.

5. Create a gain-loss plot comparing BM25vs. Pseudo-Relevance Feedback Query Expansion using BM25, as well as plots comparing BM25 vs. each rank fusion method on TREC-COVID. Using the baseline BM25 and the two best parameter settings from SciFact and NFCorpus for Query Expansion.

Discussions

1. Comment on trends and differences observed in your results. Do the methods work well on SciFact and NFCorpus also generalise to TREC-COVID? Is there a method that consistently outperforms others on all the datasets?
2. Provide insights into whether rank fusion works or not, considering factors like runs included in the fusion process and query characteristics.
3. Discuss the results obtained from SciFact and NFCorpus compared to those on TREC-COVID. Do ranking methods perform well? Why or why not? Include the statistical significance analysis results comparing the baseline with the tuned methods, and each method against another (including fusion, query expansion and reduction).

Evaluation Measures

Evaluate the retrieval methods using the following measures:

• nDCG at 10 (ndcg_cut_10): Use this as the primary measure for tuning

All gain-loss plots should be produced with respect to nDCG at 10.

For statistical significance analysis, use the paired t-test and distinguish between p < 0.05 and p < 0.01.

How to submit

Submit one .zip file containing:

1. this notebook in .ipynb format
2. this notebook saved as a .pdf by navigating to the menu and selecting File -> Save and Export Notebook As -> HTML, then open and Print it with your browser.

N.B.

• Ensure that the code is executable. Include all your discussions and analysis within this notebook, not as a separate file.

• Don't add any runs, indexes in the zip file! (If you do this, the file will be too big and you will encounter errors in submitting.)

• Submit the file via the link on the INFS7410 course site on BlackBoard by 30 August 2024, 16:00 Eastern Australia Standard Time, unless you have received an extension according to UQ policy, which must be requested before the assignment due date.