Postgraduate coursework, Information School,
INF6060 Information Retrieval (2023-24)
Review of the University of Sheffield website search system
The aim of this coursework is for you to evaluate how a search system supports users with finding information through searching, i.e. for the kind of information seeking where the starting point is formulating a query. This coursework requires you to carry out a form of ‘expert’ assessment of the University of Sheffield website search system (note: make sure you assess the website search system and not StarPlus, the University library system). You will consider how well the search system meets the needs of its users undertaking their typical search tasks. You are required to submit two assignments. For assignment 1, you will create tasks and queries that you will use to test the search system for assignment 2. For this second assignment you will produce a portfolio (a text tokenisation strategy and 3 evaluation activities) and recommendations for how the search system could be improved. As a piece of academic work, you are expected to engage with the research literature and provide references to justify and support your writing.
If you have questions about the assignments, please contact Dr Sophie Rutter ([email protected]) to discuss. The tasks and queries (assignment 1) are worth 10% of the overall module mark and the portfolio with recommendations (assignment 2) are worth 90% overall. Submission deadline for the tasks and queries is 10am Monday 23 October 2023 viaTurnitin. Submission deadline for the portfolio with recommendations is 10am Thursday 11th January 2024 via Turnitin.
Assignment 1: Tasks and queries (10%)
In weeks 2 and 3 you will learn how to create tasks and queries for testing search systems. For this assignment, you are required to create tasks and queries that you can use to test the University of Sheffield website search system. There is no word count but you should aim to be succinct.
|
Tasks and queries |
What we are looking for |
Mark |
|
Simulated work tasks (SWT) |
Two SWT that are realistic for the search system (i.e. based on what typical users might do). The situation should be different for each SWT. |
35 |
|
Search tasks |
One search task for each of the two SWT tasks. The search task should be categorised according to the research literature (e.g. Toms, Kellar etc). The two search tasks should be different types (e.g. if one is a specific item the other should be general topical). |
35 |
|
Queries |
10 queries for each search task. These queries should be categorised and sufficiently different that you could test different aspects of the search performance. |
20 |
|
References |
All use of the research literature should be acknowledged and cited using the APA style. |
5 |
|
Presentation |
Good use of English and clear presentation style. |
5 |
Assignment 2: Portfolio of activities with recommendations (90%)
For this assignment you will evaluate the University of Sheffield website search system for how well the search system meets the needs of its users undertaking their typical search tasks. In weeks 4-8 you will learn how to conduct different types of evaluation (heuristic evaluation, measuring retrieval system effectiveness, and supporting the users’ search process) and how to develop text tokenisation and processing strategies. Throughout the module, but particularly in weeks 9- 12, you will learn about what is state-of-the-art in information retrieval that you can use to help you develop recommendations.
You are required to produce a portfolio that includes
1. The tasks and queries you used to test the search system
2. Your tokenisation and text processing strategy
3. The results of your three evaluations
4. Recommendations based on your activities and what is considered best practice in the information retrieval research literature.
Each of these sections is described in more detail next.
(1) Tasks and queries.
You must state all the tasks and queries that you have used for the portfolio. You can reuse the tasks and queries created for assignment 1. However,as you become more familiar with the module content and the website to be evaluated you may wish to revise and extend the assignment 1 tasks and queries. It is also recommended that you revise your original tasks and queries in response to the feedback received for assignment 1. Additionally, you should also briefly explain why you have made changes to the tasks and queries.
There is no word limit for the tasks and queries but you should keep your entries succinct. We have suggested an approximate number of words as a guideline.
|
Portfolio - section 1 |
What we are looking for |
Marks |
|
Tasks and queries |
All tasks (simulated work tasks and search tasks) and queries used in the activities.
(guideline up to 100 words per simulated work task) |
1 mark deducted from presentation if not included. |
|
Reflection |
A brief explanation of any changes made to the assessment 1 tasks and queries (guideline 100 words) |
1 mark deducted from presentation if not included. |
(2) Text tokenisation and text processing strategy
|
Portfolio - section 2 |
What we are looking for (minimum requirements) |
Marks |
|
Text tokenisation and processing strategy |
A logical strategy for processing and tokenising text documents to be indexed by the search system. You should justify any assumptions you make when generating this strategy and give concrete examples of how your text processing strategy would transform original raw text documents. |
15 |
(3) Evaluations 1 - 3: You MUST conduct ALL 3 evaluation activities. The minimum
requirements for each evaluation is stated in the table below. You should aim to go beyond this. Some ideas to help you develop and extend your evaluations include
(i) use more tasks and queries,
(ii) compare against website search systems from other universities,
(iii) test on different devices etc.
There is no word limit for the evaluations but you should keep your entries succinct. We suggest 500 words per evaluation as a guideline.
|
Portfolio - section 3 |
What we are looking for (minimum requirements) |
Marks |
|
Evaluation 1: heuristic analysis |
A table that lists the 10 Nielsen heuristics and provides a brief description of whether the search interface deals with the heuristic effectively for 1 of your SWT. A brief summary of the findings. |
15 |
|
Evaluation 2: measuring retrieval system effectiveness |
You should submit 3 different queries for each of your SWTs (i.e. 6 in total) to the University search system and assess the objective performance of the system based on your chosen queries. This will require you to create relevance judgements for the results of each of your queries and to assess the results using appropriate performance metrics up to rank 10. You should state any assumptions you made when assessing retrieved documents for relevance. |
15 |
|
Evaluation 3: supporting the users’ search process |
A table that identifies features of the search system. Features should be categorised using Wilson (2011) and the purpose of each feature should be succinctly described. Brief description of how the user may carry out the two search tasks utilising the categorised features. You should structure this description around a search process model such as Sutcliffe & Ennis (1998).
Short conclusion on how well the system supports the two task types.
|
15 |
(3) Recommendations. You should base your recommendations on the activities you have conducted and supported by findings of the research literature. Use your test result to make recommendations for how the search system could be improved to better meet the search needs of its end users.
IMPORTANT: The word count for the recommendations is 1500. You MUST state this word count on the front cover of the assignment. References are not included in the word count.
|
Portfolio - section 3 |
What we are looking for |
Marks |
|
Recommendation1: Draw on the results of either Evaluation 1 (heuristic analysis) or Evaluation 3 (supporting the users’search process) |
Recommendations should be based on your test results. Recommendations should be made in light of the best practices for search system design and current/future developments in IR research. |
15 |
|
Recommendation2:Draw on the results of Evaluation 2(measuring retrieval system effectiveness) |
Recommendations should be based on your test results. Recommendations should be made in light of the best practices for search system design and current/future developments in IR research. |
15 |
|
References |
All use of the research literature should be acknowledged and cited using the APA style.
|
5 |
|
Presentation |
You should write the recommendations in the style of a corporate report and use appropriately formal language. You may use screenshots to aid your description.
|
5 |