Actions
  • shareshare
  • link
  • cite
  • add
add
auto_awesome_motion View all 2 versions
Publication . Article . Preprint . 2015 . Embargo end date: 01 Jan 2015

The Anatomy of Relevance: Topical, Snippet and Perceived Relevance in Search Result Evaluation

Chuklin, Aleksandr; de Rijke, Maarten;
Open Access
Published: 26 Jan 2015
Publisher: arXiv
Abstract
Currently, the quality of a search engine is often determined using so-called topical relevance, i.e., the match between the user intent (expressed as a query) and the content of the document. In this work we want to draw attention to two aspects of retrieval system performance affected by the presentation of results: result attractiveness ("perceived relevance") and immediate usefulness of the snippets ("snippet relevance"). Perceived relevance may influence discoverability of good topical documents and seemingly better rankings may in fact be less useful to the user if good-looking snippets lead to irrelevant documents or vice-versa. And result items on a search engine result page (SERP) with high snippet relevance may add towards the total utility gained by the user even without the need to click those items. We start by motivating the need to collect different aspects of relevance (topical, perceived and snippet relevances) and how these aspects can improve evaluation measures. We then discuss possible ways to collect these relevance aspects using crowdsourcing and the challenges arising from that.
Comment: SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance
Subjects

Information Retrieval (cs.IR), FOS: Computer and information sciences, H.3.3, Computer Science - Information Retrieval

[1] B. Carterette. System effectiveness, user models, and user utility: a conceptual framework for investigation. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 903-912. ACM, 2011. [OpenAIRE]

[2] O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 621-630. ACM, 2009.

[3] O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web, pages 1-10. ACM, 2009.

[4] L. B. Chilton and J. Teevan. Addressing people's information needs directly in a web search result page. In Proceedings of the 20th international conference on World wide web, pages 27-36. ACM, 2011.

[5] A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In Proceedings of the 21st

Funded byView all
NWO| Modeling and Learning from Implicit Feedback in Information Retrieval
Project
  • Funder: Netherlands Organisation for Scientific Research (NWO) (NWO)
  • Project Code: 2300171779
,
NWO| Digging archaeology data: image search and markup (DADAISM)
Project
  • Funder: Netherlands Organisation for Scientific Research (NWO) (NWO)
  • Project Code: 2300186891
,
EC| LIMOSINE
Project
LIMOSINE
Linguistically Motivated Semantic aggregatIon engiNes
  • Funder: European Commission (EC)
  • Project Code: 288024
  • Funding stream: FP7 | SP1 | ICT
,
NWO| SPuDisc: Searching Public Discourse
Project
  • Funder: Netherlands Organisation for Scientific Research (NWO) (NWO)
  • Project Code: 2300176811
Related to Research communities
Digital Humanities and Cultural Heritage DH-CH communities : CLARIN
moresidebar