Publication:
Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks

Simple item page
dc.contributor.authorCarterette, Ben
dc.contributor.authorJones, Rosie
dc.contributor.departmentUniversity of Massachusetts - Amherst
dc.date2023-09-22T21:09:13.000
dc.date.accessioned2024-04-26T09:35:24Z
dc.date.available2024-04-26T09:35:24Z
dc.date.issued2007-01-01
dc.descriptionThis paper was harvested from CiteSeer
dc.description.abstractWe propose a model that leverages the millions of clicks received by web search engines to predict document relevance. This allows the comparison of ranking functions when clicks are available but complete relevance judgments are not. After an initial training phase using a set of relevance judgments paired with click data, we show that our model can predict the relevance score of documents that have not been judged. These predictions can be used to evaluate the performance of a search engine, using our novel formalization of the confidence of the standard evaluation metric discounted cumulative gain (DCG), so comparisons can be made across time and datasets. This contrasts with previous methods which can provide only pair-wise relevance judgments between results shown for the same query. When no relevance judgments are available, we can identify the better of two ranked lists up to 82% of the time, and with only two relevance judgments for each query, we can identify the better ranking up to 94% of the time. While our experiments are on sponsored search results, which is the financial backbone of web search, our method is general enough to be applicable to algorithmic web search results as well. Furthermore, we give an algorithm to guide the selection of additional documents to judge to improve confidence.
dc.identifier.urihttps://hdl.handle.net/20.500.14394/9924
dc.relation.urlhttps://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1042&context=cs_faculty_pubs&unstamped=1
dc.source.statuspublished
dc.subjectComputer Sciences
dc.titleEvaluating Search Engines by Modeling the Relationship Between Relevance and Clicks
dc.typearticle
dc.typearticle
digcom.contributor.authorCarterette, Ben
digcom.contributor.authorJones, Rosie
digcom.identifiercs_faculty_pubs/26
digcom.identifier.contextkey1300264
digcom.identifier.submissionpathcs_faculty_pubs/26
dspace.entity.typePublication
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Ben_Carterette.pdf
Size:
182.37 KB
Format:
Adobe Portable Document Format
Thumbnail Image
Name: Ben_Carterette.pdf
Size: 182.37 KB
Kind: Adobe PDF
Collections
Computer Science Department Faculty Publication Series