Analysis of change in users' assessment of search results over time

Maayan Zhitomirsky-Geffet, Judit Bar-Ilan, Mark Levene

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large-scale user study with 86 participants who evaluated 2 different queries and 4 diverse result sets twice with an interval of 2 months. To analyze the results we investigate whether 2 types of patterns of user behavior from the theory of categorical thinking hold for the case of evaluation of search results: (a) coarseness and (b) locality. To quantify these patterns we devised 2 new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and nonlocal changes. Two types of judgements were considered in this study: (a) relevance on a 4-point scale, and (b) ranking on a 10-point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking. However, the majority of these changes were local.

Original languageEnglish
Pages (from-to)1137-1148
Number of pages12
JournalJournal of the Association for Information Science and Technology
Issue number5
StatePublished - 1 May 2017

Bibliographical note

Publisher Copyright:
© 2016 Wiley Periodicals, Inc.


Dive into the research topics of 'Analysis of change in users' assessment of search results over time'. Together they form a unique fingerprint.

Cite this