A short paper on Evaluating and Analyzing Click Simulation in Web Search by Stepan Malkevich, Ilya Markov, Elena Michailova, and Maarten de Rijke was presented at the ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2017) in Amsterdam and was published in the conference proceedings.

Abstract:

We evaluate and analyze the quality of click models with respect to their ability to simulate users’ click behavior. To this end, we propose distribution-based metrics for measuring the quality of click simulation in addition to metrics that directly compare simulated and real clicks. We perform a comparison of widely-used click models in terms of the quality of click simulation and analyze this quality for queries with different frequencies. We find that click models fail to accurately simulate user clicks, especially when simulating sessions with no clicks and sessions with a click on the first position. We also find that click models with higher click prediction performance simulate clicks better than other models.