A short paper on “Calibration: A Simple Way to Improve Click Models” by Alexey Borisov, Julia Kiseleva, Ilya Markov, and Maarten de Rijke will be presented at the ACM International Conference on Information and Knowledge Management (CIKM 2018) in Turin, Italy, and will be published in the conference proceedings.
Click models are important and widely used tools for interpreting user behavior in Web search. As for many machine learning algorithms, their prediction performance strongly depends on the hyperparameters used for training. We show that click models trained with suboptimal hyperparameters are not well calibrated. This means that their predicted click probabilities do not agree with the observed proportions of clicks in the held-out data. We adapt a non-parametric calibration method called isotonic regression to repair the discrepancy between the click probabilities predicted by a model and the proportion of clicks in the held-out data. We show that isotonic regression significantly improves click models trained with suboptimal hyperparameters in terms of perplexity, and that calibrated click models are less sensitive to the choice of hyperparameters than the original (non-calibrated) ones. Interestingly, the relative ranking of existing click models in terms of their predictive performance changes depending on whether or not their predictions are calibrated. We therefore advocate that calibration becomes a mandatory part of the click model evaluation protocol.