Evaluation of user explicit feedback based on his implicit feedback measurement

Metod Rybár

Master thesis project supervised by prof. Mária Bieliková


Motivation

Research has shown to us, that people are lying often, at least once a day. These lies occur not only when dealing with other humans but also when interacting with systems like questionnaires. Lot of questionnaires nowadays are performed online and the results of the questionnaires shall be accurate. Therefore we need to make sure, that the answers are truthful.

Today we deal with issue of non honest answers most often via questioning more people than necessary which should “smooth out” our results in a way, that if in our test group there are some objects that are not honest, quantity of honest subjects will have stronger effect on results.

This has however its downsides. First of all, we need more subjects than necessary and acquiring subjects is often not easy and not cheap. Second of all, if the questions are very personal or emotional, more of our subjects may choose to be not honest. In that case, even more test subjects does not help us.

If we could differentiate honest answers from non honest automatically, we could reduce number of subjects required and also be able to choose only honest answers for our purposes. To do this, we can focus on implicit feedback from subjects such as eye movements or response times that can serve as indicators for non honest answers.

Evaluation

We have performed several smaller and two major experiments at UX group lab at the Research Centre of User Experience and Interaction at FIIT STU Bratislava with more than 120 subjects altogether.

UX group lab at the Research Centre of User Experience and Interaction at FIIT STU Bratislava.
UX group lab at the Research Centre of User Experience and Interaction at FIIT STU Bratislava

After several pilot experiments during which we have improved our methodology the final experiment was attended by 60 subjects. Each subject answered 60 different questions, 30 in honest condition and 30 in non honest condition. For each subject we have collected 60 gaze points per second, pupil diameter and all actions that subjects have performed interacting with questionnaires.

Equipment used for experiment.
Equipment used for experiment

Based on our hypotheses we have created five different metrics that should indicate non honest answers - pupil dilation, number of fixations, longest fixation, average duration of fixation and the first fixation for particular answer. With strongest metrics we have achieved precision of 61/67 percent with recall 51/75 percent for honest and non honest answers respectively. We have also evaluated impact of each metric on accuracy and have found out, that the first fixation was very strong indicator of non honest behavior.

Publications

Rybár, Metod
Evaluation of user explicit feedback based on his implicit feedback measurement. Master thesis, Slovak University of Technology in Bratislava, May 2016. pdf (in Slovak)

Rybár, Metod, Bieliková, Mária
Automated Detection of User Deception in On-line Questionnaires with Focus on Eye Tracking Use. In SMAP 2016. Proceedings of the 11th International Workshop on Semantic and Social Media Adaptation and Personalization, IEEE, 2016, pp. 24-28. pdf

to Homepage to Teaching to the Top

Home
Research
Projects
Publications
Books
SCM
Teaching
Links
Last updated:
Mária Bieliková bielik [zavináe] fiit-dot-stuba-dot-sk
Design © 2oo1 KoXo