Motivation

Web applications are closely coupled with our everyday life. We use them at work, for shopping or during our leisure time. One of the main goals of the web applications if to fulfil the needs of the users. To reach this goal a user friendly environment has to be established. In this environment every user should be able to find the information he or she needs.

Webster dictionary describes confusion as a situation in which people are uncertain about what to do or are unable to understand something clearly. When the user visiting the application reaches the tolerability limit of the time invested to find the desired information, he or she may leave the application and never come again.

The occurrences of the user confusion hinder the business of organizations whose employees use a digital solution. Confusion slows down the work, frustrates the user and may lead to mistakes. It typically threatens the novice users (e.g., steep learning curves), but happens also to the experienced ones. Sometimes, the user needs to use some rarely-used feature or re-learn to use the application after a major update. In fact, the employees may be so entrenched in the ways they use a particular software, that its major update also means a major software adoption issue for the organization. This means that focusing solely on the novices onboarding is not enough; the applications have to provide the users with help and assistance when needed (but not bother them with any prompts in moments when these would be perceived as disturbing).

In our work we developed a confusion detection method, able to identify confused user in real time within a web application based on mouse events. We deployed this solution to the Yeself project, an existing application for online guides.

Method

We use user activity consisting of mouse events to predict moments of users' confusion. The raw activity data is first split into time windows, processed and the key features are extracted from user's mouse movements within windows.

We model a user by using a histogram representation of the features (see figure below for a schema of the proposed confusion prediction method). Before the actual prediction, the module has to be trained on labelled user behavioral data. If there are no labelled data for the user, heuristics based on the statistics of mouse movement features computed from the training data (e.g., mouse velocity features averaged over all (non)-confused users in the training data) are used for detection of his/her confusion.

Confusion prediction diagram.

Evaluation

To evaluate the proposed method, we conducted a user study on FIRO-tour travel agency's web site which has more than 200,000 unique monthly users and already a quite good level of usability. There were 60 participants who took part in the study. Eye tracking data were recorded for comparison with state of the art solution. To gather the ground truth labels of user confusion, we attached a software button with exclamation mark into the right top corner of the site (see Figure below). Participants were informed to use this button through whole session, anytime they need to. If the participants clicked the button, they were shown a questionnaire, in which they were asked to describe the reasons for pressing the button.

Confusion button on FIRO-tour website during the user study.

By evaluating our method on gathered data we reached precision in confusion prediction of 41% on confused time windows and 98% on not confused time windows. As a predictor we used just the logistic regression classifier and grid search algorithm for finding optimal parameters. We also developed a production version of our method for the YesElf project.

Publications

Hucko, M.

Identification of user confusion in a web application. Master thesis, Slovak University of Technology in Bratislava, May 2019. 72p. pdf (in Slovak)

Hucko, M., Gazo, L., Simun, P., Valky, M., Moro, R., Simko, J., Bielikova, M.

YesElf: Personalized Onboarding for Web Applications. In Proc. of UMAP 2019, Late Breaking Result Track. ACM. 2019. p. 39-44.

to Homepage

to Teaching

to the Top