Consolidation of usability problems with novice evaluators re-examined in individual vs. collaborative settings

Usability testing can involve multiple users and evaluators. In such cases, consolidating usability problems (UPs) constitutes an essential part of data analysis. In a between-subjects design, this study aims to re-examine a previous study by comparing the results of novice evaluators merging UPs individually vs. collaboratively and to assess the quality of the final UP lists, by computing the merging rate and the accuracy rate, respectively. Law and Hvannberg compared the results of evaluators merging UPs individually vs. collaboratively in a within-subjects design, revealing a tendency towards merging UPs in collaborative settings. In the present study, 45 novice evaluators consolidated four UP lists into a single UP master list while working alone or with a partner. The results showed no significant difference between evaluators in the two settings, suggesting that the UP consolidation process does not benefit from positive group decision effects.