A comparison of a traditional ranking format to a drag-and-drop format with stacking

Date of Award

2013

Degree Name

M.A. in General Psychology

Department

Department of Psychology

Advisor/Chair

Advisor: William F. Moroney

Abstract

This study investigated the differences between a modified version of the traditional ranking format (MTF) and a novel ranking format called the BINS format. The BINS format utilizes drag-and-drop technology to rank alternatives, allows respondents to indicate distance between ranks, and also allows respondents to assign ties to the same alternatives. Seventy-two participants completed two ranking tasks: a ranking of items from the Rokeach Value Survey--Form D (RVS) and a ranking of aspects according to how important they were in a participant's decision to attend the University of Dayton (UD). Participants used the MTF to complete one ranking task, and the BINS format for the other. Four variables were examined for each ranking format: Completion Time (as recorded by a computer control system and as self-reported by participants), Usability on the System Usability Scale (SUS), Format Preference, and Number of Repositionings (as recorded by a computer control system and as self-reported by participants). Participants completed the RVS ranking task more quickly using the MTF when compared to the BINS format. There were no significant differences in completion time when participants ranked aspects related to UD. However, for both the RVS and aspects related to UD, significantly more participants self-reported that the BINS format allowed them to complete their ranking task faster than the MTF. Participants rated the BINS format as significantly more usable than the MTF. The majority of participants (78%) preferred to use the BINS format more than the MTF. Participants reported repositioning alternatives (ranking an alternative and then re-ranking the same alternative) significantly more often using the BINS format than the MTF. There was not a significant difference in actual repositionings between the MTF and the BINS format as reported by the computer control system. Overall, the results of this study established that the BINS format is a clear improvement over the MTF. The BINS format outperformed the MTF on measures of usability, preference, and reported number of repositionings. Furthermore, the BINS format reduces respondent burden by displaying an ordered list of ranked alternatives throughout a ranking task. By capturing information on the distance between ranks and by permitting ties between alternatives, the BINS format allows researchers to collect rich ranking data that is also compatible with factor analytic techniques. These unique features of the BINS format make it an ideal tool for implementation in the field of electronic survey research.

Keywords

Psychological tests Software Comparison, Surveys Software Comparison, Psychometrics Data processing, Psychology; quantitative psychology; survey research; ranking; drag and drop; bins; usability; stacking; psychometrics

Rights Statement

Copyright © 2013, author

Share

COinS