2017 TestNav Application Usability Study

The October 2017 TestNav Usability Study for Students with Learning Disabilities was conducted with 17 students in grades 3 through 11 who were identified with a variety of non-visual disabilities typically requiring testing accommodations. Testing for all students took place October 23-27 in Leander, TX, with most students being recruited from the surrounding public school systems or from home school environments.

This Usability Study consisted of 13 items which were selected to compare student perceptions of the usefulness of a variety of item types and tools. Four items consisted of the exact same reading passage and questions, but explored the usability of various screen layouts. Two items different versions of the same simple match test construct. Other items presented students with four different versions of the same complex match construct, an opportunity to try out video speech adjustment and closed caption capabilities, and the ability to test two different colored reading filters.

Overall, student efficiency was a major factor in perceptions of usability. Things like limiting the need for scrolling, the ability to quickly see all answer choices available, to easily see what answers they have selected, the ability to quickly make an answer selection, and the ability to quickly change an answer were all important considerations.

Important finding from this study can be generalized as follows:

• For the four reading layouts tested (Items 1 through 4), more students preferred the layout used in Item 3 than any other (8 out of 17 students). While this was not a full majority of students, a further analysis was done in which reading layout preference choices were compared with student ratings for the reading layout students found most difficult to use. This analysis reveals that the reading layout used in Item 3 was the clear winner in overall student usability, with a score 5 points higher than the closest competitor. Based on comments and observations, students who were less comfortable and/or less experienced in using the mouse scroll wheel/technology tended to prefer the layout used in Item 3, while students who have a strong preference for knowing what the questions are before they begin reading gravitated to either Item 1 or Item 4. No students preferred the layout used in Item 2.
• For the two simple match models tested (Items 5 and 6), the majority of students (10 out of 17 students) found the checkbox interaction of Item 6 easier to use than the drag and drop used in Item 5. Students who expressed a preference for using checkboxes cited the fact that making answer selections was quicker than using the drag and drop method, more similar to what they had experienced with paper tests, and required less physical effort and expertise with the mouse.
• For the four complex match interaction models tested (Items 7 through 10), the match table was most preferred by students (7 out of 17 students), although there was no clear majority. However, a more complete usability analysis of the data revealed that this interaction style was the clear winner in overall student usability, with a usability score 5 points higher than the closest competitor. Students stating a preference for the match table interaction most often commented on the fact that this format allowed them to easily see all answer choices at once.
• For the video options tested, the use of closed captions was viewed as more valuable than controlling the video speed, although slowing down a video was seen as having greater utility than speeding it up.
• For the colored reading filters, students were less likely to see these as useful, although the blue filter had a much higher preference than the yellow filter.