2015 TestNav CogLab

A usability study of the TestNav platform and several test item interactions and stimuli was conducted at the Arkansas School for the Blind (ASB) on May 26–28 (including one student who was tested at her home) and the Maryland School for the Blind (MSB) on June 1–3, 2015. The interaction types included radio buttons, checkboxes, text entry, extended text entry and inline choice. The stimuli included testing the following:

• Whether students had experience with and understood how to interact with a tabbed interface used to present multiple passages of text.
• Whether the test question should be included in the label of the answer choice interaction.
• Different ways of marking up paragraph numbers in multiple paragraph passages.

A total of 12 students participated in the study: Seven in Arkansas and five in Maryland. All students were legally blind and had a variety of experience with various screen readers, ZoomText, and/or mobile devices for accessing digital content. For the purpose of this study, the students used a Windows 7 computer running Firefox and Job Access with Speech (JAWS), version 15. Two students also used a refreshable Braille display.:

The findings from the study indicated the following:

• Most students didn’t know the JAWS Navigation Quick Keys, for example, ‘H,’ ‘A,’ ‘B,’ ‘E,’ ‘X,’ etc. Some knew a few of these shortcuts but not all, and no one used short cut keys regularly as a means of navigation.
• None of the students chose to navigate with the tab key at any point in the study unless they were prompted to do so, or when they were problem solving about how to navigate through the tabbed passages.
• Students generally knew what headings were, but most did not know how to bring up a headings list, nor could they explain what the heading levels meant. Only one student noticed and was bothered by a section end page where heading levels were skipped.
• Students knew that radio buttons and checkboxes were used for selecting multiple choice items, but few could explain the difference.
• When the JAWS Navigation Quick Keys were explained, students caught on quickly. However, when navigating on their own, they tended to revert to their learned behavior of using the down arrow. This may be due to the fact that students encounter very little accessible content and using the down arrow is a strategy they are taught in order to not miss important information. The only shortcut key that students used consistently without prompting was the “B” key to navigate to the “Next” button.
• The text passages that were separated on in-page tabs were extremely difficult for all students. First and foremost, students didn’t have a conceptual understanding of “tabs,” and often confused them with the tab key on the keyboard or alt-tab to move among open programs on the computer. When tabs were explained as a way of displaying one passage at a time, all students had extreme difficulty navigating and selecting a different tab, as well as understanding (and using) the 3-key combination (JAWS key-alt-m) to jump to the beginning of the passage. JAWS prompted the students by voicing, “Use JAWS key plus Alt plus M to move to the controlled element.” Students did not know what was meant by “controlled element.” If a student was able to navigate to a tab within the tabbed interface, it was by accident and they couldn’t replicate the behavior to navigate to a tab on a different panel.
• Most students had difficulty interacting with the inline choice item. While many students were familiar with the concept of a combo box, they were confused by the fact that JAWS read “Choose dot dot dot button.” They did not understand why it was labeled as a button when it contained a list of multiple choice items.
• Half of the students said they liked it when the item question was included in the answer group label. The other half didn’t like it and thought it was confusing. When the question was embedded in the label of the interaction, several students immediately pressed the down arrow to skip the question and navigate to the answer choices. In doing so, they missed hearing the first radio button choice. When the question is included in the interaction label, the first answer choice is read immediately after the question. Hitting the down arrow caused the question and the first answer choice to be skipped. This was not the behavior that the students expected and it caused confusion. When the question wasn’t included in the label, all students knew that they needed to navigate back to hear the question and they knew that they could either navigate back with the up arrow or by pressing shift-H to move to the question heading.
• Approximately half of the students said they liked it when the numbered paragraphs were marked up as an ordered list, but this was only after it was explained and it was demonstrated how they could use the shortcut key ‘L’ to navigate to the list and the shortcut key ‘I’ to quickly navigate to a list item (e.g. paragraph). After having this demonstrated to them, some students were very excited about how quickly they could navigate from the question to the paragraph they needed, but the majority of the students did not initially notice that JAWS notified them that there was a list. They quickly navigated through the notification and didn’t listen to it. Even though it was easy for the students to learn how to use the shortcut keys to navigate the ordered list, students made it very clear that they have never seen numbered paragraphs organized in this manner. Without extensive practice and instruction, we believe that students would not remember to navigate through passages with list commands because this is not a common tactic used in their instructional materials. It is important to note that even when the passage was marked up as an ordered list, students could still use “P” to navigate by paragraph, which was a tactic that a few students knew to use.
• Approximately half of the students said they liked it when JAWS announced the word “paragraph” before each paragraph number. Other students said they did not like it. One of these students, who preferred Braille access to the screen reader, said she didn’t like it because it gave her more to read.
• As an overarching comment, we’d like to emphasize that all students, not just those with disabilities, have different experiences with digital content and instruction. This means there is a very wide range of skills and expectations based on these previous experiences. This study demonstrates how inexperience can lead to confusion when the test design assumes certain skills, for example, the difficulties students had when navigating items with tabbed passages and interacting with items that don’t make sense, such as the inline choice on a button.

Given the above findings, here are our recommendations:

• Offer online practice tests so students (and teachers) can review and rehearse the different interaction types and stimulus presentations, as well as the relevant Screen Reader Navigation Quick Keys.
• For a given test, inform students what interaction types are included, and provide reminders of the relevant Screen Reader Navigation Quick Keys.
• Even with practice items and reminders, don’t rely on students knowing, or using, the Screen Reader Navigation Quick Keys.
• Ensure that similar item types are as consistent as possible. For example, code multiple choice interactions so that JAWS announces each radio button and checkbox the same (e.g., “[item] [answer] radio button not checked [n] of [n]” or “[item] [answer] checkbox not checked [n] of [n].” )Also allow radio buttons and checkboxes to be selected using the same key (enter or space bar).
• In the item question, or answer label, inform students what type of interaction is required, e.g., multiple choice single answer, multiple choice multiple answer, short answer (text entry), essay (extended text entry), combo box (inline choice), etc.
• Identify interaction types for what they are, i.e., don’t call a combo box a button.
• Ideally, don’t separate multiple stimuli on in-page tabs. Minimally offer skip links to multiple stimuli above the tab panel and disable the tabbed interface from being announced and interacted with by screen reader users. We recommend additional user studies as alternative access is explored for this interaction.
• Don’t include questions in answer group labels. While some students said they liked it, it was unexpected and confusing to others and would therefore impose an unnecessary barrier for some students.
• Mark numbered multiple paragraph passages as an ordered list. While many students won’t remember or use the screen reader shortcut keys to navigate, it might help some and will not interfere with other navigation techniques and should not cause any confusion for students, especially if they are introduced to the concept through practice items.
• Don’t announce the word “paragraph” in front of paragraph numbers by default. It requires extra reading for Braille users and some students said it was not helpful. Perhaps this could be explored as a possible personal needs and preferences setting. Some students were confused by the numbers when announced before the paragraphs, but most just ignored them. It would be worth exploring different options and conducting additional user studies with students with a variety of print disabilities to determine how best to present numbered paragraphs and numbered lines so that reading fluency is not affected.
• Finally, simplify, simplify, simplify. To the extent possible:
  • Minimize the number of interaction types in a given test.
  • Make each test item consist of a single question.
  • Be consistent in the presentation of similar stimuli, interactions, and answers.