Here's how we can improve online learning for deaf students

COVID-19 rapidly and extensively drew students to online learning in 2020.

From the outset, YouTube, TED, MOOC and Zoom stepped up to support students in a physically restricted environment. At the same time, Artificial Intelligence’s automatic sound-to-text captioning helped media services to include DHH students.

The market and technologies seemed to be inclusively expanding the beneficiaries to anybody, any time and any place. However, online learning is not entirely inclusive.

This is because DHH students’ hearing impairment clashes with one of the assumptions of online learning models. These students have to follow the teacher’s message just with their eyes. This means that online learning can over-burden the visual organ of DHH students, while non-DHH people don’t register this at all.

We interviewed students and teachers in schools for DHH teenagers (Figure 1).

Online learning videos too fast

Twelve of the students participating in the interview complained of the video being too fast. The students, who are usually more familiar with sign language than text, felt it too fast even when sign language subtitles were included.

DHH students claim that on-screen sign language is interpreted too quickly, even when performed at a consistent speed. This is likely the result of their visual burden; DHH students are relying on visual stimuli for two pieces of information – voice (be it text or sign language) and visuals (the overall image).

Merely slowing the video not a solution

All teachers participating in the interview agreed that DHH students have broader reading skills than non-DHH students. Classrooms usually consist of a few students who are excellent at reading video subtitles' text, and a majority of lower-level students who find it hard to do so.

Ordinary materials do not benefit most DHH students because they contain too much content. However, we don’t think the solution is to be found in DHH-specific materials because the market for them is small and the price is therefore high.

Both text subtitles and sign language can work

DHH students vary in their preference of text subtitles versus sign language. Seven students preferred the former, and seven the latter. Three students said they could actually watch the video without subtitles, despite the fact two of them have a profound impairment.

Online learning services can include DHH students by adding sign language to existing subtitle options. Still, problems remain when it comes to the subtitle interface.

The first problem is recognising who is talking in situations in which subtitles remain on one side of the screen. Non-DHH students quickly identify the speaker by their tone of voice. However, DHH people seek clues on the image. But these clues don't appear when for example a debate overheats and the video doesn't show the speaker. And so, DHH people have to infer who is speaking using just the context of the conversation written on the text.

Some recent studies have suggested solutions in supporting speaker tracking via multimodal partitions and speech bubbles.

Subtitle size and language choice an imperfect art

Larger-sized subtitles can hide the video, while smaller ones become hard to read. In both cases, DHH students miss the video material either because they cannot see it or are intently focusing on understanding the subtitles. The students in the interview demanded the function to control the subtitles’ size and position.

Challenging, or indeed entirely wrong, words and phrases frustrate DHH students. While errors and complications matter for non-DHH students too, they can at least compare the voice information with the subtitles. DHH students lack an alternative source of information to double-check wrong interpretations, misunderstandings, and the knowledge gap.

One solution to challenging words is simple; additional comments made in sign language or using images can help DHH students to understand challenging dialects, foreign languages, and professional jargon. One student suggested that drawings and sign language subtitles could really help with these challenges.

Artificial Intelligence, too, helps with with subtitle accuracy. However, all interviewees complained about the inaccuracy of the auto-subtitle system.

Redesigning online for DHH students

DHH students’ demand for online learning points to two new design principles. First, this group of students has more desire for Context-Based Preference Adjustment than non-DHH students. Their broad reading skills require speed regulator and playback content blocks so that they can return to the shot they missed.

Subtitles can improve their understanding if they are rescaled and relocated according to the context and have enough sign-text language and positioning options.

Secondly, DHH students require Interactive Dictionary Assistance. Such a dictionary would support DHH students in avoiding misunderstandings that can come about from the video’s message being transmitted through eyes only.

An interactive interface would improve the DHH students’ follow-up if, for instance, they were able to choose between click-and-explain and automatic show-up and between text and sign language.

Online learning services should integrate these design principles, which would involve providing a control panel to allow for easy interface customisation and a panel for interactive dictionary assistance.

The classroom has spoken: DHH students' real-life needs for online learning are now known. Ears usually supplement incomplete and sometimes wrong information transfer that has come through the eyes. But an accumulation minor troubles that have come about from propagating a one-size-fits all design put DHH students at a real disadvantage.

If we don’t recognise this, it could be detrimental to DHH students’ ability to climb the ladder of education and improve their lives. And this is why our research group is developing DHH-student-friendly design guidelines for online learning services.

Thanks to Junyong Noh (Professor, Graduate School of Culture Technology, KAIST) and JinHwan Choi (MSc Student, Graduate School of Artificial Intelligence, KAIST) for their contributions to this article.