PROJECT NAME:
Here to Hear
TEAM MEMBERS:
Simrit Mangat, Daisy Zhang, Samantha Ouyang & Caroline Huang
SCHOOL(S):
Semiahmoo Secondary School, Western Canada High School, Bayview Secondary School, Abbey Park High School
PROJECT DESCRIPTION:
The challenges of connecting in a digital world are amplified for the Deaf and Hard of Hearing community. In particular, young DHH students who have trouble reading and writing English and can only communicate through ASL. Our online platform Here to Hear builds a DHH-accessible environment for building community and enriching language studies. We tackle the difficulty of learning a new language through two ways: subconscious and conscious learning.
On the Here to Hear platform, using deep learning and computer vision, DHH students can learn English through subconscious learning as their ASL will automatically be translated into English text captions and displayed on screen when communicating with other DHH students through a video call. Through this method of subconscious learning and repeated exposure, they are able to draw implicit connections between English and ASL to gain more fluency in English. Research has also demonstrated the effectiveness of subtitles in learning languages, indicating a 10% improvement than without subtitles. This method offers a low pressure, low effort, and non-intimidating approach to improve English literacy while being able to socialize and have fun with friends.
Students on the platform will also engage in conscious learning, which reflects the more traditional way of learning languages, in which an individual’s attention and ability to recall information is actively at play. We achieve this on the Here to Hear platform through three different methods: flashcards, an immersive reading tool, and learning games.
After a call, it saves a transcript of the conversation and uses Natural Language Processing and Named Entity Recognition to automatically generate flashcards using keywords from their conversations, and cues in the immersive reader, which has an English to ASL animated interpreter, allowing them to independently review vocabulary and situate words into sentences. Research has not only shown that flashcards are the most effective method of studying, but they are specifically helpful for children, allowing them to make associations, strengthen the links in their brain, and act as an effective self-testing method as it allows students to engage in active recall. From the flashcards, the platform also presents 3 different games the children can play and earn points from to reinforce the material on the flashcards in a fun and engaging manner: write, learn, and match. Research has cited gamification as an effective method for second language education, with improvements in engagement, motivation, and satisfaction. This will help tremendously in motivating young children.
This project was named a Top 10 Global Winner in the Microsoft Imagine Cup Junior AI for Good Challenge.
WHAT MAKES YOUR PROJECT REMARKABLE?:
34 million children worldwide are deaf or hard of hearing. Due to the lack of support for DHH students, only 52% of them complete elementary education in Canada. This means that almost half of them will never be financially stable, independent, or able to land a well-paying job, all because we failed to take action when they were young and support them from the beginning. In fact, the hard of hearing employment rate is almost 33% lower than that of non-disabled Canadians.
What many people fail to understand is that American Sign Language is NOT the visual form of English, but its own language, with different grammar and syntax, and nuances such as tone and expression. Without being able to hear the language, learning English becomes a near impossible task for young DHH children, and they naturally gravitate towards using ASL during childhood development, such that their English falls behind. To make matters worse, the deaf community has been severely impacted by the pandemic. For deaf children, the DHH community at school is their family. As 90% of deaf children are born to hearing parents, who typically do not know how to sign, they feel even more isolated than usual because they are unable to be in a place where they can receive the support they need and feel like they belong.
This however, is only for schools with special DHH programs. There are currently only 13 schools across all of Canada for the deaf, and most of the time, they will have to enroll in private schools who have this special support, due to the large lack of programs in public schools across the nation.
Moreover, the struggles of DHH students are often ignored. Deaf children are usually placed in programs that accommodate their needs only after they fail academically in oral/auditory-verbal based programs. At that point, the critical years of language development are gone. English is also one of the primary languages that is used to communicate with the DHH students, which is a problem because the majority of DHH students also have trouble learning English. The lack of equity and inclusivity in current education for DHH significantly hinders their future opportunities.
Hearing impairments and its consequences on children and their education are largely unspoken about. Our project is remarkable due to its unique ability to address a large problem that has little governmental action taken, and that has been exacerbated by the pandemic, making it not just a frontrunner in this space as one of the only solutions, but all the more prevalent as well given the current circumstances.
Moreover, we took special effort to involve the DHH community in every step of the process, gaining insight into the problem from someone who is a part of the DHH community himself, as well as an experienced DHH program educator. Our solution has not only been validated by Microsoft through our achievements, but by members of the DHH community themselves in the value they would find from it, citing that “it shows a lot of promise” and that they “haven’t heard of something like this before, but would be very interested in trying it out!”
Our focus on learning English through a variety of different scientifically-backed methods also makes Here to Hear remarkable. Firstly, with a novel method of subconscious learning by doing nothing more than chatting as per normal, secondly, through flashcards to enable active recall, thirdly, with an immersive reader to support young children with visual and interactive associations to words, and lastly, through gamification to ensure their motivation.
Our large emphasis on building community also makes Here to Hear unique, as DHH students can chat with each other as they would normally through ASL. By reinforcing relationships and building community, we can combat the feelings of social isolation created by the pandemic. Social connection boosts mental health, as it can lower anxiety and depression, regulate emotions, and lead to higher self-esteem and overall well-being. In fact, a review of 148 studies showed that individuals with stronger social relationships had a 50% increased likelihood of survival.
Using various methods of Artificial Intelligence, we present a unique solution to this issue that is both innovative, yet feasible. Backed by leading technology companies such as Microsoft and validated by members of the deaf community themselves, we are eager to take our project further in building out a fully functional prototype, conducting more user interviews, and receiving mentorship from leaders in the space.
PROJECT WEBSITE:
https://heretohear.simritmangat.repl.co/
PROJECT VIDEO:
https://youtu.be/XXeMr6dwt8o