Qualitative and Quantitative Study: Accessibility During a Pandemic

ROLE: Conducting interviews, creating surveys, analysis & research paper write-up

DURATION:  September 2020 – December 2020

TEAM: Erin Ansari, Chloe Keilers, Murtaza Tamjeed, Juliann Thang

*results and data from this study cannot be released as there is no IRB approval.

Overview

COVID-19 has forced many educational institutions to pivot towards remote learning as a replacement for in-person classes. The Deaf/Hard of Hearing (D/HH) community is especially impacted by this transition. The advent of masks and plexiglass in physical spaces, as well as the shift to online learning has created communication barriers which uniquely affect D/HH students. This study took a further look into how D/HH students are able to adapt to remote learning as part of the pandemic, as well as the impact of masks on automatic speech recognition (ASR) tools.

This was a semester long group research project.

Qualitative Research

The first step for this project was conducting a qualitative study. Two research questions were created to guide our studies:

RQ1: Which communication-related tools and technologies changed for D/HH students since COVID-19 began?
RQ2: How has the change impacted the educational experience of D/HH students?

Participants

Participants (n = 4) were recruited and had three requirements to be part of the interview:

1. Be deaf, Deaf, or Hard of Hearing

2. Be in academics before COVID-19

3. Currently participate in an academic setting during COVID-19 

Interviews

Questions were created as a group to outline the structure of the interview. All questions were open-ended to allow a wide range of answers and detail from participants. Each interview had at least one observer to take notes. Interviews were conducted via an online video communication application and were recorded and transcribed for further analysis with participant approval.

A transcription tool was used to aid in transcribing the interview, however we found that it was not very accurate due to the different deaf accents our participants had. A team member who observed in every interview was able to aid in this and help with any miscommunication issues during the interview. Also, some participants opted to use American Sign Language (ASL) during the interview, which needs to be inputted and translated correctly in the transcription.

Reflection

To further expand the qualitative section of this study, future studies should seek responses from those who are not D/HH but are a part of the community, such as interpreters and captioners. They should also reach a wider sample of D/HH individuals of varying age, ethnicity, and academic level.

Quantitative Research

The second part of the study consisted of quantitative research. A survey was created via Google Forms to gather data from American D/HH students (n = 9) about their personal experience with in-person and online learning. One participant’s data was removed as the did not meet eligibility requirements for the study.

Survey

Our survey consisted of 13 questions that were split across 5 sections:

1. Identification in the D/HH community + mode of classes before COVID-19

2. Preference of online education before and after COVID-19

3. Preference of in-person education before and after COVID-19

4. Frequency of technology and services for classes before and after COVID-19

5. Open answer text box allowing users to list additional tools used in classes since COVID-19

Most questions were a five-point Likert scale ranging from “Non-Preferred” to “Preferred” along with “Never” to “Always” depending on the type of question asked.

Multiple methods were used to distribute the survey including emailing a mailing list of members a part of the D/HH community and posting of the survey on public forms. We are unable to track how many people were aware of our survey, but we were able to gather a total of 6 unique respondents.

Analysis

A fellow team member who is a data analyst conducted analysis of survey responses via RStudio. Two correlation plots were created, one showing the relationship between video conferencing tools (VCT) and online academic experience, the other showing the relationship between VCT and in-person experience. A mirrored density plot helped to compare VCT usage before and after COVID-19, and a mirror bar chart helped to show the frequency of certain communication tools before and after COVID-19 (Video Conferencing Tools, Automatic Speech Recognition (ASR), Text Communication (TC).

Reflection

Due to the small sample size, we were not able to generate significant data or correlations from our surveys. Going forward, it would be beneficial to find a way to gather more respondents to conduct more significant data.

Experimental Research

Masks are one of the first barriers of protection to stop the spread of COVID-19 and are a requirement for students and faculty in most educational institutions; however, they exacerbate communication difficulties.. For example, sound becomes muffled with a mask making it difficult for people, especially those who are deaf and hard of hearing, to understand people clearly. Many of those who are D/HH also rely on lipreading when communicating, but most masks block the view of lips. 

One solution for D/HH students are to use captioning or automatic speech recognition (ASR) tools. Since masks generally muffle speech, we wanted to see how accurate this tool can be during a pandemic.

Hypotheses

H: There is no significant difference between the error rate of Automatic Speech Recognition of

speech while wearing a mask vs. not wearing a mask.

 

Ha :There is a significant difference between the error rate of Automatic Speech Recognition of

speech while wearing a mask vs not wearing a mask.

Experiment

Type: Within-group study

Independent variable: Type of ASR tool

Dependent Variable: Rate of errors the ASR tool transcribed

To be a part of the study, participants needed to be a native English speaker with no accent to ensure this was not a limitation. A total of 9 participants were able to be a part of the experiment.

We conducted a within-group study over Zoom to reduce time in recruiting participants and to eliminate individual effects. Two short passages (Carrots for Everyone and Grandfather Passage) were given to read, and two different ASR tools (Ava and Otter.ai) were used to record the participants reading twice, once with the mask and once without the mask.

Analysis

We collected a total of 4 transcriptions from each participant (Ava Mask, Ava No Mask, Otter.ai Mask, Otter.ai No Mask) and identified each error that was made by the ASR. A table was created to layout our data, each cell containing the number of mistakes made. Any human error was not taken into account.

Data analysis lead us to create 2 different graphs. One showed the error rate without a mask on the left x-axis, and would connect to the right x-axis which represented the error rate with a mask. The second was a boxplot of the error rate with a mask and without a mask. 

Reflection

Moving forward, the study could be expanded by gathering more participants to receive a wider array of data. Future studies should consider performing this experiment in a real world setting where external variables such as background noise, distance from speaker, or physical barriers may impact AI speech comprehension and in turn, D/HH student comprehension.