Gesture Cues in Video Calls

UX Design Research & Inclusive Design for Discord

Fall 2021

#UX Design Research

#Prototyping

#Javascript

#Computer Vision

#Accessibility

Program

Major Studio course at MFA Design & Technology at Parsons School of Design

Tools

Figma/FigJam, Javascript, p5.js, handsfree.js

Project Description

Developed during a core studio course at Parsons School of Design, this project aims to expand digital communication, specifically video call communication. This project goes through extensive research, idea conception, prototyping, and a working code implementation involving hand tracking with handsfree.js.

Context

Computer-mediated communication, or any socialization that occurs over social media or the internet, is an inclusive & comforting space for many adults with ASD. This is because it requires less on-the-spot decoding of social information, since a person can take time to process and respond to a text message or DM.

Problem Discovery

Video calls can be alienating and distressing for autistic participants. This is because video calls remove any potential for cue assistance through bodily gestures. Furthermore, lack of feedback can cause misunderstandings, resulting in social & conversational insecurity.

The Problem

Shoulder-up video calls lack visual and bodily cues that are often helpful in decoding conversations.

Goal

Rather than further alienating disadvantaged groups with temporary assistive technologies that may not be adopted by everyone, video calls are already a successful and widely used method of communication.

Instead, the video call system itself should be improved, and conversational hand gestures should be reintigrated for cue assistance.

Assistive Technology (AT)

It is important for AT to not be alienating. Oftentimes, this occurs from unwanted visibility resulting from use of AT in public settings. By making the system open to everyone, it fosters empathy in communication between groups of people.

79.6% of adults with ASD prefer social media for socialization and find it to be a more comfortable way to communicate with others. (source)

Project Principles

1. Purposely design for adults with ASD in mind

Currently, assistance widely focuses on children and those caring for them. As a result, it implies that adults with ASD require less support.

2. Actively design againt the infantilization of ASD

Infantilization occurs when people are treated like children and expected to be naive or "pure". This also occurs when someone addresses an accompanying person rather than acknowledging a disabled adult themselves.

3. Design for the target user with the potential to improve it for everyone

Many well-designed interventions have the ability to assist more than just their target group (ex. subtitles, curb cuts, text-to-911...)

4. Intervene digitally to holistically empower offline socialization

Once comfortable with de-codified online communication, people can carry that with them into real life.

User Research

Online Survey

An online survey was shared with classmates and acquaintances. The purpose was discover what messages people would want to be able to gesture and people's perceptions of different hand gestures.

  • 100% would want to convey agreement with 👍
  • 92.3% wish to convey "I have something to say" with ✋
  • 76.9% want to convey disagreement with 👎
  • 61.5% would like to convey they are changing the topic, and a percentage can infer that meaning from a closed fist gesture ✊ when accompanied by the other gestures

Prototypes

1. Testing Physical Cues

Low-tech experiment with neurodivergent & neurotypical individuals to analyze how gestures can provide cue assistance.

Conclusion:

  • It is better to motion when someone wants to speak rather than provide cues when someone's finished speaking.
  • People benefit from personalized signals.

2. Technology Prototype with Handsfree.JS

Test the technology that could add hand gestures & conversational cue gestures into video calls.

Conclusion:

  • Integration into existing software lowers barriers for use.
  • I could input my own gestures, so theoretically others can input their own hand gestures as well.

3. Figma Initial Wireframes

Figma was used to ideate where and how gesture integration could be included in the UI. I settlef on having a hand icon at the top right so it is grouped with the video call view type button.

EDIT: Since then, Discord's UI has changed. As of 2024, if I were to redo this project, I would include it at the bottom.

Final Prototype with Gesture Tracking

Try using it directly from the fullscreen p5.js viewer >>>
In order to use, do the following:
1. Allow webcam access to the browser
2. Click the hand button at the bottom on (it will turn green)
3. For more info, click the carrot button directly under the hand
4. To turn it off, select the hand button again

Figma Components

Final Gesture Set & Reasonings

Future Considerations

If I were to design further:

More Projects

Block Party

Gesture Integration in Video Calls as Assistive Tech

Reimagining Malta's Bus System & App

Buddy System

Motion Design System for Lutron Software

<commonLang>

Let's chat!(´∀`)

Or feel free to email me via the email listed on my resume!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.