Paper Reading #11: Activity analysis enabling real-time video communication on mobile phones for deaf users

Comments
Shena Hoffmann - http://csce436-hoffmann.blogspot.com/2011/02/paper-reading-11-chronicle-capture.html
Evin Schuchardt - http://csce436spring2011.blogspot.com/2011/02/paper-reading-11-activity-analysis.html

Reference Information
Title: Activity analysis enabling real-time video communication on mobile phones for deaf users
Authors: Neva Cherniavsky, Jaehong Chon, Jacob O. Wobbrock, Richard E. Ladner, Eve A. Riskin
Presentation Venue: UIST 2010: 22nd annual ACM symposium on User interface software and technology; October 4-7, 2009; Victoria, British Columbia, Canada

Summary
This paper describes a system called MobileASL that is designed to help deaf people communicate by mobile phone though real-time video. The researchers have developed algorithms that address the issues of low bandwidth, low processing speed and limited battery life in mobile phones.

In their algorithms they use a dynamic skin-based region-of-interest (ROI) that focuses on a person’s skin to display their hands in higher quality than the rest of the video. They’ve also developed their algorithm to recognize when a user is signing and when they are not so that they can lower the frame rate to save resources. They call this technique Variable Frame Rate (VFR).

Taken from paper: their system, MobileASL

The researchers also evaluated how fifteen users fluent in ASL (American Sign Language) liked their system. They recorded nine different conversations. Five were between strangers and four between people who knew each other well (one pair being a husband and wife). The users talked for five minutes and then the settings on the phone were changed to adjust the ROI and VFR. After each change the users were asked to rate how easy or difficult it was to understand the conversation.

They found that the users had to guess less frequently about what was being said at higher ROI. As for the VFR, the users had to guess more frequently and repeat things when VFR was on. However, their overall study showed that the users did not experience very many conversational breakdowns (they reported breakdowns occurring on average once every third conversation).

They did notice that it was during finger spelling (spelling out a word in Sign Language) that conversational breakdown occurred the most.

Discussion
Overall, this was a really good paper. They explained the different challenges they have overcome and gave an in depth report on their user study. I think this could make mobile phones more useful for deaf users. And as the researchers state in the paper, this form of communication would be faster than texting.

As for future work, they plan to improve the algorithm so that it works better with finger spelling. They also would like to compare this technique to other techniques in video communication. They are also considering making it so that the ROI can track the face through face recognition algorithms.