Play, Comic Strips, and Enhancing Gesture-Based Human Computer Interfaces

The Times — They Are A Changin

Keyboards and mice are so last century. The means of interacting with computers is shifting away from only point-and-click to include a variety of gestural interfaces (facial expressions, body posture, hand motions, etc.).

Nintendo’s Wii, Apple’s iPhone, and Microsoft’s Project Natal, for example, are all computing products that not long ago would have been said to possess “non traditional” interface technologies. Now they are quite familiar (or, in the case of Project Natal — will soon be familiar).

A Gesture is Worth a Thousand Data Points

So what does play — apart from the two video game systems just mentioned — have to do with any of this? Glad you asked. Everyone putting effort into using gestures as an input to computing devices is just getting warmed up. There’s so much yet to come (check out oblong industries’ g-speak if you would like to briefly freak out). There’s also a great many challenges to overcome.

Recognizing human gestures and facial expressions is not easy. Not easy for software algorithms, anyhow. It just so happens that humans are very, very good at reading gestures. Just take a gander at the Facial Action Coding System to get a feel for all the subtleties involved in discriminating various facial expressions. And then realize that you read people’s posture and faces effortlessly.

Know How You GOTTA SPEAK UP for Your Hard-of-Hearing Uncle?

Ever notice how adults play with children? Every gesture is wildly exaggerated. The same is often true with the voices and movements in skits, comedies, impressions, and parodies. Comic strips and animations are usually also comprised of caricatures of human speech, emoting, and gesturing.

So how do we help gesture recognition systems along? Enter playful interfaces. Need clearer facial and movement cues? Make the system that elicits, tracks, and responds to gestures a playful one. The users of that system will naturally enhance their responses as they have fun and play along.

Gestural interfaces are relatively immature — not unlike the children with whom adults naturally express their exaggerated play faces and movements. I suspect playful interfaces can help gesture recognition techniques grow up, go off to college, get jobs, and become upright members of society.

The Visual Lexicon of Comic Strips — Starring You

I wish I could say I first had the insight of how play and gestural interfaces could, uh, play nice together and then dreamed up an example. But it went the other way around. I had the idea for a fun project and then from it I came to a handful of ideas on the link between playful technologies and gestural interfaces.

How fun would it be to walk up to a video display that illustrated in real-time your feelings and thoughts in the visual language of a comic strip? Stay with me. I’m talking about displaying all manner of speech balloons and symbols right over your head and around you.

You’ve got a frown on your face and your head is drooping? There’s a storm cloud above you. Your eyes and mouth are wide open? An exclamation point appears hovering over your head. You flash your finger in the air with your face elongated and eyes wide? Yup. You get a light bulb. Speak and a speech bubble shows your words. Curse and the bubble is filled with @%&*!# (known as grawlixes). Eye something amazing and dotted lines trace your gaze. Swing your arms and lines trail their motion. Furrow your brow and squint your eyes to see drops of stress sweat arc away from your temples and forehead. Scream and your speech bubble becomes jaggy and pointed. You get the idea.

The visual lexicon of comics is really quite expansive. It’s quite surprising how cartoonists and comic artists have captured so much of human expression visually. It occurs to me, oddly enough, that there is a compelling link to explore between gestural interfaces and the visual language of comic strips. That said, the larger point here is that enticing users to actively play with a gestural interface could enhance the system’s ability to recognize and act upon users’ gestures — especially in these early days of the field.

UPDATE (July 7, 2011): An old post now with even more comic strippiness