Playing the Game of (Your) Life

Jesse Schell’s Future of Gaming

Jesse Schell is a professor at Carnegie Mellon University’s Entertainment Technology Center, founder of Schell Games, and a former Disney Imagineer. He recently gave a presentation at the 2010 DICE (Design, Innovate, Communicate, Entertain) Summit.

The first 20 minutes of the talk are about the unexpected success of Facebook games. Skip all that and watch the last 10 minutes.

Schell’s “Design Outside the Box” Presentation (video).

The Newest in Gaming Technology? Brushing Your Teeth.

Schell paints a picture of what the future may hold due to the ever-decreasing cost of sensor and communication technologies. He imagines toothbrushes, cereal boxes, public transportation, advertisements, books, and essentially every artifact of your life able to score and track points. He predicts that basic sensors and wireless technologies will eventually be so inexpensive and small that they can be embedded in a cardboard box or in a toothbrush. He’s not talking about video games. He’s talking about living your life like playing a game. Brush your teeth? 50 points. Take the bus instead of driving? 100 points. Eat high fiber cereal? 75 points. Ace your exam or totally nail your piano practice? 150 points. Compete with your friends. Why not?

He doesn’t make this, uh, point in his talk, but let me, ahem, point out that much of our retail commerce experience already works this way. Reward cards, frequent flier miles, sweepstakes, and even those foil lids on yogurt cups — aren’t they all a more primitive form of the points-based game systems Schell is describing?

Winning at Life.

Schell suggests a compelling upside — tax breaks, better insurance rates, credits toward scholarships. All by gaming the system, if you will.

What’s the downside? Obviously there’s concern about privacy here. Schell’s response? How many of us know the small details of the lives of our grandparents or great grandparents? Very few. If we know that we will leave a record of our decisions, might we not make better ones? Read more and better books? Watch less and better TV? Treat our bodies better and maybe even one another better?

The Mud Tub: Go ahead. Play in the mud. Get your hands dirty. You won't get in Trouble. Promise.

Tom Gerhardt’s Mud Tub looks like a ridiculously fun Human Computer Interface. And, in fact, you’re actually encouraged to make a beautiful mess…

The Mud Tub is an experimental organic interface that allows people to control a computer while playing in the mud.

computer, meet mud — By sloshing, squishing, pulling, punching, etc, in a tub of mud (yes, wet dirt), users control games, simulators, and expressive tools; interacting with a computer in a new, completely organic, way. Born out of a motivation to close the gap between our bodies and the digital world, the Mud Tub frees the traditional computer interaction model of it’s[sic] rigidity, allowing humans to use their highly developed sense of touch, and creative thinking skills in a more natural way. 

Be sure to check out the Mud Tub video & the Mud Tub + Lumarca video to get the idea. (In my opinion, the Lumarca video gives a better sense of the mud-based interface than does the overview video.)

A few thoughts and observations:

1. Awesome. I mean seriously. It’s hard to beat playing in the mud.
2. Tom casts the Mud Tub as an experiment in art and design. Though he uses the word “play” here and there, he doesn’t concentrate on its playfulness. Certainly, of course, I would highlight that aspect (as I am here). But, it goes to show how complementary play is to so many disciplines — especially those of the creative persuasion.
3. The Mud Tub is a very literal incarnation of the Digital Sandbox idea I wrote about previously.

Mud Tub users… Note the smile.

Play, Comic Strips, and Enhancing Gesture-Based Human Computer Interfaces

The Times — They Are A Changin

Keyboards and mice are so last century. The means of interacting with computers is shifting away from only point-and-click to include a variety of gestural interfaces (facial expressions, body posture, hand motions, etc.).

Nintendo’s Wii, Apple’s iPhone, and Microsoft’s Project Natal, for example, are all computing products that not long ago would have been said to possess “non traditional” interface technologies. Now they are quite familiar (or, in the case of Project Natal — will soon be familiar).

A Gesture is Worth a Thousand Data Points

So what does play — apart from the two video game systems just mentioned — have to do with any of this? Glad you asked. Everyone putting effort into using gestures as an input to computing devices is just getting warmed up. There’s so much yet to come (check out oblong industries’ g-speak if you would like to briefly freak out). There’s also a great many challenges to overcome.

Recognizing human gestures and facial expressions is not easy. Not easy for software algorithms, anyhow. It just so happens that humans are very, very good at reading gestures. Just take a gander at the Facial Action Coding System to get a feel for all the subtleties involved in discriminating various facial expressions. And then realize that you read people’s posture and faces effortlessly.

Know How You GOTTA SPEAK UP for Your Hard-of-Hearing Uncle?

Ever notice how adults play with children? Every gesture is wildly exaggerated. The same is often true with the voices and movements in skits, comedies, impressions, and parodies. Comic strips and animations are usually also comprised of caricatures of human speech, emoting, and gesturing.

So how do we help gesture recognition systems along? Enter playful interfaces. Need clearer facial and movement cues? Make the system that elicits, tracks, and responds to gestures a playful one. The users of that system will naturally enhance their responses as they have fun and play along.

Gestural interfaces are relatively immature — not unlike the children with whom adults naturally express their exaggerated play faces and movements. I suspect playful interfaces can help gesture recognition techniques grow up, go off to college, get jobs, and become upright members of society.

The Visual Lexicon of Comic Strips — Starring You

I wish I could say I first had the insight of how play and gestural interfaces could, uh, play nice together and then dreamed up an example. But it went the other way around. I had the idea for a fun project and then from it I came to a handful of ideas on the link between playful technologies and gestural interfaces.

How fun would it be to walk up to a video display that illustrated in real-time your feelings and thoughts in the visual language of a comic strip? Stay with me. I’m talking about displaying all manner of speech balloons and symbols right over your head and around you.

You’ve got a frown on your face and your head is drooping? There’s a storm cloud above you. Your eyes and mouth are wide open? An exclamation point appears hovering over your head. You flash your finger in the air with your face elongated and eyes wide? Yup. You get a light bulb. Speak and a speech bubble shows your words. Curse and the bubble is filled with @%&*!# (known as grawlixes). Eye something amazing and dotted lines trace your gaze. Swing your arms and lines trail their motion. Furrow your brow and squint your eyes to see drops of stress sweat arc away from your temples and forehead. Scream and your speech bubble becomes jaggy and pointed. You get the idea.

The visual lexicon of comics is really quite expansive. It’s quite surprising how cartoonists and comic artists have captured so much of human expression visually. It occurs to me, oddly enough, that there is a compelling link to explore between gestural interfaces and the visual language of comic strips. That said, the larger point here is that enticing users to actively play with a gestural interface could enhance the system’s ability to recognize and act upon users’ gestures — especially in these early days of the field.

UPDATE (July 7, 2011): An old post now with even more comic strippiness

A Touch of Playfulness Added to User Interfaces

The image that follows is actually of an advertisement for a plastic surgeon’s office.
But COME ON, PEOPLE. Even if you have no interest in cosmetic surgery, you know you want to push that button so much.

So this advertisement (unintentionally) makes a great case study about playful user interfaces. Here we have a single button — to call an elevator. You can’t get more pedestrian than a single button interface. And, yet, by supplementing the context of the button (making the user part of Michelangelo’s famous Sistine Chapel ceiling), the entire experience of using the elevator call interface is fundamentally altered.

Creating playful technology is, therefore, as much about the user’s experience of context and expectation as it is about creating game-like interactions or whiz-bang interfaces. Playful design is bigger than the technology itself. Here the technology involved is intimately linked to but also exists subservient to the larger designed experience.

It makes me giddy every time I see this image. I totally need to push that button.

Oh. One more thought: It’s also interesting to note how this playful experience is aimed squarely at adults. I wonder how many adults who enjoy pushing that elevator call button would allow themselves to recognize that they were, in fact, playing around. 

(via Null We Know)

The Digital Sandbox: Playful Information Interaction

Fascinating concept: digital sandboxes for interacting with a library’s primary sources.

Lisa Norberg discusses the importance of students working with primary sources to develop critical thinking skills in studying history. She cites the difficulties of connecting students with said sources — primarily because of their perceptions regarding access to the material.

I even suggested that we help faculty build digital sandboxes in the backyards of their course pages. These sandboxes would be filled with an array of engaging primary materials and tools that would enable students to explore, to discover, to play. My playful argument was based on a growing body of research that indicates that students need the opportunity to connect with primary sources on a cognitive and emotional level in order to assess their meaning and put them into historical context.

Norberg proposes the idea of a digital sandbox for playfully interacting with primary sources. In her comment she is speaking of a web interface and may not necessarily be suggesting a true sandbox-like interface (possibly using the word sandbox more generally).

No matter the case, she’s struck on something great here. It seems to me that this interaction model is an interesting one to flesh out for creatively exploring, assembling, and correlating information sources in the form of a physical sandbox. Playing in a sandbox and working with information share a number of interactions: sifting, digging, piling, doodling, etc.