Finding the deep experience in technology

The best digital technologies don’t have to offer bombastic multimedia sensations. They can be much more powerful and elegant, according to technologist and pianist Xiao Xiao, a Ph.D. candidate in the MIT Media Lab who studied computer science and architecture as an undergrad also at MIT.

It is time, Xiao Xiao says, to design in a way that lets the user fill in the sensory blanks. The trick is to find the “sweet spot” by giving people just enough informational cues while allowing them to inhabit and participate in a communicative space in their own way.

“Designing technology in this way is closer to being a master at Chinese brush painting or playing a fugue than just being a technologist,” she says.

Born in China, Xiao Xiao moved at the age of eight to New Orleans, where she said the aesthetic of the city made a mark on her. Musicians improvising on Frenchman’s Street exposed her to the visceral quality of live performance. As a technologist, she tries to capture the deep experience of listening to music wafting through Frenchman’s Street, the way it emanates from the instruments and bodies of musicians and reverberates in the fibers of the listener’s body.

Live musical performance happens in culturally-constructed spaces, a reality that is always integral to Xiao Xiao’s design efforts. When she goes to an art gallery, she practices “indwelling,” projecting herself into the art, imagining herself at different scales, wandering around in the terrain of the work. The exercise helps her to think about the way we encounter information from different perspectives, “turning it around in our heads, but also turning ourselves around inside it.”

Her office chair at MIT is a piano bench; during the day she sits on it facing her desk and computer, at night she turns to play on a Yamaha grand piano. Every day, she combines technology with art, carrying insights from one world to the other. “Both the piano and the computer have keyboards that enable you to externalize your ideas using the dexterity of your fingers,” she explains. 

Xiao Xiao’s current project is MirrorFugue, a technology that projects onto a player piano the image of a person playing the keys of a musical composition. It makes the abstract entity of music into a more visceral experience by showing the body of the person playing the piece. Music, she says, is physical as well as intellectual. Before it became possible to broadcast or record music, disembodying it from performance, it was always tied inextricably to the body.  

It is important to see the person playing the music. “There is something about the human body that people just respond to,” Xiao Xiao says.  “Being human, we like looking at ourselves, even data about ourselves.”

She points out some of the whimsical and enriching possibilities of MirrorFugue. For instance, one performer can sit down to the piano and play while another is projected simultaneously. The player piano itself strikes the notes of the virtual performer. In this way, one could play a duet with a remote individual—even with one’s earlier self, or with a deceased grandparent.

The purpose of this technological exploration, according to Xiao Xiao, is to find ways to deepen our connection to the physical world. "For billions and billions of years, people have interacted with the world through our bodies,” she says. “We pick things up, climb trees, smell flowers, and feel the wind in our hair or the rain falling down around us. But these days, because of convenience, we’ve given up a lot of this rich experience in favor of viewing it through a computer screen, tapping a screen with just a couple of fingers.”

We’ve developed a pathological preoccupation with the visual and nothing else, she says. We are losing the affective aspects of interaction—body language, tone of voice—which predominate face to face communication.

Computer science abstracts the world into something that computers can easily store and communicate, but musical performance still has the power to completely envelope us. As Xiao Xiao turns back and forth between the computer and piano keyboards in her office, she searches for ways to make the dry conveyance of information as rich as a musical performance. 

“The most profound things in life are derived from deep experience with the world,” she says.