People trust eyes -- not ears -- when judging musicians [Video]

By Eryn Brown

Aug. 19, 2013 2:33 PM PT

[This post has been corrected. See note below for details.]

Over 20 years or so of competing in classical piano competitions, psychologist Chia-Jung Tsay often noticed that she was more likely to win when she was onstage and audiences could make a visual connection with her.

“I could go into a studio and produce a really clean recording, but it didn’t seem to resonate as much,” she said. “The type of process seemed to affect the outcome.”

So when it came time to write her doctoral thesis at Harvard, the researcher (and Juilliard-trained pianist, and former Miss Long Island) decided to test her experience out empirically, and made a very surprising discovery: listeners judge music on how it looks, not on how it sounds. (Scroll down to the end of this post to view a video of Tsay performing.)

In a study involving 1,164 participants, Tsay, now an assistant professor at University College, London, found that novice listeners and experienced musicians alike could more accurately tell the winners of past classical music competitions by watching silent performance video than they could by listening to performance audio. Study subjects also had an easier time identifying winners after watching silent video than when they viewed video including both images and sound.

“It was shocking -- it was only in the silent videos that were stripped of sound that participants were able to identify the winners,” said Tsay, who published her findings Monday in a report in the journal Proceedings of the National Academy of Sciences. “Choices based on audio alone didn’t align with the actual winners, because actual winners had been chosen by visual means.”

To figure it out, Tsay conducted seven experiments. First, she conducted a test to determine what information listeners thought would be most useful in determining competition winners. By and large, participants said they thought audio recordings would be most important for figuring out winners.

The second, third, fourth and fifth experiments were designed to see how well participants could surmise competition winners by listening to or viewing brief snippets of audio-only, silent video or video with sound. Experiments 2 and 3 tested novices’ ability to identify actual winners; experiments 4 and 5, experts’ ability. In both cases, subjects did the best job identifying the actual winners of the competitions after viewing the silent video.

The final two experiments attempted to tease out what key information viewers got from visual clues that helped them choose winners. In experiment 6, viewers were able to pick out winners even when viewing obscured, six-second silent videos where performers appeared only as outlines. Tsay said that experiment 7 demonstrated that seeing a performer seemed to help viewers differentiate passion, motivation and creativity -- all qualities perceived as hallmarks of great performance.

“It’s surprising that sound doesn’t convey this, in the field of music,” Tsay said. While a show like “The Voice,” where judges are not allowed to see contestants in the first round of competition, acknowledges the role of visual information in assessing popular music, classical music is usually regarded as “a genre where sound is more sacred,” she added.

Tsay said that she would like to continue studying how visual cues impact decisions, in judging music competitions and in other arenas, including politics and hiring.

Watch a clip of Tsay playing Franz Liszt’s Totentanz with the Bay Area Symphony Orchestra.