Observation

The following diagrams illustrate the way, until recently, our television was setup. The television and speakers were connected to a Media Centre, with both Speakers markedly offset to the left of the Screen.

With this setup, when the television program is speech based the sound “appears” to come from the Screen. That is, there is a strong illusion that the words being spoken are coming directly from the Screen, and in fact the speaker’s mouth, and not from their physical source.

This is an automatic response, that is even if you ‘know’ the sound is coming from the Speakers you cannot, by effort of will, make the sound appear to come from anywhere other than the Screen.

The situation is much the same with music, when the sound being heard is synchronised with the moving image, the music “appears” to come from the Screen.

However if the sound source is changed, to something that does not correspond to the image, then the sound “appears” to come from the Speakers and not the Screen.

Discussion

Firstly on a trivial level it is difficult to avoid using the word “appears” when it comes to talking about the location of sound, but this does somehow also suggest the primacy of the visual. This is probably just a personal bias; in this context, “sounds as if it comes” probably does just as well as “appears to come”, even if a little more verbose.

The automatic nature of this phenomena however means that it is similar to optical illusions such as the Müller-Lyer illusion where knowing the answer does not overcome the effect. So it does not seems inappropriate to call the observed phenomena an audio illusion.

This seems to have some resemblance to the McGurk-MacDonald effect where seeing someone’s lip movements effects the sound that one hears, again this is an automatic response.

http://www.youtube.com/watch?v=G-lN8vWm3m0&feature=related

Questions

1. How far away and in what configuration can the Speakers be placed and the illusion continue to work?
2. Is the illusion learnt or instinctive? If learnt at what age?
3. What, if anything, does the illusion indicate about the internal representations of sound and moving images?
4. How do transition states work, for instance a transition from someone speaking to hearing background music that has no obvious source? In this respect does the illusion have a sort of after image?
5. What is the effect of increasingly poor synchronisation of speech and music on the illusion? cf Gillian Wearing’s 1997, 10–16 where adults lip synch the voices and act out the physical tics of seven children.
6. Many more.

Bibliography

McGurk H., MacDonald J. (1976). “Hearing lips and seeing voices.”. Nature 264 (5588): 746–8

Deutsch, D.(2003) Phantom Words, and Other Curiosities. La Jolla: Philomel Records

Wearing, G. (1997) 10-16 Whitechapel Gallery 28 March-17 June 2012

Chion, M. (1999) The Voice in Cinema. Trans. C. Gorbman. New York: Columbia University Press. First published in French in 1982.