Mind, Body, World- Foundations of Cognitive Science, 2013a

evaluation, interpretation, and description” (Leman, 2008, p. 4). This suggests that
cross-modal interactions may be critical determinants of musical experience.
Some research on musical cognition is beginning to explore this possibility. In
one study (Vines et al., 2006) subjects were presented with performances by two
clarinetists. Some subjects only heard, some subjects only saw, and some subjects
both heard and saw the performances. Compared to the first two groups of subjects,
those who both heard and saw the performances had very different experiences.
The visual information altered the experience of tension at different points, and
the movements of the performers provided additional information that affected the
experienced phrasing as well as expectations about emotional content. “The auditory
and visual channels mutually enhance one another to convey content, and . . .
an emergent quality exists when a musician is both seen and heard” (p. 108).
In a more recent study, Vines et al. (2011) used a similar methodology, but they
also manipulated the expressive style with which the stimulus (a solo clarinet piece
composed by Stravinsky) was performed. Subjects were presented with the piece
in restrained, standard, or exaggerated fashion. These manipulations of expressive
style only affected the subjects who could see the performance. Again, interactions
were evident when performances were both seen and heard. For instance, subjects
in this condition had significantly higher ratings of “happiness” in comparison to
other subjects.
The visual component of musical performance makes a unique contribution to the
communication of emotion from performer to audience. Seeing a musician can
augment, complement, and interact with the sound to modify the overall experience
of music. (Vines et al., 2011, p. 168)
Of course, the embodied approach to music makes much stronger claims than
that there are interactions between hearing and seeing; it views cognition not as a
medium for planning, but instead as a medium for acting. It is not surprising, then,
to discover that embodied musical cognition has studied the relationships between
music and actions, gestures, and motion in a variety of ways (Gritten & King, 2011).
One of the most prominent of these relationships involves the exploration
of new kinds of musical instruments, called digital musical instruments. A digital
musical instrument is a musical instrument that involves a computer and in
which the generation of sound is separate from the control interface that chooses
sound (Marshall et al. 2009). This distinction is important, because as Marshall et
al. (2009) pointed out, there are many available sensors that can register a human
agent’s movements, actions, or gestures. These include force sensitive resistors,
video cameras, accelerometers, potentiometers, and bend sensors, not to mention
buttons and microphones.
The availability of digital sensors permits movements, actions, and gestures
to be measured and used to control the sounds generated by a digital musical
304 Chapter 6