loading1
loading2

 

Summary

Overview
Methodology
Approach
Results
Preliminary Conclusions
Implications
Observations
Applications
Tools
Related (external) references

 

Overview

Welcome to the Clarinet Gestural Analysis Website! Here we present an overview of work being done investigating
expressive movements of musicians at the Music Technology
Area, Schulich School of Music, McGill University.

We are interested in analyzing the correlation between physical and musical gestures. We specifically
focus on expressive movements (also called ancillary or non-obvious gestures), movements that do
not have a direct link to the generation of sound but are an integral part of the performance.

Expected results, in addition to a better understanding of music performance and the role of expressive movements,
include the design of new computer-based musical instruments as well as hardware interfaces that take expressive
movements into account.

Please check the other sections of this project for an overview. Some of the results of this research have been
published in various papers — please check the publication list.

 

Methodology

A number of clarinetists were recorded using digital video cameras and a three-IR-camera Optotrak 3020 system, from NDI.
Analysis of the data was then made using EyesWeb
(on raw video data), and with MATLAB and Labview (utilizing data captured with the Optotrak). A
sample of what a recorded video looks like may be viewed here in Quicktime format.

The Optotrak system itself enables the precise tracking of predefined points on a performer’s body. An example of a
performer utilizing this system is given in the image below (Bradley Vines, at the Motor Control
Laboratory, Psychology Department, McGill University). Note the silver points on the performer’s body, as well
as on the saxophone itself. These are tracked by three infra-red cameras to give an extremely accurate representation of
each point’s 3-dimensional location over time. This data is then used in determining correlations between different
performances. To see a MATLAB plot of how the points evolve over time, view the
short clip here. It is interesting to note in this case that, even with such a few number of points, one can
make out the performer’s movements.




Fig.1 Brad Vines at the Motor Control Laboratory.
Fig.2 Pierre Dutrieu at the Vrije Universiteit Amsterdam. Points are tracked in three dimensions.

 

Approach

The methodology taken in this research is to find other ways of representing the data. Both video and Optotrak data have
been recorded, and we find each data set useful for different reasons. For instance, video data gives a less-precise
(gesturally speaking) yet more holistic overview of what is transpiring. Thus, it is useful to form generalizations
about performer movements and to rapidly parse a performance. The Optotrak data, on the other hand, is extremely verbose
and would not be suitable to this end. It is, though, useful in comparing the minute nuances between repeated gestures
and in quantifying the precise amount of variation.

Below are images captured from the video data. While it is not possible to make these kinds of comparisons as from the
Optotrak data, it is possible to study the variety of motions and postures which are used during a performance. Three
selections are given which illustrate this.


Fig.3 Three shots showing different postures during an (expressive) performance. Lars Wouters at the Vrije
Universiteit (Amsterdam).

Initial attention has been given to the visualization of the data. EyesWeb and MATLAB have been used to create
alternative representations of the data for comparison. This approach has proven instrumental in forming connections
between observed movements and the actual extracted data. EyesWeb has also been used to extract just the performer from
the frame; an example clip may be viewed here.

Formalizing the data in the ways described herein has helped to facilitate observations and has led to a number of
interesting discoveries. We hope to show a number of correlations using the techniques and data, as described on this
page.

 

Results

It will come as no surprise that most instrumentalists exhibit a degree of movement during a performance. Through
careful analysis of these performances, we are able to form a number of general conclusions.

Firstly, all players who were kind enough to allow themselves to be videotaped displayed a number of ancillary
gestures, even though these types of gestures are not necessary to the production of sound. Furthermore, although
unnatural at first, all clarinetists were able to suppress most of their expressive movements. This shows that this
ability can be learned, and raises questions as to which ancillary gestures have been learned and why.

 


Fig.1 A plot of vertical bell movement over time for three performances of Domaines, by P. Boulez. Pierre
Dutrieu at the Vrije Universiteit Amsterdam. From top to bottom: Standard performance, exaggerated
(expressiveness) and immobilized. Note the suppression of movements for the third performance
(immobilized).

 


Fig.2 Similar picture to the one on the left, this time showing the marker on the clarinet mouthpiece and with
the third performance zoomed-in (immobilized). Check that many movements are still present,
although their amplitudes are very small.

 

Next, we find that with expert players there exists a strong correlation between their movements at the same
points of the score during different performances. This suggests that gestures are by no means randomly
produced; rather, they bear a relationship to what is being played and how. For example, a plot of the Optotrak data
showing the vertical movement of the clarinet bell is shown on the left, below. On the right is a comparison of two
different performances from the same player. Note the similarities in both time and spatial movement over the course of
the performance.

Fig.3 A plot of vertical bell movement over time


Fig.4 A comparison of two bell-movement plots from the same performer – Excerpt of the First
Clarinet Sonata, J. Brahms, First Movement.

 

Different performers will show overall different patterns, although some of the movements are similar. The next figure
shows two performers playing the excerpt of the Brahms sonata (performer #1: red, performer #2: black).

Fig.5 Several performances of Brahms first clarinet sonata (excerpt), two different players (red, and
black).

Also, the types of common movements can be isolated. One example is the circular movements of the bell in sustained
notes.

Fig.6 Circular movements of the bell, Brahms first clarinet sonata.

Fig.7 Circular movements of the bell, another player.

 

Preliminary Conclusions

Some of the current conclusions of this study are:

Further research will determine whether certain ancillary gestures are common to a certain subset of the performer (for
instance, students of a particular teacher), or if there exists a relationship to certain musical gestures.

 

Implications

(Text by Wes Hatch)

Although it is still too early to draw any conclusions, there were a number of significant observations made that will
be discussed in brief.

Consistency of Gestures

Data from the lastest set of trials confirms earlier results that indicated a clarinetists’ ancillary gestures are
consistent across his or her performance. Additionally, it seems that ancillary gestures are an integral part of musical
performance.

Gesture and Sense of Time

One interesting observation was that there were noticeable differences in the length of a performance when a performer
was asked to play immobilized as when compared to his or her normal playing style. For example, for the majority of
players, the immobilized performances were faster (both in total duration and individual phrase lengths) than their
standard and expressive counterparts. This held true for all but one player, whose immobilized performances were
consistently slower.

When comparing normal performances with those of exaggerated emotion, no consistent differences in duration could be
found (for all performers). This seems to indicate that the presence of gesture (or absence thereof) has an impact on a
performer’s sense of time (for example, the tapping of one’s foot to keep rhythm) and is one area that additional trials
could further quantify.

Gesture and Note-Groupings

A relationship between a performer’s ancillary gestures and note-groupings is indicated by this research. Performers
exhibited a tendency to “group” by equal-duration note clusters, or by phrasing. In the first case, this means that
performers’ movements were “characterized by regular and consistent movements that coincided with rhythmic structures
suggested by the score.” The other dominant trend was to group according to the phrasing. In this instance, players were
generally quite still until the end of the phrase, at which time a flourish or “finishing gesture” was made. This served
to punctuate the end of the phrase.

Grouping Performers based on Dominant Gestures

The dominant body regions that a performer tended to move while playing served well as another basis for sorting players
into groups. For example, certain groups of performers displayed a strong tendency to express ancillary gestures in
their knees, while we found that for other groups, waist-bending gestures dominated.

Awareness and Perceptual Impact

A questionnaire submitted to several of the performers indicated that, while they were aware of their expressive
movements in general, they were not aware of the intricacies of them. The performers also mention that motion must seem
natural and it should not be intentional.

Furthermore, it became apparent that it was unnatural and/or very difficult for players to completely suppress their
learned movements (as was asked of them for the immobilized performances). Even during immobilized performances, each
player made many of the same gestures, but on a smaller, reduced scale. One hypothesis is that these movements are so
ingrained in a clarinetist’s mental representation of a piece that it becomes difficult to suppress them.

Perceptual investigations into the influence of the clarinetists’ movement on the experience of an observer have
revealed the importance of visual information in musical performance. The gesturing of clarinetists was found to either
augment or dampen the magnitude of tension experienced at important points in the performances, to convey phrasing
structure throughout the music, and to cue phrase boundaries by extending beyond the sound and by beginning in
anticipation of a new phrase.

These findings point to a complex interrelationship between musicians’ movements, musical sound, and the observing
audience, where movements further convey the performer’s musical intentions to observers. Future research is needed to
elucidate the influence of different performance styles and quantities of movement on timing perception for the
performer.

 

Observations

(Text by Louise Campbell)

While the study of gesture and music is a relatively new field, the observation of movement has a long and varied
history. Dancers, who are by definition specialists in movement, have developed and continue to develop ways of
observing and recording movement for reasons such as artistic expression and exploration, health, education, and
preservation of complex choreographies for future dancers. The means of observing movement translates to fields other
than dance because most of what we accomplish each day is through some form of movement, whether it is to dance
choreography, play a musical instrument, wash dishes, or type on a computer. Even though each of these activities is
different, the means of accomplishing it is the same: the use of the human body. Because of this, any movement of the
human body can be examined using a system that has evolved for observing, recording, and developing movement.

Laban-Bartenieff Movement Fundamentals (LBF) describes movement on many levels in order to come close to a rich
and accurate description of not only where the movement takes place in the body and in space, but how the movement
occurs. In order to do this there are three key assumptions:

  1. the whole body is connected,
  2. all parts are in relation to one and other,
  3. a change in one part affects the whole.

All three of these assumptions are linked through the idea that any movement in one part of the body will affect the
whole body.

LBF describes movement through the four inter-related categories of Space, Effort, Shape, and Body. Within these
categories, there are various attributes. By observing one movement using the different categories and attributes, we
can begin to observe the relationship between parts of the body and the use of the body. The combination of these
different observations gives a means of finding a complete description of a single movement. The method of observation
presented here is not the complete system as this is beyond the scope of this project and full training in LBF
observation requires dedication and time to complete. However, a basic understanding of fundamental concepts can be
learned and used to facilitate observation by people without formal movement training. For the purposes of
simplification for non-trained observers, specific attributes were chosen by certified LBF practitioner Valerie Dean in
order to address the most essential aspects of observation as it applies to musicians. The goal is to provide a starting
point for observing movement for observers with no formal training in movement.

 

Applications

Interestingly, it has also been found that a performer’s movements can subtly shape the sound production. For instance,
consider the following series of pictures:


J.-G. Boisvert playing Praescio IV, by Bruce Pennycook at McGill University in 1999.

In the case of the clarinet, bell-movement is the cause of amplitude modulations that are perceived as a subtle flanging
effect. In fact, for a microphone placed in front of the clarinet, the interaction between the direct sound coming from
the instrument and the first (floor) reflection can be modeled as two delay lines.

Model for the influence of ancillary gestures in the resulting instrument sound (recorded by a microphone).
Mathematical representation.

 

We have performed room response measurements in order to obtain the values for the model above. The following
pictures show one of the measurements made at IRCAM, in Paris, France (Gerard Bertrand is the technician).

Various clarinet angles used in the measurement.

Gerard Bertrand and the device used for the room measurements at IRCAM.

The resulting delay difference is dependent on the position of the bell with respect to a
microphone or listener. Depending on the position of the instrument and of the microphone, delay differences
between 2 and 5 ms can be obtained dependent on the clarinet angle.

Results of measurements (gains and delays).

Therefore, a sort of gesturally-controlled flanger can be obtained, the delay being a function of the
clarinet angle.

An example of application was produced by applying the effect of performer movements onto a clarinet sound
recorded in an anechoic chamber. This sound file presents a (very)
simple simulation with the anechoic sound plus several repetitions of it where simulations of one-dimensional
movements were created (using a slider) in Max/MSP. The simulations run from simple to more complex movements.

This part of the research is performed in collaboration with Prof. Philippe Depalle.

 

Tools

As described in the methodology, several tools and techniques were employed. Gestures were analyzed using both
holistic and accurate methods. Video data provided more generalized information about a performer’s movements,
while motion tracking using the Optotrak or the Vicon systems coupled with MATLAB analysis routines provided
accurate information about performer movements.

Video Analysis

Video analysis was used for a global, holistic examination of the performances. We used several tools for this
purpose, including Final Cut, EyesWeb,
Anvil, and the Musical
Gestures Tools.

EyesWeb (developed at the DIST laboratory at the
University of Genova, Italy) was an important tool used to quantify elements not easily determined using
the preceding tools. Specifically, the “amount of movement” that a player exhibits was determined by using
EyesWeb, and then measured by drawing comparisons across successive
performances.

Anvil was used to annotate performer movements using two different sets of parameters: a) descriptive
parameters (clarinet movements, postural adjustments, etc.) and b) Laban-Bartenieff movement principles (see the
section on observation for more information on LBM principles).

Anvil (created by Michael Kipp) uses a movie file, allowing the user to
have a holistic picture of the event to be annotated. The user declares all the gesture types he/she wishes to
annotate in a .xml file (for example see oswald_project.xml ).
Once this is done he/she can then indicate the time a particular gesture occurs while viewing the movie file.

 

This screenshot shows the types of gestures one wishes to capture on the left. It is immediately obvious that
the circular movement under ‘clarinet gestures’ takes a longer time than an ‘up’ or ‘down’ movement, for
example. Also one can spot trends, for example, the up-down movement of the clarinet is very correlated with the
up-down movement of the head and to a lesser extent the movement of the back (under ‘upper body’).

In the screenshot, it was decided to annotate variables–without any prior reading of gesture classifications
such as Laban-Bartenieff techniques–which were very obvious in the video. The variables were grouped in
‘clarinet gestures’, ‘head movement’, ‘upper body’, and ‘lower body’. Also in hindsight, there isn’t really any
difference between ‘body
sweep’ and ‘bodyweight emphasis’.

The anvil files used are available on demand. Anvil text by Oswald Quek.

The Musical Gestures Tools is a collection
of software tools for helping in qualitative and quantitative analysis of movement
and sound. The toolbox is built with Max/MSP/Jitter from Cycling ’74 and is available for Mac OS X and
Windows.

MATLAB and Optotrak

 

Related (external) references

Part of this work has been or is performed in collaboration with the following groups: