This thesis presents a multimodal soniﬁcation system that combines video with sound synthesis generated from motion capture data. Such a system allows for a fast and eﬃcient exploration of musicians’ ancillary gestural data, for which soniﬁcation complements conventional videos by stressing certain details which could escape one’s attention if not displayed using an appropriate representation. The main ob jective of this project is to provide a research tool designed for people that are not necessarily familiar with signal processing or computer sciences. This tool is capable of easily generating meaningful soniﬁcations thanks to dedicated mapping strategies.
On the one hand, the dimensionality reduction of data obtained from motion capture systems such as the Vicon is fundamental as it may exceed 350 signals describing gestures. For that reason, a Principal Component Analysis is used to ob jectively reduce the number of signals to a subset that conveys the most signiﬁcant gesture information in terms of signal variance. On the other hand, movement data presents high variability depending on the sub jects: additional control parameters for sound synthesis are oﬀered to restrain the soniﬁcation to the signiﬁcant gestures, easily perceivable visually in terms of speed and path distance. The following figure presents an example of control signal used to drive sound syntehsis parameters (left/right knee angles) and their related principal components.
Then, signal conditioning techniques are proposed to adapt the control signals to sound synthesis parameter requirements or to allow for emphasizing certain gesture characteristics that one ﬁnds important. All those data treatments are performed in realtime within one unique environment, minimizing data manipulation and facilitating eﬃcient soniﬁcation designs. Realtime process also allows for an instantaneous system reset to parameter changes and process selection so that the user can easily and interactively manipulate data, design and adjust soniﬁcations strategies.
Source code and demonstration videos are kept on a public GitHub account here.