Graphical Interfaces for Mapping DMIs

John Sullivan

Description:

During a variety of projects, we have found that the “mapping” task – in which correspondences are designed between sensor/gesture signals and the control parameters of sound synthesizers – is by far the most challenging aspect of digital musical instrument design, especially when attempted in collaborative settings. We have developed tools for supporting this task, including the Digital Orchestra Toolbox for MaxMSP and the software library libmapper. The latter project enables the creation of a network of distributed “devices” which may be sources of real-time control data (instruments) and/or destinations for control data (e.g. sound synthesizers). The software library handles device discovery, stream translation (e.g. type coercion, padding), and network transportation, but does not attempt to create mappings automatically. Instead, the mapping designer(s) command the library to create connections between signals, usually using a graphical user interface to interact with the mapping network. To date, GUIs for libmapper have been implemented in MaxMSP, Javascript/HTML5, and Python/wxWidgets.

The primary view used in our mapping GUIs is based on the common structure of diagrams used to describe DMI mapping in the literature – a bipartite graph representation of the connections, in which sources of data appear on the left-hand side of the visualization and destinations or sinks for data appear on the right. Lines representing inter-device links and inter-signal connections may be drawn between the entities on each side, and properties are set by first selecting the connection(s) to work on and then setting properties in a separate “edit bar”. They use a multi-tab interface in which the leftmost tab always displays the network overview (links between devices) and subsequent tabs provide sub-graph representations of the connections belonging to a specific linked device. We have also explored alternative visualization and interaction techniques, which allow more informed and flexible interaction with the mapping network. Crucially, we believe that there is no need for a single “correct” user interface; rather, different network representations and interaction approaches may be useful to different users, for different mapping tasks, or at different times. All libmapper GUIs function as “dumb terminals”—no handling of mapping connection commands takes place in the GUI, but rather they are only responsible for representing the current state of the network links and connections, and issuing commands on behalf of the user. This means that an arbitrary number of GUIs can be open simultaneously supporting both remote network management and collaborative creation and editing during the mapping task. This approach has brought our attention to interesting research into other collaboratively-edited systems [Ressel and Gunzenhäuser 1999]; our approach to undo/redo functionality is based on this work.

Alternative Visualization Techniques

Although the spreadsheet-like interface shown above has been used successfully in a number of individual and collaborative projects, there are scenarios in which alternative layouts and interaction techniques may be useful. Since signal names are used to distinguish mapping sources and destinations, this approach lends itself well to mapping scenarios that involve small numbers of devices with heterogeneous parameters-spaces; a scenario that requires mapping of a large number of identical devices (e.g. wireless motes) may confuse the user since the GUI display would be full of entries like this:

/mote.1/acceleration
/mote.2/acceleration
/mote.3/acceleration
etc.

The user would likely be better served by a GUI that displayed devices based on their physical location, or the performer’s name if they are worn or hand-held, or perhaps their activity over some specified period of time. To this end, we began exploring alternatives for the network visualization, with the philosophy that no one approach would be appropriate for all users or at all times. Instead, the GUI should allow the user to choose from different views of the mapping network, and even customize and create new views. The data driving the GUI view is supplied by libmapper, which already supports tagging of all entities (devices, signals, connections) with extensible metadata. Prototype versions of the GUI have enabled users to both sort entities using tagged metadata, and to apply new tags.

GUI functionality:

browsing active devices and their signals
drag-and-drop mapping connections
mode and function editor for connections
collaborative undo/redo
filtering parameter lists by OSC prefix or string-matching
saving and loading mapping sets (including consideration of mapping transportability per the GDIF project)
multiple “views” of the mapping network, using different visualization techniques

IDMIL Participants:

Research Areas:

Mapping

Funding:

NSERC

Publications:

Malloch, J., Sinclair, S., Wanderley, M. M. (2015). Distributed Tools for Interactive Design of Heterogeneous Signal Networks. In Multimedia Tools and Applications (pp. 5683-5707).
Malloch, J. (2013). A Framework and Tools for Mapping of Digital Musical Instruments. In Ph.D. thesis, Music Technology. Montreal, Canada.
Krajeski, A. (2013). A Flexible Tool for the Visualization of Musical Mapping Networks. In M.A. thesis, McGill University. Montreal, Canada.
Wilansky, J. (2013). A Production Tool for Analysis of the Mapping Layer in Digital Musical Instruments. In M.A. thesis, McGill University.
Rudraraju, V. (2011). A Tool for Configuring Mappings for Musical Systems using Wireless Sensor Networks. In M.A. thesis, McGill University. Montreal, Canada.
Malloch, J., Sinclair, S., Wanderley, M. M. (2008). A network-based framework for collaborative development and performance of digital musical instruments. In CMMR 2007, - Proceedings of Computer Music Modeling and Retrieval 2007 Conference, LNCS 4969 (pp. 401–425). Kronland-Martinet, R., Ystad, S., Jensen, K. (Eds.). Springer Verlag.Berlin, Germany.
Malloch, J., Sinclair, S., Wanderley, M. M. (2007). From controller to sound: Tools for collaborative development of digital musical instruments. In Proceedings of the 2007 International Computer Music Conference (ICMC2007) (pp. 65–72). Copenhagen, Denmark.
Malloch, J. (2007). A Consort of Gestural Musical Controllers: Design, Construction, and Performance. In M.A. thesis, McGill University. Montreal, Canada.