AUMI interviews: Ivan Franco

Interviewee: Ivan Franco (IF) is an electronic musician, digital artist and music technology researcher. He served as the main AUMI desktop app developer from 2015 – 2016 at McGill University while pursuing his PhD in music technology. During his time with the AUMI project he authored the current version 4 of the desktop application software.

Interviewer: John Sullivan (JS) is a music technology researcher and PhD candidate at McGill University. He served as the developer for the AUMI desktop application from 2016 – 2019.

The interview was conducted in person at the Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT) at McGill University on September 12th, 2019.

JS: Okay, so this is interview #2 for the AUMI book chapter. It is Thursday, Sept 12th, here at CIRMMT. Today I’m here with Ivan Franco and Ivan, I’ll let you introduce yourself and your involvement with the project.

I’ll start out by asking this first question. What is or was your involvement with the AUMI project, and if you could, describe a little your involvement with the IDMIL (the Input Devices and Music Interaction Laboratory at McGill University) and how it came about.

IF: I first heard about the AUMI project through Ian (Hattwick, previous AUMI developer and co-researcher at IDMIL). Ian told me about the project, the general goals of the project, that they essentially the project needed a developer that would work, a little bit, on the continual development of the AUMI. That’s when I got in touch with Eric Lewis, well, this is all happening while I was doing my PhD at the IDMIL and, essentially they told me that they needed somebody to support development but they were open to new ideas and anything that I could bring to the project, and in that way it was very open and to me this was very good because I was able to kind of look at what was AUMI at that point and suggest modifications, and also in a way contribute a lot to the vision of what AUMI is. So my involvement was essentially a full redesign of the AUMI application that we did for version 4. And so I started by essentially analyzing what I thought were the weaknesses of AUMI, I tried to understand how it could be – what kind of improvements we could bring to the application. And so this led eventually to a full redesign of the AUMI and the release of version 4.

JS: So I should show you, Sherrie (Tucker, AUMI project coordinator, University of Kansas) sent me a recent interview she did with Leaf Miller, who was one of the original instigators of the project with Pauline and amazingly, Leaf, her old laptop has 6 or 7 versions of AUMI, going back. And up until the version that you designed, they’re all pretty much the same. Each each version had improvements, changes, and stuff, but it all looked the same, and all the way through, until your version when you pretty much redid it from the ground up. Even though functionally it still works very similarly, it was kind of a big departure.

IF: Right. I would say that there were lots of challenges that I saw, things that could be improved. But biggest challenge was, how do you get users to adapt to a new application. There are expectations regarding what the application did before, there’s new interfaces on the new version, there are new capabilities or functionalities. Of course, for long-time users this is always a challenge and something that happens in almost any area with any software, right, when there’s big designs you have, users that use the previous version that are a bit more conservative and so it’s difficult to use the new version. So I think that the goal is really to make people understand where we are going with the new version, why we are doing the modifications we are doing, and essentially to train them also. You have to evangelize a little bit, especially here where there’s a big team involved with AUMI. So I think that could lead us to the second question, maybe, which is what I thought was important to change in the way AUMI was designed?

JS: Right, you said in your introduction that the first thing you did was analysis of the existing version in things, so I’m interested to see.

IF: Right, so I would say that the initial version had the concept of what AUMI was supposed to be, it was there, which was essentially how do we give children with physical disabilities a way to play music, a way to engage in music. And of course, it was designed and developed on top of Max (a visual programming language for music and multimedia), right? And one of the big problems we see from Max, all the time, especially when you have a team that is always changing because new people come in and others leave from the developer team, is always, how do you manage the complexity of something like a Max patch when you’re diving into it. And so, I took a look at the functionality, the core functionality of the application, and I thought that, there was space to create a version that would be more flexible and more open to improvements, so the idea [was that] that we could continuously improve AUMI and have a code base that’s still done in Max but strives to become a more modular approach to what AUMI is.

And so in a way, AUMI had a couple of functionalities. Basically it was able to play a piano, it had one one way of of tracking the movement of the user using the cameras, basically doing computer vision, on the camera, to extract the movement and the gestures of the user. But all of this was very fixed, there was no way, for example, if someone was to use other types of gestures, or other types of sounds, other types of senses, it was not easy for a developer, for example, to create a new module that would do certain sounds or a new module for acquisition that would use other gestures into play. And for my experience with AUMI, especially in the deployment at the schools, each child has their own requirements. Each child has their way of seeing the world, the things, the type of ways to engage with music both from a gestural level and what the musical response is.

So my idea was, okay, we should try to create a standard basic structure that would allow developers to continue to create both modules to new gestural input, and modules for sound output. And so to do this, in Max, is a bit of a challenge. But I think we’ve somewhat accomplished that. Now, with version 4 of AUMI we’ve introduced several different ways, of different gestures that children could use. And we also introduced samplers, we introduced other types of sound modules and sounds that they could create. And my vision for AUMI was that, so we can create more of these modules so that could sit well and be easy to integrate application so that it could continue to grow. So I think this was the main challenge for version 4. And the future developers that will work on top of it, I hope that they understand and the vision of, of how the main version 4 can be the basis for the growth of AUMI. So this essentially, was my goal with the redesign of the application.

JS: Yeah, so I can say because I (obviously) took it over from you, and in working on it, and going back from previous versions, it has become a lot more clear, developing for it. Because you can develop several modules. The way to develop it is a lot more clear cut. Something just in my own personal work on AUMI, and just long term project management in general is how do you pass it on. Several pairs of hands have worked on AUMI over several the years and especially, I think, a bit more specific to using Max versus another language, but really in any sort of language you might have the same sort of thing. Everybody has their certain own way of programming Max and the way that they choose to lay things out, so once you have 6 different people working on it for 6 years, it gets to be… For instance, Ian I notice, I was just digging through his patches recently and he was using lots of JavaScript embedded in Max patches to do a lot of things, and so that’s like one approach, and you get this sort of patchwork thing. So, absolutely with version 4 that you built and I took over it’s much more codified, and it makes it a lot easier to work around and expandable, which is really nice.

IF: Yes that was my main goal, and then of course another thing I tried to observe and try to improve on, is taking from my experience in terms of what what the user experience is. For example, the ability to have the presets, which is something that is once again very important for AUMI to adapt to the children that are working with AUMI. You have special settings. And so to be able to work with this more proficiently, we had to develop a preset system that would work with AUMI, and together with that, little usability issues of previous version that we tried to solve. And we tried to do it somewhat, in an easy and relaxed way of doing user centered design. BEcause I was able to go to the teams several times and present them with new ideas and developments and then come back and try to implement those in a way that was suited to most of the team, and that was a reflection of their own ambitions for the application. So I remember we had a big discussion on the color palettes we could use.

JS: (laughter) I remember this discussion.

IF: Seems like a very small thing but it’s actually very important in terms of usability. You know?

JS: Right.

IF: So those little things, I think that, at that point, with AUMI there was no discussion on those little things, that make or break a great application. So I also tried to a lot incorporate a lot of those things I learned by working more closely with the team, in terms of development. So instead of just bringing them something and saying hey, this is how it works, try to understand how they wanted it to work, and try to incorporate it with my own experience in user interaction and so on. So I think that was also a very important and positive step, because maybe people feel more engaged with an application if they see their own input and their own feedback reflected on the application.

JS: Yeah, just thinking about what you started talking about, the need for, what did you say, to evangelize the practitioners to use it and to learn a new system, and it’s something that is still a challenge. One thing that I think we need is, and maybe it’s a lot just around what funding is available for this project and things like that, but through a lot of version three there was a lot of support just writing documentation and doing videos, and user guide videos, and stuff like that. One of the big projects, when I took it over from you, in preparing it for its final full release, was writing a user guide for it, so that was a good start. But it would be really great to do a bunch of videos. And there was a lot of feedback too, because before there was a good system, really more on the academic research side, where practitioners were actively, I know there was some sort of spreadsheet feedback form, that they were actively replying to the team, that is not so much done anymore, that I think was really helpful to guide to this process. And understand who was using it. Because at this point, I’m not sure who was using it.

IF: Right, and also think one thing that I never go to do, which would be an important work moving forward, is to create also documentation for the developers, right? So that the next person wouldn’t be totally lost on how to create a module for the system. So the same thing doesn’t happen where he or she doesn’t have to go dig down into what the engine is to be able to do some sort of modification. I think that the ground work, the technological groundwork is there, but if you give people proper documentation on how to develop it. maybe it could be easier to give people models to create, to continue to grow in this sort of modular approach to AUMI.

JS: Good. Let’s see. Well, let’s take a swing at the third one. So what would have been some of the most effective or meaningful aspects of AUMI in your involvement with it?

IF: well, I think the most meaningful thing for me personally always is to actually see the deployment. The visits we do to the Mackay Centre School, and seeing the children engaging with the application. I think that’s really the reward at the end of the day that you get from being involved in this sort of project. So it was fun to see, for example, some kids want to play with notes, or sound textures, or others that really wanted to be able to create rock music, for example, with something like AUMI. And in a way the, the new version responds better to those requirements, right? It helps, I believe it’s better to help kids engage and to make it a little bit personal for themselves, right? And so, I think this was the most rewarding thing for me. The second probably would be the whole engagement of the AUMI community. The other people in AUMI seem very passionate about it, which is great, and you know, working in a team like that is really rewarding. It’s something that I was very happy with, especially when we did the AUMI meetings here in Montréal, and put a face to everyone that was involved in the project. Some people you never imagined where there, you know, and so in the end it feels a little bit like a work of passion which is something that I thoroughly enjoyed when I was working with AUMI. Then the second, well, maybe it’s also good to see, it’s a long term project. It’s been like what? 10 years maybe. (laughter) for AUMI.

JS: 12 Years. Officially the start was 2007.

IF: It’s very good. And it seems to have enough strength to continue, which is very good. Even if it takes a completely new form, I think that the goal is, it’s a very valid goal, right? How do we bring music to these kids, and how do we help them to engage with music right? So there’s a lot of other ideas that we have. And I think there’s still space to develop them. Like I said, there’s a lot of good ideas we’ve talked about. So AUMI is essentially based on computer vision. But we talked a lot about, yeah, there are kids that do have some ability so how do we engage them physically too. So we get to the idea of how do we create physical interfaces for these kids. Right? So we’re looking at, how do we, for example, adapt standard switches that some of them have on their wheelchairs to be able to interact with AUMI. That we would go, from just from something that you would gesture in the air, to something that is much more physical that would maybe be more rewarding for some of the kids . So a lot of those ideas, when I left the project, they were still in the air, but it seemed that, there were good ideas and everybody was like kinda, uhm, excited, to go through these new avenues, and so this constant look for innovation, for new things we can do with kids, to observe them, to be able to come back with, with things that they can use, I think that’s fantastic. You know, that kind of engagement, that kind of community, sense of community, is probably sometimes one of the hardest things to really build, and here it seems that it just came organically, so it was like, everybody was very passionate about doing this. So that for me was very refreshing and good.

JS: Yeah, as far as it being a long term project, we had a nice moment when Sherrie and I went to the OHMI conference (a conference on music and disabilities in Birmingham, UK organized by the One Handed Musical Instrument Trust). We presented the AUMI project. And it was really funny. There is so much interest in it now, and there are a lot of startups and new things, and we got to be up there and say, we’ve had this project going for 11 years now. It was definitely on one of the longest standing projects that addressed this. And you know, it continues with the team now. In fact, so, Pauline passed away two years ago now? A bit over a year? And there was a period of time when things kinda got quiet, and there was sorta – not question of whether the project was going to end or anything but just, you know, not sure what direction it was going to take, and then 6 months after her passing there was just this really strong force – alright let’s do this, let’s really push this. So I think it’s been kind of more active in the last year than since I started working on it, which is really fun to see and there’s a lot of energy and enthusiasm for it.

IF: I think a good goal would be to try to find better funding for these projects, right? Because I feel that they’re a necessity, they’re very justified projects, so they should have proper support to continue and I think that there’s still some work to do there because we’ve already accomplished so much with so little, and maybe with a little bit more resources we would even go further. For example I’m even thinking about not only AUMI, or not only musical interaction, but for example, there was a discussion that came up when we were visiting the Mackay Centre School which was just the price of of a lot of these interaction systems for kids with disabilities. They were very expensive systems, and so a lot of kids don’t have access to those kind of tools. They might have access at school but not at home or whatever. I remember, something like the system that some of them used to talk, which was basically kind of an iPad – it wasn’t an iPad, it was a certified solution but it was an embedded solution with a screen they used to write something they want and to say yes or no, these kind of things. These are all systems that are patented or covered by certifications that are really expensive, and this and that.

JS: Right.

IF: I believe, well, it’s good to have some sort of control for what kind of products are put out there. But I also think, immediately, that if kids don’t have anything, if they don’t have access to all of that, wouldn’t it be great if we could build an open source version that would be free, that anyone could put on a standard device like an iPad and and maybe have at least some of that, for those kids that can’t afford the the expensive systems, right? So I felt there was a lot of space for innovation, especially if it’s a project that is publicly funded, for example, to create more interactive systems for these kids, not not necessarily things that have to do with music, or even different types of musical application, but also, just those basic systems that felt like they were out of reach to lots of these kids.

JS: Sure.

IF: So I feel that something that starts with AUMI can go in a lot of different new avenues, there are a lot of different things we could do for these kids.

JS: I’m sure you and I have talked about it too but a good example of that is eye tracking systems. Which are one of those. They’re all proprietary medical devices that are several thousand dollars. And I think it was one of the earlier developers that developed simple eye tracking system using the jit.cv library (a Max library for computer vision) that worked okay. It wasn’t great, but it worked fine and it was basically free. And so if that can get you like 75% of the functionality of a system that costs $3,000 dollars…

IF: Right, exactly. So I feel that there’s still space for contributions there from the team. And maybe there’s space to grow what the goals of something like AUMI is, what are the areas we’re working on, what kind of developments we are doing, there’s still a lot of space for growth.

JS: Yeah, I think that’s a little bit of where we are right now, in terms of the actual development. One is continuing to develop the current platform that we have and add functionality to it and make fixes and optimizations. But then I think there’s a need, and we’ve really gotten into the conversation of what is what is the future of AUMI, and not just the future of the application, the overall vision for the project and where it is going.

IF: So I hope to see growth there. And something that the team addresses at some point. Because it’s not just about an application, it’s about how can technology bring something to these kids, right? How can it be more adapted to their needs, and how can it make their lives better?

JS: Yeah. I’m sort of out of questions. (laughter) I’m just curious, we’ve sort of talked about it, and maybe this is something as much for my own interest as something to contribute to the book, but, one of the questions I have written down is this: how have specific technologies used in the AUMI app, and obvious examples being using Max and the jit.cv library (Max library for computer vision) say, but how have specific technologies we’ve used for a really long time helped or hindered the on going development, in your opinion?

IF: Well, we had a bit of debate about Max actually, when I thought about rewriting the application I thought, should I do it in Max? Because somebody is going to have the same problem as me, right? They’re going to open up a patch and it’s confusion, it’s a confusing system at times. And people think differently at times, and is difficult. But eventually we decided to stick with Max and maybe its still a good balance, because I think that it’s important, because the theme is changing all the time, to have a system that a lot of people are familiar with. I think that Max is that system, so everybody that works in music technology is somehow familiar with Max and its kind of easy to pick up, even with less experienced programmers. So at one point there was a discussion like that, should we just leave Max behind? And eventually I pushed for continuing with Max because, although it has that level of complexity and can be sometimes difficult to get into (in terms of reviewing past work and adapting), I think that there’s still a lot of people that do it so it’s easy to find somebody that continues to contribute.

Even I imagine with a modular system it would be easier for somebody to have a set of rules and design a new modules for it. I think that… Well, there were little things for example. Something that for me compromised a lot, and I ended up having to develop a complex system to go around this, was the fact that a lot of the webcams were different, right? There were 4 by 3 webcams and 16 by 9 cameras, which had different formats. And for me, I always felt a bit shocked every time I opened the application I would see the imagine skewed, like it was compressed on the sides or something, and people were not proportionally represented. So in my mind, and this was not something I ever really talked to anybody about, but in my mind, okay if a kid is looking at himself distorted it must thought of some psychological impact, for him, right? Because he’s always seeing himself skewed so it doesn’t really represent reality which is what we want to do with that image on the computer. So I had to do a lot of work, just to be able to always present a good image without any distortion of what the kid was interacting with. I thought that that was important, and so it took some time, though it was not impossible to do. It was something that is easily overlooked, it’s something that is easily missed, that people don’t really think about it. So the diversity of hardware might be a challenge.

Another thing that I remember, these is more technical details, I don’t know if they’re very important but, the security measures that Apple introduced, in terms of having to sign the applications and do this and to do that, and of course, for us, doing Max application, it becomes difficult to distribute the application.

JS: It’s still a major pain point, for Windows too, because Windows security has also followed Apple’s lead. So anytime there’s a new update to put out there’s, yeah, headaches for a few weeks.

IF: Yeah, to sign, basically you have to go through a process of certifying the application while Max is not really prepared for this.

JS: Yeah Max’s own tools for distributing a standalone application aren’t necessarily the best.

IF: So this was something that seemed problematic and apparently it still is. And I know just recently, that it keeps getting worse, right? There’s more and more security layers that keep coming over, and this leaves small developers like us in a tight spot, right? It’s like you always have to test a lot to make sure the application works, but mostly you have to update continuously. And this is a problem maybe with all of digital systems at this point. Every time there is a major update on the system, or there’s a diversity of different versions of the operation system, how can you make something like AUMI work for all of them? So at some point you might have to leave some users behind, or tell them sorry, this only works on systems from something forward. And this might be especially important in schools, right? Because they don’t have so many resources and a lot of times they’re working with older computers, and older operating systems, so I find this is a challenge. It’s a challenge, and it was something that put a lot of pressure on the development of the AUMI. Because a lot of people were just, said, oh it doesn’t run, it has a throws an error. And I am like, you’ve got to go somewhere in the system, put in that you authorize an unknown application to run on your computer, so explaining this to a user is complicated. And so I hope in the future we can make this a bit smoother. Of course, for somebody proficient with computers it’s not a problem at all but you find all kinds of users, right?

JS: Yeah, and one thing we were talking about yesterday was that one of the primary objectives of the application is to make it basically incredibly simple for anyone to use, non-technical teachers, practitioners, therapists, whoever, should be able to just open it up, turn it on, and it runs.

IF: Right.

JS: So getting mired in system configuration to allow permissions on your computer…

IF: Yeah, and that’s an extreme case. But even that challenge is always there, right? Because any application is a trade off between functionality and complexity. There’s no way around it. I also had to deal a bit with this. A lot of people were like, oh I can’t make this work, because before we had less things, and now we have to press certain buttons for something to work. Uhm, but at the same time you have users that were like, it would be great if it did this, and it did that, and something like… So I guess that’s a balance you have to reach with any application. Something I proposed wouldn’t it be better for AUMI to be a group of applications? So if you want a sampler, there’s the sampler. There’s a bunch of samplers. If you want this or that that, or you want to play the piano, you launch the piano, or you can launch a lot of them at the same time.

JS: Right.

IF: You would be able to transform all those functionalities in little applications that would do something really well, instead of trying to start building something that continues to grow and try to implement more and more functionality, but in the end of the day, some of the teachers just want a simple thing. But I saw also the opposite, which was some people at the Mackay Centre School were like, okay, so how do I put like heavy metal guitar riff here, because its what the kids want? (laughter). And we’re like, okay, you can’t really do that, this is just a piano, right? So we went back and we were looking at how can we fulfill these requirements without making the application very complex. But I that, in some ways the tradeoff is good at this point. If the functionality wants to grow a lot, maybe it’s not such a bad idea to have a group of applications, simpler applications.

JS: Right, that’s come up a little bit because, like you’ve mentioned before and we continue to talk about it, and I’ve done some prototypes around developing types of interfaces, physical interfaces, or things that aren’t necessarily camera based.

IF: Right.

JS: So, you know, the the current AUMI framework can work but more and more when I… the farther I go out the more I’m sort of bending the modularity a little bit. Ok now I need to, this type of data export that will correspond to this, and now this other module doesn’t work, so at a point you’re really pushing the boundaries of what this application is designed to do. So at what point does it make more sense to have something that is a separate application, or a separate group of applications, like you say.

IF: And even not not just applications, for example. As you know, a lot of the work I’ve done during my PhD has to do with embedded computing. So the ability to put the processing of this type of musical interactions inside objects, so that you have a physical object that makes sound. Which is something… You know, I have a young daughter now, and a lot of kids are used to toys, they have electronic toys that they push buttons and they light up and they interact with the world physically. They’re interactive toys but they’re very physical, right? And so of course some of the Mackay students, they have very deep physical disabilities so they won’t able to work with these, but a lot of them could, so one idea is that maybe we we could expand from just a notion of a computer application that works on a screen to the idea of an interactive object that kids could interact physical that had sounds, haptic feedback, maybe lights, or something like that.

JS: Right, right.

IF: And so, of course this is a bit more demanding because it’s not just loading an application, we’re talking about building devices. But I feel that, these kind of toys adapted to specific needs of those students would be also interesting avenue to pursue. And, as you know, I’ve developed a framework for the avenue of those things. So half of the work is already done. (laughter)

JS: Right.

IF: And to get the right people on board, maybe we can come up with some interesting solutions.

JS: I think it was specifically related to a project that, I think it’s mostly Edu, one of the colleagues from our lab is heading up now, but it’s with the Viger group here in Montreal, they’re working a bit with us here to, I don’t know the full details but basically we’re going to support them… they are putting together mobile work spaces for interactive music making and things like that. So Marcelo (Wanderley, director of the IDMIL) had thrown out at our meeting and I followed up with Eric, there could be definite ties to continued AUMI development and we have a potential grant opportunity in the future to develop playground installations for for children with disabilities and things like that.

IF: Exactly.

JS: And one thing that Eric was really excited about was, I don’t know what is happening with this grant in the future who know where we’ll end up with it. But it’d be very easy with just the stuff we’re developing now, let’s sit down at the beginning and just have some conversations. His point (he said this in our interview yesterday), he said basically any instrument you design is adaptive, to some performer, some situation, you know, everything is adaptive in some way, maybe not specifically for kids with physical disabilities. So it only takes a little preparation at the beginning to say, hey, if we did want to use some of this for specific use cases, for kids with physical handicaps and stuff, it would be really easy to plant that early on and have things in place to design really easily and integrate all of the other stuff we are working on.

IF: Right, yes.

So, yeah, in the future I’d like to see this, us understanding better the requirements, for example, those kinds of devices, how to take them forward, working closely with the Mackay Centre School and other schools, professionals of this area. And to see how we can explore other types of technologies, of music technologies that we have or just interactive technologies we have. Because I think there’s so much to be done.

JS: Absolutely. Alright, we’ve gone well over our time. Good conversation though!

IF: (laughter) Okay.

JS: Thank you for your time.

IF: Thank you for having me.