With the immersive audio app The Planets you can experience the famous eponymous orchestral suite by Gustav Holst, played by the Munich Philharmonic Orchestra, as an interactive audio walk in a park near you, increasingly all over the world.
The app was just nominated for the Sound Walk Awards 2022 and hopes to win an Early 2023 award. We talked to the sound director and producer of the app - Mathis Nitschke - to get a deeper insight into this inspiring and rousing production.
Mathis, what inspired you to translate Gustav's composition into an interactive app? - And why did you decide to use spatial audio?
Several trains of thought came together here. In 2015, as an opera composer, I experimented with the idea of creating a music theatrical piece along the Munich river Isar, which you experience by walking it. When the VR revival hit in 2017, I immediately started trying out different recording techniques to explore the question of whether it would be sonically interesting - and also musically satisfying - to take the listener's perspective from inside the orchestra. This was also the moment when I met Dear Reality’s founders Achim Fell and Christian Sander, whose technology has accompanied and supported my developments ever since.
The artistic co-creator of "The Planets" Gunter Pretzel already had this inside perspective of the orchestra: he was a violist with the Munich Philharmonic for over 30 years. He describes this experience inside the orchestra as so exciting and powerful that he, and many of his colleagues, have difficulties feeling the same excitement when they take the standard listener's perspective in the concert hall. That's why, over the course of his career, he has developed a number of projects that allow listeners to get close to the musicians.
After Pretzel tried out my soundwalk “Vergehen”, he approached me with the idea to use smartphones and headphones to create this perspective, not knowing that I was already experimenting with this. As a proof of concept, we created the app “Inside MPhil” in 2019 which allows one to walk through the imaginary orchestra, set on a meadow. With this, you can experience one movement out of Schumann’s “Frühlingssymphony”, in high spatial resolution. You enter the orchestra acoustically and hear the instruments as if you were standing next to them, just as the musicians hear themselves and their colleagues: the violins, the oboe, the timpani, or whatever you want to hear.
After that successful PoC, we aimed for a full-length symphonic experience and started the work on “The Planets”.
How were you able to get the Munich Philharmonic on board?
Gunter was already well-known in the orchestra administration for his cutting-edge projects. That’s why we had no problem convincing them to be part of this endeavor. But even for an orchestra like the Munich Philharmonic, the development and production costs of such a complex project are challenging. Therefore, we were all very grateful for the newly established XR prototype funding opportunity from FilmFernsehFonds Bayern which we received.
Did you use any kind of special microphone techniques for the recording process and at what stage of production did Dear Reality's plugins come in?
Projects dealing with immersive, interactive audio unfortunately often focus on the technical aspects and lack musical qualities. This is understandable in that, for example, crosstalk between microphones greatly affects spatial resolution, and the comb filter effects often created by binaural filters can be very unpleasant. For this reason, many projects try to record one instrument at a time. But this is a guarantee for musical boredom! An orchestra must play as a whole, in a hall, in front of an audience.
So, for me, it was clear that we must use a live concert recording - and I will have to deal with all the technical compromises. But the music itself should remain uncompromised at all costs. If the music doesn’t work, nothing works!
The Munich Philharmonic records all their performances anyway, in their standard setup using main and spot microphones. Usually, they don’t use spot microphones on the brass as those are loud enough in the main mics. The orchestra’s recording producer Johannes Müller was extremely cooperative and provided not only the necessary spots on the brass but also went with me on some more experimental techniques like a quad outrigger setup which in the end I didn’t use, though.
But there’s also always room for improvisation: The distant chorus at the end of the “Planets” I recorded very simply and elegantly with two handheld Zoom recorders, as we couldn’t set up “real” microphones in the staircase for security reasons.
The Dear Reality products were used in two different ways. dearVR Unity carries the main bulk of the work: instantiated directly in Unity to provide the high-quality Dear Reality sound in the virtual scenarios in Unity, it provides the magic of inter-active binaural 3D Audio in real-time. Additionally, the Spatializer plugin dearVR PRO was used to binauralize and embed the static elements, that were not dependent on head tracking, into the audio mix. The two together complement each other nicely and tie the audio events together nicely.
During the process, what hurdles did you have to overcome that you wouldn't have encountered in a normal production?
In the 2019 Proof of Concept we used every single spot microphone as one 3D audio source in Unity, spatialized by the dearVR Unity plugin. The resulting high spatial resolution was possible because the smaller Schumann orchestra demanded “only” 22 simultaneous audio sources and the program length was limited to around10 minutes.
The Holst orchestra is double the size, and additionally, we talk about a full symphonic experience of 1-hour in length. 40 simultaneously playing tracks, individually spatialized, are not feasible at all for even the most powerful smartphones.
After some experimentation, I came up with a downmix to an octagonal circle with 8 audio sources through which you can move. That works very well and reduces CPU and memory costs to 13 simultaneous 3D audio sources, including atmospheric FX and voice elements.
What exactly did your workflow look like with the Dear Reality plugins?
To be able to pre-listen the effect of the interactive 3D binauralisation of the octagonal circle in the user’s smartphone, I created a static binaural listening environment in an 8.0 bus in Nuendo, using 8 instances of dearVR Pro.
The implementation of the dearVR Source in Unity is more or less textbook. Notice the strong distance correction to enable a larger-than-life acoustic experience (sound would be too soft too soon in a quasi-realistic setting).
One has to find a planet first before one can land on it and listen to the music. An attractive tone coming from the planet leads the user. As there might be quite some way to go (100-200m), the distance attenuation has to be very flat. To allow for a little more feeling of getting closer, a distance-controlled low pass filter is added on top.
In the preliminary interviews, we experienced you as particularly committed, curious, and open-minded. That's why we're asking ourselves: What's next for the app and what do your next projects look like?
I love soundwalks. But at the same time, I have my gripes.
Usually, soundwalks are firmly linked to GPS coordinates. In this way, however, they can only be experienced at select locations for which the soundwalk was designed or at least adapted. This is a blessing and a curse.
“The Planets” is already one step further in this regard: it is site-adaptive and will be available at more and more locations over time (currently over 80 worldwide).
But my personal holy grail is a soundwalk that automatically adapts to the user's current location in a meaningful way. I’m working on something like this now….
Experience The Planets yourself, available on the App Store for iOs, or Google Play for Android.
If you are feeling creative, try out dearVR’s audio spatialization tools!
dearVR’s tools are designed to give artists and engineers an unprecedented level of control and flexibility over spatial audio workflows. Whether you are new to, or a professional in, the spatial audio scene, these tools help to expand and streamline the creative process of working spatially with sound.