The Planets: exploring new realities of sound in an inter-galactic soundwalk in 3D audio

With the immersive audio app The Planets you can experience the famous eponymous orchestral suite by Gustav Holst, played by the Munich Philharmonic Orchestra, as an interactive audio walk in a park near you, increasingly all over the world. 

The app was just nominated for the Sound Walk Awards 2022 and hopes to win an Early 2023 award. We talked to the sound director and producer of the app - Mathis Nitschke - to get a deeper insight into this inspiring and rousing production. 

Mathis, what inspired you to translate Gustav's composition into an interactive app? - And why did you decide to use spatial audio?

Several trains of thought came together here. In 2015, as an opera composer, I experimented with the idea of creating a music theatrical piece along the Munich river Isar, which you experience by walking it. When the VR revival hit in 2017, I immediately started trying out different recording techniques to explore the question of whether it would be sonically interesting - and also musically satisfying - to take the listener's perspective from inside the orchestra. This was also the moment when I met Dear Reality’s founders Achim Fell and Christian Sander, whose technology has accompanied and supported my developments ever since.

The artistic co-creator of "The Planets" Gunter Pretzel already had this inside perspective of the orchestra: he was a violist with the Munich Philharmonic for over 30 years. He describes this experience inside the orchestra as so exciting and powerful that he, and many of his colleagues, have difficulties feeling the same excitement when they take the standard listener's perspective in the concert hall. That's why, over the course of his career, he has developed a number of projects that allow listeners to get close to the musicians.

After Pretzel tried out my soundwalk “Vergehen”, he approached me with the idea to use smartphones and headphones to create this perspective, not knowing that I was already experimenting with this. As a proof of concept, we created the app “Inside MPhil” in 2019 which allows one to walk through the imaginary orchestra, set on a meadow. With this, you can experience one movement out of Schumann’s “Frühlingssymphony”, in high spatial resolution. You enter the orchestra acoustically and hear the instruments as if you were standing next to them, just as the musicians hear themselves and their colleagues: the violins, the oboe, the timpani, or whatever you want to hear.

After that successful PoC, we aimed for a full-length symphonic experience and started the work on “The Planets”.

How were you able to get the Munich Philharmonic on board?

Gunter was already well-known in the orchestra administration for his cutting-edge projects. That’s why we had no problem convincing them to be part of this endeavor. But even for an orchestra like the Munich Philharmonic, the development and production costs of such a complex project are challenging. Therefore, we were all very grateful for the newly established XR prototype funding opportunity from FilmFernsehFonds Bayern which we received.

Did you use any kind of special microphone techniques for the recording process and at what stage of production did Dear Reality's plugins come in?

Projects dealing with immersive, interactive audio unfortunately often focus on the technical aspects and lack musical qualities. This is understandable in that, for example, crosstalk between microphones greatly affects spatial resolution, and the comb filter effects often created by binaural filters can be very unpleasant. For this reason, many projects try to record one instrument at a time. But this is a guarantee for musical boredom! An orchestra must play as a whole, in a hall, in front of an audience.

So, for me, it was clear that we must use a live concert recording - and I will have to deal with all the technical compromises. But the music itself should remain uncompromised at all costs. If the music doesn’t work, nothing works!

The Munich Philharmonic records all their performances anyway, in their standard setup using main and spot microphones. Usually, they don’t use spot microphones on the brass as those are loud enough in the main mics. The orchestra’s recording producer Johannes Müller was extremely cooperative and provided not only the necessary spots on the brass but also went with me on some more experimental techniques like a quad outrigger setup which in the end I didn’t use, though.

But there’s also always room for improvisation: The distant chorus at the end of the “Planets” I recorded very simply and elegantly with two handheld Zoom recorders, as we couldn’t set up “real” microphones in the staircase for security reasons.

The Dear Reality products were used in two different ways. dearVR Unity carries the main bulk of the work: instantiated directly in Unity to provide the high-quality Dear Reality sound in the virtual scenarios in Unity, it provides the magic of inter-active binaural 3D Audio in real-time. Additionally, the Spatializer plugin dearVR PRO was used to binauralize and embed the static elements, that were not dependent on head tracking, into the audio mix. The two together complement each other nicely and tie the audio events together nicely.

During the process, what hurdles did you have to overcome that you wouldn't have encountered in a normal production?

In the 2019 Proof of Concept we used every single spot microphone as one 3D audio source in Unity, spatialized by the dearVR Unity plugin. The resulting high spatial resolution was possible because the smaller Schumann orchestra demanded “only” 22 simultaneous audio sources and the program length was limited to around10 minutes.

The Holst orchestra is double the size, and additionally, we talk about a full symphonic experience of 1-hour in length. 40 simultaneously playing tracks, individually spatialized, are not feasible at all for even the most powerful smartphones.

After some experimentation, I came up with a downmix to an octagonal circle with 8 audio sources through which you can move. That works very well and reduces CPU and memory costs to 13 simultaneous 3D audio sources, including atmospheric FX and voice elements.

What exactly did your workflow look like with the Dear Reality plugins?

To be able to pre-listen the effect of the interactive 3D binauralisation of the octagonal circle in the user’s smartphone, I created a static binaural listening environment in an 8.0 bus in Nuendo, using 8 instances of dearVR Pro.

The implementation of the dearVR Source in Unity is more or less textbook. Notice the strong distance correction to enable a larger-than-life acoustic experience (sound would be too soft too soon in a quasi-realistic setting).

One has to find a planet first before one can land on it and listen to the music. An attractive tone coming from the planet leads the user. As there might be quite some way to go (100-200m), the distance attenuation has to be very flat. To allow for a little more feeling of getting closer, a distance-controlled low pass filter is added on top.

In the preliminary interviews, we experienced you as particularly committed, curious, and open-minded. That's why we're asking ourselves: What's next for the app and what do your next projects look like?

I love soundwalks. But at the same time, I have my gripes.

Usually, soundwalks are firmly linked to GPS coordinates. In this way, however, they can only be experienced at select locations for which the soundwalk was designed or at least adapted. This is a blessing and a curse.

“The Planets” is already one step further in this regard: it is site-adaptive and will be available at more and more locations over time (currently over 80 worldwide).

But my personal holy grail is a soundwalk that automatically adapts to the user's current location in a meaningful way. I’m working on something like this now….

Experience The Planets yourself, available on the App Store for iOs, or Google Play for Android. 

If you are feeling creative, try out dearVR’s audio spatialization tools

dearVR’s tools are designed to give artists and engineers an unprecedented level of control and flexibility over spatial audio workflows. Whether you are new to, or a professional in, the spatial audio scene, these tools help to expand and streamline the creative process of working spatially with sound.

Mathis Nitschke
Mathis Nitschke is an artist who works between disciplines. He combines his studies of classical guitar, visual arts and musical composition with theatre and opera, as well as for some years also coding. In collaboration with artists from various disciplines, Nitschke tries to give contemporary music theatre a new face. In 2015, Mathis Nitschke founded Sofilab, a Munich-based sonic design studio and innovation lab working at the crossroads of art, industry and technology.

Also in Overview

dearVR SPATIAL CONNECT introduces VR mixing to Cubase 12
dearVR SPATIAL CONNECT introduces VR mixing to Cubase 12

Spatial audio redefines how artists, engineers, and listeners interact with sound. The new audio standard engulfs the listener and provides a much more emotional auditory experience by having sounds from every angle and distance. To unlock creative mixing possibilities and to empower producers to craft more lively and engaging music productions, Dear Reality introduces its VR application to Steinberg’s Cubase 12 audio workstation.

Read More

Discover MIYA and add Texture to your audio
Discover MIYA and add Texture to your audio

Re-synthesize your audio based on its DNA and go from slight alterations to a complete aggressive reconstruction with MIYA, Dear Reality's latest wavelet distortion plugin for audio signals. MIYA follows a radically different approach to distortion by creating a wide range of permutations of the original input using adjustable harmonics. Enhance your sounds to punch through the mix by sculpting them like a synthesizer and go from subtle textures to gnarly distortion and even massive sub-harmonics.

Read More

Immersive storytelling through spatial sound design
Immersive storytelling through spatial sound design

Imagine you are in the middle of an exciting experience, you are spellbound following the intense performance of a passionate theatre group.  Or a very different treat for the senses - you are enjoying an excellent dinner.  Now imagine the whole event is taking place in complete darkness.

Read More