The Sounds of Slumberland 🔊
[p]So in this update, what I want to talk about is the process of getting the sound effects created and implemented in the game. But before I get into that process, it would probably help for me to give some context about the systems I’ve designed for the sound in the game.[/p][p][/p][h2]Audio Systems[/h2][p]Sound Channels
In Little Nemo there is one sound channel for the music, and then four separate sound channels for the sound effects. They are broken up into interface, voices, and then two diegetic channels: ambient and the default diegetic SFX channel. That last one is probably what one thinks of when thinking about game sound effects, but as you can see, there’s a lot more to it than just that.[/p][p]Here’s a look in the Unity editor at our Audio Mixer panel:[/p][p]![]()
[/p][p][/p][p]And here are the levels that are presented to the user in the settings so that they can adjust the audio:[/p][p]![]()
[/p][p]The sorting and nesting of the channels allows us to break them out for distinct user-adjustable volume levels, and it also provides us with differing audio effects for each channel.[/p][p][/p][p]Diegetic Sound Effects
The most interesting of these are the diegetic sound effects. That channel has two filters, reverb and lowpass, whose settings get adjusted dynamically as the player moves throughout Slumberland, based on the environment they’re currently in. This allows us to produce a variety of effects like making an area sound more like a cavernous or interior space, or making it sound like we’re a bit underwater. The sound effects themselves don’t need to know anything about this, it will happen automatically to the entire channel as needed, so each sound effect shouldn’t worry about trying to fit a certain acoustic space.[/p][p]These diegetic sound effects get spawned into the world based on where the sound occurred. This is important because we also fade and pan a sound effect based on its location as compared to the camera.[/p][p][/p][p]![]()
[/p][p]The sound effects (represented in this editor view above by the white icons) on camera will be at full volume, while the one just off camera on the right will be quieter and panned slightly to the right ear.[/p][p][/p][p]Voice Sound Effects
The sound channel for the voices has a lot more filters available to tweak. But rather than being location specific, these get tweaked based on whoever is currently speaking. Each actor has a defined voice, which is a combination of audio clips (one for each letter) and parameters for the channel’s filters.[/p][p][/p][p]Interface and Ambient
The interface and ambient audio channels are a bit simpler as these don’t need any type of filtering or spatial considerations. These sound effects just get slotted in pretty simply. And for clarity, ambient audio is the stuff you probably don’t even know is there: the very gentle sound of wind in the Dreamswept Plains, the gentle trickling of water from nearby fountains in the Palace exteriors, the distant din of car and foot traffic in Nightlight City.[/p][p]Okay, so with that context set, what does the process actually look like?
 [/p][h2]SFX Implementation Process 🔨[/h2][p]Documenting
It all starts, of course, in a document. I will create a list of sound effects that are needed, broken down into groups (by enemy or character for instance). Here is where I try to convey what we’re looking for exactly and some in-game footage that Jonathan can use both as reference, but also to test his sounds by syncing it up with the video and making sure it feels right.[/p][p]![]()
Here’s a pic of the most recent doc I sent to Jonathan that we’ve been working from[/p][p][/p][p]Making the Clips From here, Jonathan will create sound effects for each specific item I have called out, usually with several different versions to try out. Usually at least one of these will be perfect, but if not, I can mark it in our shared document that it needs further revisions. I can’t say too much about his process in this step, but if anyone is interested to know more please let me know in the comments and I can talk to him and get more details about how the .wav files actually get made.[/p][p][/p][p]Implementing in Unity
When I’m testing out new sound effects, ideally it’s a simple use case of “this sound fires whenever this thing happens”. Those are typically very easy to implement, and it looks something like this:[/p]
 [/p][h3]The Complexities of the Implementation Step 🔍[/h3][p]In the above description of the implementation, the third step made things sound pretty simple, and it often is when we’re just attaching a single, momentary sound to a visual effect or playing it in a specific part of a character’s attack animation for instance.[/p][p]But often sound effects have more particular needs. Perhaps it’s some kind of looping audio that will need to neatly fade in and out as desired. Or maybe it is a simple momentary audio clip, but it’s important that the audio stops immediately if the character that emitted that sound is destroyed. For these more specialized use cases and considerations, I’ll typically develop a minor system for how to implement them. We rarely ever have to do anything a single time, so if I find I have to do something unique with a sound effect, there’s a good chance I will later need to do something similar in another scenario.[/p][p]So here are a few different scenarios for more specialized audio needs that are kind of interesting:[/p][p][/p][p]Proximity Audio Loops
Oftentimes when something has some looping audio component, it risks becoming a bit annoying or overwhelming. My solution for these scenarios is to have the volume of the sound effect be tied to Nemo’s physical proximity to the collider of the source of the audio. I first used this to introduce a threatening hum that comes from the Oblivion (it is also paired with a visual effect, slightly over-contrasting the screen as the sound gets louder), but have since used it in several other places. The solution was fairly complex to build out once, but now it’s just a matter of adding a ProximityAudioProxy component to an entity which references a Sound Effect asset.[/p][p]![]()
While you can't hear the humming in this screenshot, you might be able to tell that the contrast is increased due to Nemo being so close to the Oblivion.[/p][p][/p][p]Looping Audio More Generally Looping audio in other scenarios has important considerations about when and how it starts and stops. For simple audio loops, the Sound Effect has a “Should Loop” property you can check. That loop will continue until told to stop or the entity responsible for emitting it is gone. But loops suddenly starting at full volume, and then later abruptly ending, tend to sound bad. So in a lot of cases we use a custom subclass of the SoundEffect which is a FadeInOutLoopingSoundEffect (it does what it sounds like it does). I used this very recently while working with Jonathan’s sound effects for the Crystal Cruncher boss’ idling engine noise. The sound effect is fairly loud and helps communicate which sub-phase you are in (is the core vulnerable or not) based on whether or not it’s running. But if we just stop the sound without fading it out (even very briefly) it’s very jarring.[/p][p][/p][p]Surface Effects
Another common need is a sound when something collides with a floor or a wall. This gets a bit more generally into our Surface Effects systems, which also includes visual effects like having different bits of grass, snow, or dust kick up when walking on a surface, but without going into those details, we can simply hook into that system to specify that some objects should make sounds when making horizontal and/or vertical collisions. A good example of this is the breakable gems which appear in the Palace. In an effort to communicate that they can and should be shattered, they make a crystalline sound whenever bouncing off a floor or wall.[/p][p]![]()
[/p][p]Here's another in-editor screenshot in which you can see the gem emitting a sound effect right where it's hit the wall.[/p][p][/p][p]There are probably some other minor variations of how the diegetic sound effects get implemented, but I think that helps give a sense of how much their implementation can vary. The sound effects used for interface elements, ambient sounds, and the voices are all much more straight-forward. Interface sounds are typically immediate and momentary, ambient sounds are just looped with the volume attenuated based on how much screen space that type of environment takes up, and voice sound effects are all implemented in the same way so that I can just slot new clips in for new voices.[/p][p]But hopefully that helps paint a picture for the general workflow and process that goes into getting sound effects into Little Nemo! Let me know what you thought of this deep dive. Did I go into enough detail, too much detail, not touch on a particular aspect you’re more curious about? These are fun to dig into so I’m happy to share more details![/p]
In Little Nemo there is one sound channel for the music, and then four separate sound channels for the sound effects. They are broken up into interface, voices, and then two diegetic channels: ambient and the default diegetic SFX channel. That last one is probably what one thinks of when thinking about game sound effects, but as you can see, there’s a lot more to it than just that.[/p][p]Here’s a look in the Unity editor at our Audio Mixer panel:[/p][p]
The most interesting of these are the diegetic sound effects. That channel has two filters, reverb and lowpass, whose settings get adjusted dynamically as the player moves throughout Slumberland, based on the environment they’re currently in. This allows us to produce a variety of effects like making an area sound more like a cavernous or interior space, or making it sound like we’re a bit underwater. The sound effects themselves don’t need to know anything about this, it will happen automatically to the entire channel as needed, so each sound effect shouldn’t worry about trying to fit a certain acoustic space.[/p][p]These diegetic sound effects get spawned into the world based on where the sound occurred. This is important because we also fade and pan a sound effect based on its location as compared to the camera.[/p][p][/p][p]
The sound channel for the voices has a lot more filters available to tweak. But rather than being location specific, these get tweaked based on whoever is currently speaking. Each actor has a defined voice, which is a combination of audio clips (one for each letter) and parameters for the channel’s filters.[/p][p][/p][p]Interface and Ambient
The interface and ambient audio channels are a bit simpler as these don’t need any type of filtering or spatial considerations. These sound effects just get slotted in pretty simply. And for clarity, ambient audio is the stuff you probably don’t even know is there: the very gentle sound of wind in the Dreamswept Plains, the gentle trickling of water from nearby fountains in the Palace exteriors, the distant din of car and foot traffic in Nightlight City.[/p][p]Okay, so with that context set, what does the process actually look like?
 [/p][h2]SFX Implementation Process 🔨[/h2][p]Documenting
It all starts, of course, in a document. I will create a list of sound effects that are needed, broken down into groups (by enemy or character for instance). Here is where I try to convey what we’re looking for exactly and some in-game footage that Jonathan can use both as reference, but also to test his sounds by syncing it up with the video and making sure it feels right.[/p][p]
When I’m testing out new sound effects, ideally it’s a simple use case of “this sound fires whenever this thing happens”. Those are typically very easy to implement, and it looks something like this:[/p]
- [p]Bring one or more .wav files into Unity as Audio Clip assets.[/p]
- [p]Create a Sound Effect asset, which uses those Audio Clips. If there are multiple clips, it’s typically something like footsteps which want several variations so that we can randomly select one each time the sound effect is used. This Sound Effect asset is important because, not only can a sound require multiple Audio Clips, but it also has associated data such as the playback volume.[/p]
- [p]And finally there’s the triggering of that Sound Effect in one of our few most common ways: include it in a effect prefab (these are our effects which are generally used for effects that have some visual element), or include it in an object’s sound effect collection to be referenced from that object’s Animator.[/p]
 [/p][h3]The Complexities of the Implementation Step 🔍[/h3][p]In the above description of the implementation, the third step made things sound pretty simple, and it often is when we’re just attaching a single, momentary sound to a visual effect or playing it in a specific part of a character’s attack animation for instance.[/p][p]But often sound effects have more particular needs. Perhaps it’s some kind of looping audio that will need to neatly fade in and out as desired. Or maybe it is a simple momentary audio clip, but it’s important that the audio stops immediately if the character that emitted that sound is destroyed. For these more specialized use cases and considerations, I’ll typically develop a minor system for how to implement them. We rarely ever have to do anything a single time, so if I find I have to do something unique with a sound effect, there’s a good chance I will later need to do something similar in another scenario.[/p][p]So here are a few different scenarios for more specialized audio needs that are kind of interesting:[/p][p][/p][p]Proximity Audio Loops
Oftentimes when something has some looping audio component, it risks becoming a bit annoying or overwhelming. My solution for these scenarios is to have the volume of the sound effect be tied to Nemo’s physical proximity to the collider of the source of the audio. I first used this to introduce a threatening hum that comes from the Oblivion (it is also paired with a visual effect, slightly over-contrasting the screen as the sound gets louder), but have since used it in several other places. The solution was fairly complex to build out once, but now it’s just a matter of adding a ProximityAudioProxy component to an entity which references a Sound Effect asset.[/p][p]
Another common need is a sound when something collides with a floor or a wall. This gets a bit more generally into our Surface Effects systems, which also includes visual effects like having different bits of grass, snow, or dust kick up when walking on a surface, but without going into those details, we can simply hook into that system to specify that some objects should make sounds when making horizontal and/or vertical collisions. A good example of this is the breakable gems which appear in the Palace. In an effort to communicate that they can and should be shattered, they make a crystalline sound whenever bouncing off a floor or wall.[/p][p]