Audio pitch shifting, recording & basic editing in-game, audio optimizations
Welcome to another weekly update!
As part of the optimizations, we have decided to rewrite Neos’ audio system, since it was one of the old and hacky pieces of the codebase that needed to be reworked for a while. With the new system not only the performance is improved, but it brings lots of new audio-related functionality as well!
Neos now supports playing audio clips at various speeds (including reverse), changing their pitch as a result and allowing for a wide range of audio effects. It supports basic audio editing as well, allowing you to quickly normalize volume, trim sounds or add fade in and fade out effects without leaving Neos.
The new audio system supports encoding as well (currently WAV, OGG Vorbis and FLAC) which enables the addition of a new Microphone tool, which lets you record audio clips directly in-game. Some long-standing bugs were fixed too, like short audio clips not always playing.
[h2]Audio Playback Speed / Pitch shifting[/h2]
Thanks to the new and improved audio system, built around a new class called AudioX for handling audio data, Neos now supports playing any audio clips at arbitrary playback rates. Playing the audio clip faster will result in increased pitch, while playing it slower will lower the pitch. Using a negative number the clip can now also play in reverse.
This allows for a wide variety of effects. The ability is exposed through the inspector, as well as through LogiX. For example using PlayOneShot you can change the pitch of played sound effects to create musical instruments or to provide random variations.
Using the DrivePlayback node, you can smoothly vary the playback speed of the clip to create effects like a car engine speeding up or slowing down. To achieve this, the node now has two new inputs - Speed and MaximumPlaybackError.
The MaximumPlaybackError can be particularly important, as it determines if the node will force a playback position to resync each update. By increasing the value you can give a small tolerance for error, but still ensuring the playback stays synchronized. By setting the allowed error to a really large value (e.g. infinity), it will never re-sync (unless an impulse is sent to force it), preventing any playback jumps.
For sound effects like the revving of an engine the exact playback position might not be important so it’s a good choice.
[previewyoutube][/previewyoutube]
[h2]Basic Audio Editing[/h2]
Neos has allowed basic editing/filtering for meshes and textures for a while, but not for sounds. Thanks to the new AudioX system we have full control over the audio data and now expose basic editing functionality on the StaticAudioClip component in the inspector.
[previewyoutube][/previewyoutube]
You can now normalize the audio volume if the clip is too quiet, trim a fixed duration from the beginning or end of the clip, trim silence at the beginning/end of the clip and add a volume fade in or fade out effect. You also have a choice to convert the clip into WAV, OGG Vorbis and FLAC.
This should help reduce the need to use external software for a lot of common tasks, with a lot more features to come in the future (and at some point, we plan to have full non-linear audio editing support, but that’s longer off right now).
[previewyoutube][/previewyoutube]
[h2]Audio Recording & Stereo streams[/h2]
Thanks to the AudioX being able to encode any audio data, we were able to add a new Microphone tool, which you can now find in your Neos Essentials. Using this tool you can record your voice and have it immediately spawn as an audio clip in the world.
You can choose to record in WAV, OGG Vorbis or FLAC formats and either use raw audio data from your microphone or use the filtered ones (noise-gated and normalized). We hope that this tool will be useful, either for taking notes or quickly prototyping and making sound effects and voice clips.
[previewyoutube][/previewyoutube]
On related note, the audio input system now also supports stereo audio input, which is used by the microphone tool. The audio synchronization streams support it as well, but voice is still mono by default, as stereo isn’t needed in most cases and will save some bandwidth (it might change if the bandwidth increase would be negligible).
[h2]Improved audio performance[/h2]
The new audio system offers several improvements in performance. It uses modern C# features like Span, which lets it efficiently reinterpret audio buffers and handle multi-channel (e.g. stereo) buffers using vectorization. Thanks for this new framework many redundant copies and conversions were eliminated.
The system also supports directly reading audio data at varying rates, which is used both for pitch shifting and re-sampling on the fly in one operation, together with conversion from mono to stereo and vice versa. It heavily simplifies the underlying code as well, reducing bugs and making it easier to add new features.
Another performance related addition is an Audio Buffer Pool system. For any audio clips that are decoded on the fly, the decoded chunks are cached in a memory pool, to avoid the need to re-decode them every time. This can help particularly in cases when a compressed audio clip is played multiple times or when it’s read multiple times (e.g. VolumeMeter).
In related fashion, Neos now also allows multiple ways of loading an audio clip. It can be either fully decoded in memory, which is ideal for shorter sound effects that are played often. The clip can also be decoded on the fly, either from the file on the disk or from memory. This is useful for anything that’s longer, as completely decoding the clip would take a lot of memory, e.g. ambience, music or voice.
By default the choice is automatic, which currently fully decodes anything shorter than 10 seconds and streams from file for everything else, but you can override this behavior if needed. When the clip is fully decoded, Neos can also resample it to the system's output sample rate, which speeds up the reads at the 1.0 playback speed, since they can just copy the raw data.
[h2]Fixed audio issues[/h2]
Reworking the old error-prone code also let us fix some of the long-standing issues. Particularly the issue with short audio clips not playing after they were first played. This actually affected all audio clips, due to the first few hundred samples being skipped. Now all clips should play perfectly every time.
The real-time synchronization streams were reworked and tweaked as well, to better decide when to start playing back data from the circular buffer (filled from the decoded data transmitted over the network). This should help reduce voice stutters due to network jitter and also eliminate some artifacts, where a small chunk of audio data would get skipped.
[h2]Community Highlights[/h2]
[h3]Chao Garden by Sykes[/h3]
Sykes, one of our newest moderators, has recreated the Chao Garden from Sonic Adventure 2! This build was inspired by the game and fully remade in blender, utilizing custom assets and was previously made for Second Life. Come and give it a visit!
[h3]Hallwyl Museum by Enverex[/h3]
The Hallwyl Museum’s photogrammetry model has been repurposed as a world in Neos! The doors act as teleporters so you can visit the many bright and beautiful rooms of this magnificent place. Photos taken in here look almost like real world photographs of the actual museum, so bring a camera and a friend!
------------------------------------------------------
Anyway, that's all for this week! Thank you again for reading this weekly update! All of this work is possible thanks to your help and support and we can’t wait to see what will you make with the new audio capabilities in Neos!
As part of the optimizations, we have decided to rewrite Neos’ audio system, since it was one of the old and hacky pieces of the codebase that needed to be reworked for a while. With the new system not only the performance is improved, but it brings lots of new audio-related functionality as well!
Neos now supports playing audio clips at various speeds (including reverse), changing their pitch as a result and allowing for a wide range of audio effects. It supports basic audio editing as well, allowing you to quickly normalize volume, trim sounds or add fade in and fade out effects without leaving Neos.
The new audio system supports encoding as well (currently WAV, OGG Vorbis and FLAC) which enables the addition of a new Microphone tool, which lets you record audio clips directly in-game. Some long-standing bugs were fixed too, like short audio clips not always playing.
[h2]Audio Playback Speed / Pitch shifting[/h2]
Thanks to the new and improved audio system, built around a new class called AudioX for handling audio data, Neos now supports playing any audio clips at arbitrary playback rates. Playing the audio clip faster will result in increased pitch, while playing it slower will lower the pitch. Using a negative number the clip can now also play in reverse.
This allows for a wide variety of effects. The ability is exposed through the inspector, as well as through LogiX. For example using PlayOneShot you can change the pitch of played sound effects to create musical instruments or to provide random variations.
Using the DrivePlayback node, you can smoothly vary the playback speed of the clip to create effects like a car engine speeding up or slowing down. To achieve this, the node now has two new inputs - Speed and MaximumPlaybackError.
The MaximumPlaybackError can be particularly important, as it determines if the node will force a playback position to resync each update. By increasing the value you can give a small tolerance for error, but still ensuring the playback stays synchronized. By setting the allowed error to a really large value (e.g. infinity), it will never re-sync (unless an impulse is sent to force it), preventing any playback jumps.
For sound effects like the revving of an engine the exact playback position might not be important so it’s a good choice.
[previewyoutube][/previewyoutube]
[h2]Basic Audio Editing[/h2]
Neos has allowed basic editing/filtering for meshes and textures for a while, but not for sounds. Thanks to the new AudioX system we have full control over the audio data and now expose basic editing functionality on the StaticAudioClip component in the inspector.
[previewyoutube][/previewyoutube]
You can now normalize the audio volume if the clip is too quiet, trim a fixed duration from the beginning or end of the clip, trim silence at the beginning/end of the clip and add a volume fade in or fade out effect. You also have a choice to convert the clip into WAV, OGG Vorbis and FLAC.
This should help reduce the need to use external software for a lot of common tasks, with a lot more features to come in the future (and at some point, we plan to have full non-linear audio editing support, but that’s longer off right now).
[previewyoutube][/previewyoutube]
[h2]Audio Recording & Stereo streams[/h2]
Thanks to the AudioX being able to encode any audio data, we were able to add a new Microphone tool, which you can now find in your Neos Essentials. Using this tool you can record your voice and have it immediately spawn as an audio clip in the world.
You can choose to record in WAV, OGG Vorbis or FLAC formats and either use raw audio data from your microphone or use the filtered ones (noise-gated and normalized). We hope that this tool will be useful, either for taking notes or quickly prototyping and making sound effects and voice clips.
[previewyoutube][/previewyoutube]
On related note, the audio input system now also supports stereo audio input, which is used by the microphone tool. The audio synchronization streams support it as well, but voice is still mono by default, as stereo isn’t needed in most cases and will save some bandwidth (it might change if the bandwidth increase would be negligible).
[h2]Improved audio performance[/h2]
The new audio system offers several improvements in performance. It uses modern C# features like Span, which lets it efficiently reinterpret audio buffers and handle multi-channel (e.g. stereo) buffers using vectorization. Thanks for this new framework many redundant copies and conversions were eliminated.
The system also supports directly reading audio data at varying rates, which is used both for pitch shifting and re-sampling on the fly in one operation, together with conversion from mono to stereo and vice versa. It heavily simplifies the underlying code as well, reducing bugs and making it easier to add new features.
Another performance related addition is an Audio Buffer Pool system. For any audio clips that are decoded on the fly, the decoded chunks are cached in a memory pool, to avoid the need to re-decode them every time. This can help particularly in cases when a compressed audio clip is played multiple times or when it’s read multiple times (e.g. VolumeMeter).
In related fashion, Neos now also allows multiple ways of loading an audio clip. It can be either fully decoded in memory, which is ideal for shorter sound effects that are played often. The clip can also be decoded on the fly, either from the file on the disk or from memory. This is useful for anything that’s longer, as completely decoding the clip would take a lot of memory, e.g. ambience, music or voice.
By default the choice is automatic, which currently fully decodes anything shorter than 10 seconds and streams from file for everything else, but you can override this behavior if needed. When the clip is fully decoded, Neos can also resample it to the system's output sample rate, which speeds up the reads at the 1.0 playback speed, since they can just copy the raw data.
[h2]Fixed audio issues[/h2]
Reworking the old error-prone code also let us fix some of the long-standing issues. Particularly the issue with short audio clips not playing after they were first played. This actually affected all audio clips, due to the first few hundred samples being skipped. Now all clips should play perfectly every time.
The real-time synchronization streams were reworked and tweaked as well, to better decide when to start playing back data from the circular buffer (filled from the decoded data transmitted over the network). This should help reduce voice stutters due to network jitter and also eliminate some artifacts, where a small chunk of audio data would get skipped.
[h2]Community Highlights[/h2]
[h3]Chao Garden by Sykes[/h3]
Sykes, one of our newest moderators, has recreated the Chao Garden from Sonic Adventure 2! This build was inspired by the game and fully remade in blender, utilizing custom assets and was previously made for Second Life. Come and give it a visit!
[h3]Hallwyl Museum by Enverex[/h3]
The Hallwyl Museum’s photogrammetry model has been repurposed as a world in Neos! The doors act as teleporters so you can visit the many bright and beautiful rooms of this magnificent place. Photos taken in here look almost like real world photographs of the actual museum, so bring a camera and a friend!
------------------------------------------------------
Anyway, that's all for this week! Thank you again for reading this weekly update! All of this work is possible thanks to your help and support and we can’t wait to see what will you make with the new audio capabilities in Neos!