Neos VR News

File share support! You can now bring in any file into Neos as an item, whether supported for import or not and easily share them with others - hand them over, export them, import them from the item, save to your inventory for storage/backup (or to transfer between your computers) or publish supplemental files with your worlds (e.g. you can put a Python script with your websocket example in-world).

This is using the same asset subsystem that all the other assets do, except it does no actual processing/decoding of the data. For supported file types, like .FBX, .Blend, .PSD and so on, you can now either import them or bring in as raw file and share with others unmodified (e.g. if you want to collaborate on a Blender or Photoshop file). You can quickly export them to your Documents through the context menu as well.

Since this wasn't too difficult to add thanks to the existing subsystem, but can help improve some workflows and enable new use-cases. I hope you'll find this useful!

Also a bunch of other tweaks and bugfixes as usual.

[h2]New Features:[/h2]
- Added ability to import raw files as items. Those can be shared with others, saved to your inventory, exported and imported (based on recent request by @Aegis_Wolf | Art Director and @Rue Shejn | Artist 3D and many before)
-- When copy & pasting files or drag and dropping, hold the Shift key to import as raw file
-- For models, images and videos there is now a new import option "Raw File"
-- Any unsupported file format will be automatically imported as raw file (e.g. ZIP)
-- You can also use the orange file button in the built-in file browser
- You can Export raw file through the context menu. This will place it in "Documents/Neos VR" using its original name (or with number appended if there's already one)
- You can also import the file as regular using the context menu. This lets you bring in the files and re-run the import anytime in-game
- Added "Binary" asset type and StaticBinary asset provider
-- This is raw binary data that is not processed in any way in Neos and powers the new file share functionality.
-- More functions (like accessing bytes with LogiX) will come at some point in future.
-- It uses same asset synchronization as everything else - if imported within a session, it's transfered over P2P connection without touching the cloud. If stored to inventory or cloud, it'll be uplaoded and counted against your quota
-- Note that by default it's not transfered to other users until they perform an action (Import / Export or Save to their inventory), so simply bringing it in won't cause network load to all users. However any file you bring into a public session CAN be downloaded by anyone in that session.

- Added "FileMetadata" component, which stores the original file name and it's MIME type. More info can be Added
- Added "BinaryExportable" component, which allows to export the raw binary asset through the File browser
- Added FileVisual component, which provides dynamic visual for the raw files, showing their name, extension and type class color
-- It will show an animation anytime the file is processed (imported/exported), flashing once it's done
-- Base 3D model was created by @Aegis_Wolf | Art Director
-- File type icons will be added at later point

[h2]Tweaks:[/h2]
- When saving item/world, Neos will now skip any workers (e.g. Components) that fail to save due to an error
-- This allows for most of the world/item to be saved with partial data loss, rather than not being able to save the whole thing at all (based on report by @Jeff07734 and @reddragon through @Shifty | Quality Control Lead)
- Improved type name beautification to prevent inserting spaces in middle of numbers (based on report by @art0007i)
- Avatar creator will now recognize "closed" in the name of eye blendshape as blinking, instead of just "close" (e.g. "Left Eye Closed" will now work, based on report by @Earthmark)
- "Save as screenshot" is now localizable
- Import dialogs are no longer persistent
- Updated AdminX utility, to fix any deserialization issues with the new voice messages causing the tool to break
- Neos now updates any asset URL's in the Record tags when syncing to cloud, ensuring that they have a correct cloud URL
-- This is now used to also reference the Photo, File or Audio clip asset directly in tags, which will enable future features (old items wil need to be reclassified though)

[h2]Bugfixes:[/h2]
- Fixed UserDistanceValueDriver being instancable with incorrect generic argument (e.g. Material, causing things like saving to break as reported by @Jeff07734 and @reddragon through @Shifty | Quality Control Lead)
- Fixed Neos crashing due to a circular reference for proxy bounds source on SkinnedMeshRenderer (e.g. setting the proxy source to itself, based on reports by @Hayden (PolyLogiX Studio), @H3BO3 and @Enverex#6528)
- Fixed saving of screenshot through context menu breaking of the URL is null
- Added extra diagnostic information for audio clip read errors

Just a small maintenance build. Improvements to the new voice message system, improving security of the "Silence" action and few other tweaks.

[h2]Tweaks:[/h2]
- Voice message recording will now trim any beginning silence
- Voice message recording will now wait a small amount of time after released, to prevent last segment of the message getting cut off
- Voice message won't be sent if the audio clip is empty (e.g. just silence)
- GlobalRotationToLocal and LocalRotationToGlobal rotation inputs now default to identity quaternion, rather than invalid one (reported by @H3BO3)
- WorldOrb labels will now always point at the local user, rather than nearest (based on report by @H3BO3)
- Dynamic Variables are now immediatelly linked to the space upon creation, preventing issues where multiple instances are created (reported by @Robyn (QueenHidi) and @Zyzyl)

- Merged Japanese, Chinese and Esperanto locale updates by @Melnus
- Merged Russian locale updates by @Shadow Panther

[h2]Security:[/h2]
- Silence now persists for user across rejoins to the same session (but not session restarts) (based on report by @Turk)
- Improved Silence security, to avoid users unsilencing themselves, even if they have permission to silence/unsilence other users (based on report by @Rukio)

[h2]Bugfixes:[/h2]
- Fixed voice messages sometimes not playing (reported by @Veer | Chief Moderation Officer)
- Added extra error checking to audio clip playback to provide more diagnostic information during errors and to break out of potential infinite loops (based on report by @Rukio)
-- This doesn't fix the underling problem, but should prevent Neos from freezing and only cause audio glitches instead

Neos now supports sending voice messages through the chat system! :smile: Simply press and hold the record button to send one. The UI is a bit ugly since it's still using the old Friends UI (it's going to be reworked in the future), but functionality should be there.

It also supports procedural audio clips now! You can use these to make simple audio clips right inside of Neos. I've added a few so far, but there's possibility for many more (and a few more that I already plan to add).

There's also a bunch of new components for easily setting up playback UI, seeking and so on, they're used for the voice messages UI internally, but are general and will work with any playback (audio, animations, video)!

Some other tweaks and fixes as usual too!

[h2]New Features:[/h2]
- Added ability to send Voice messages using the in-game chat
-- Simply press and hold the red circle "Rec" button on the right of the message text field in the Friends dialog and record your message
-- Voice messages count against storage quota, but they're heavily compressed with OGG Vorbis, so are usually just a few dozen kB
- Added system for procedural audio clips, which also supports "Bake Audio Clip" option, which will produce a regular audio file from the procedural one
- Added procedural audio clips (under Assets / Procedural Audio Clips) with configurable frequency & amplitude (if it's above 1, it'l be clipped)
-- SineWaveClip
-- SquareWaveClip
-- SawtoothWaveClip
-- SimplexNoiseClip (has configurable duration (max 1 minute) and offset)
- Added ButtonPlaybackAction (under Common UI/Button Interactions) which allows easily setting up playback actions on button press/hover actions
-- You can set action for button press/release and hover enter/leave events
-- Available actions are: None, Play, Resume, Pause, Stop, ToggleResumePause, TogglePlayStop
-- You can reference either playable component (e.g. AudioClipPlayer) or directly the Playback field
- Added ButtonPlaybackSeeker (under Common UI/Button Interactions) which will seek a target playable based on the normalized press position
-- You can toggle whether seeking is Continuous or not (continous keeps seeking as long as you hold the button)
-- You can also toggle it to use vertical axis instead of horizontal
- Added PlaybackPositionDriver (under Media/Utility) which will drive a float field based on current playback position of source playable
-- It can drive either using Normalized Position (default) or absolute
- Added PlaybackStateDriver, which drives boolean fields based on IsPlaying, IsPaused and IsStopped state
- Added AudioMetadata component, which is now attached and filled on audio clips recorded with the Microphone tip (requested by @Shifty | Quality Control Lead and @Alex from Alaska)
-- It shares relevant fields with the PhotoMetadata component, such us world where the clip was taken, user, timestamp, present users and so on

[h2]Tweaks:[/h2]
- Removed old never-implemented AudioRecorder component (based on report by @Alex from Alaska)
- Improved type name beautification to avoid putting spaces between numbers and capital letters (e.g. "AttachTexture2D" beautifying to "Attach Texture 2D" instead of "Attach Texture2 D") (based on feedback from @art0007i)
- Playback Resume will now start playing again from the beginning if the clip has reached the end

- Upgraded Oculus Plugin to 1.52.1 from 1.51.0
-- I've also decided to upgrade the OVRLipSync library as well given the changes made to how is Viseme analysis computed. Please let us know if you start running into sudden high CPU usage again, but I think it should be good now.

- Merged Spanish locale additions by @Ruzert
- Merged Swedish locale additions by @Coffee | Programmer
- Merged Japanese, Chinese and Esperanto locale additions by @Melnus

[h2]Bugfixes:[/h2]
- Fixed Logix Input nodes not updating text field with actual parsed value when the value is dropped

New additions, tweaks and bugfixes for the new audio system! You can now easily make audio clips loopable fully in-game! Lots of fixes for stutters, freezes, ends of clips getting cut after encoding, corrupted encoding, messed up playback state and more were fixed too!

Security was improved for the Microphone Tip, so it can't be abused to record people in whisper bubbles (you can still record yoruself though). There are a few new LogiX nodes as well and other general bugfixes and tweaks.

[h2]New Features:[/h2]
- Added "Make Loopable (using crossfade)" processing option to StaticAudioClip, which allows easily making clips loop well by crossfading the end to the beginning (based on request by @Aegis_Wolf | Art Director)
-- This uses the position/duration field to determine the length of the crossfade
-- The resulting clip is shortened by the crossfade duration. Using longer clips can produce better results.
- Inspector now also shows number of samples in the audio clip
- Added support for Quad channel configuration to the new AudioX system
-- This fixes certain microphones not working due to only having 4 channel configuration available (e.g. Kinect microphone as reported by @art0007i)
- Added nodes to reinterpret floating point values as integers and vice versa (requested by @Coffee | Programmer and @Ukilop)
-- This can be used to extract/compose them as bits
-- They can be found in Math/Binary
-- HalfAsUShort, FloatAsUInt, DoubleAsULong, UShortAsHalf, UIntAsFloat and ULongAsDouble

[h2]Neos Classroom changes:[/h2]
- Added a board, which will automatically load any pictures selected from the database, without having to unload the 3D model (on desktop the images can also be navigated using the arrows on keyboard) (requested by @Karel | CEO on behalf of VR Lab)

[h2]Tweaks:[/h2]
- Security improvements for the Microphone Tip
-- When triggered with Tooltip Simulator, it will only record the user if their voice is set to Normal and above
--- This fixes being able to record users in whisper bubbles (reported by @Zyzyl and @Elektrospy)
--- If you start recording yourself, the tool will record you even in whisper bubble
--- However note that with whisper bubbles your voice data is still streamed over network, use private sessions for sensitive conversations!
--- When muted, the voice data is blocked at Neos input driver level and cannot be recorded in any way (this protection was already in place, just reiterating for clarification)
- Limited IcosphereSubdivisions to 8, to prevent generating excessive amounts of geometry (suggested by @Epsilion)

- Merged Polish locale additions by @art0007i
- Merged Russian locale additions and fixes by @Shadow Panther

[h2]Bugfixes:[/h2]
- SyncPlayback now filters invalid values (NaN and Infinity) being assigned to Speed or Position, preventing the internal state of things getting corrupted (discovered by @Kal)
- Fixed DrivePlaybackNode not respecting the looping parameter when computing playback position error, causing time jumps around the looping point
- Fixed opened inspector for playback constantly overriding the playback speed, preventing it from being set using other means (e.g. Set Speed node) while the inspector window is open (based on report by @Dante and @Aegis_Wolf | Art Director)
- Added validation to Vorbis encoder settings, preventing things from breaking when invalid Quality setting is used (e.g. negative quality on MicrophoneTip as discovered by @Ryuvi and @SmolCookie, reported by @Shifty | Quality Control Lead)
- Fixed "Position/Duration" field in the inspector being clamped to 1
- Fixed in-memory encoding cutting of last chunk of bytes for some formats (particularly OGG Vorbis), resulting in incomplete or corrupted file
- When processing OGG Vorbis audio clips that are shorter than 2 seconds or use samplerate lower than 32 kHz, Neos will encode them as WAV now to work around bugs in the Vorbis encoder producing corrupt files and sample length imprecision
- Fixed AudioBufferPool recycle queue growing to enormous sizes in some cases, causing stutters, long freezes and out of memory errrors (reported by @Shifty | Quality Control Lead on behalf of @Rukio)
- Fixed plugin postprocessing failing when ReleaseNotes.txt file is missing (reported by @Epsilion)

Welcome to another weekly update!

As part of the optimizations, we have decided to rewrite Neos’ audio system, since it was one of the old and hacky pieces of the codebase that needed to be reworked for a while. With the new system not only the performance is improved, but it brings lots of new audio-related functionality as well!

Neos now supports playing audio clips at various speeds (including reverse), changing their pitch as a result and allowing for a wide range of audio effects. It supports basic audio editing as well, allowing you to quickly normalize volume, trim sounds or add fade in and fade out effects without leaving Neos.

The new audio system supports encoding as well (currently WAV, OGG Vorbis and FLAC) which enables the addition of a new Microphone tool, which lets you record audio clips directly in-game. Some long-standing bugs were fixed too, like short audio clips not always playing.



[h2]Audio Playback Speed / Pitch shifting[/h2]
Thanks to the new and improved audio system, built around a new class called AudioX for handling audio data, Neos now supports playing any audio clips at arbitrary playback rates. Playing the audio clip faster will result in increased pitch, while playing it slower will lower the pitch. Using a negative number the clip can now also play in reverse.

This allows for a wide variety of effects. The ability is exposed through the inspector, as well as through LogiX. For example using PlayOneShot you can change the pitch of played sound effects to create musical instruments or to provide random variations.

Using the DrivePlayback node, you can smoothly vary the playback speed of the clip to create effects like a car engine speeding up or slowing down. To achieve this, the node now has two new inputs - Speed and MaximumPlaybackError.

The MaximumPlaybackError can be particularly important, as it determines if the node will force a playback position to resync each update. By increasing the value you can give a small tolerance for error, but still ensuring the playback stays synchronized. By setting the allowed error to a really large value (e.g. infinity), it will never re-sync (unless an impulse is sent to force it), preventing any playback jumps.

For sound effects like the revving of an engine the exact playback position might not be important so it’s a good choice.

[previewyoutube][/previewyoutube]

[h2]Basic Audio Editing[/h2]
Neos has allowed basic editing/filtering for meshes and textures for a while, but not for sounds. Thanks to the new AudioX system we have full control over the audio data and now expose basic editing functionality on the StaticAudioClip component in the inspector.

[previewyoutube][/previewyoutube]

You can now normalize the audio volume if the clip is too quiet, trim a fixed duration from the beginning or end of the clip, trim silence at the beginning/end of the clip and add a volume fade in or fade out effect. You also have a choice to convert the clip into WAV, OGG Vorbis and FLAC.

This should help reduce the need to use external software for a lot of common tasks, with a lot more features to come in the future (and at some point, we plan to have full non-linear audio editing support, but that’s longer off right now).

[previewyoutube][/previewyoutube]

[h2]Audio Recording & Stereo streams[/h2]
Thanks to the AudioX being able to encode any audio data, we were able to add a new Microphone tool, which you can now find in your Neos Essentials. Using this tool you can record your voice and have it immediately spawn as an audio clip in the world.

You can choose to record in WAV, OGG Vorbis or FLAC formats and either use raw audio data from your microphone or use the filtered ones (noise-gated and normalized). We hope that this tool will be useful, either for taking notes or quickly prototyping and making sound effects and voice clips.

[previewyoutube][/previewyoutube]

On related note, the audio input system now also supports stereo audio input, which is used by the microphone tool. The audio synchronization streams support it as well, but voice is still mono by default, as stereo isn’t needed in most cases and will save some bandwidth (it might change if the bandwidth increase would be negligible).

[h2]Improved audio performance[/h2]
The new audio system offers several improvements in performance. It uses modern C# features like Span, which lets it efficiently reinterpret audio buffers and handle multi-channel (e.g. stereo) buffers using vectorization. Thanks for this new framework many redundant copies and conversions were eliminated.

The system also supports directly reading audio data at varying rates, which is used both for pitch shifting and re-sampling on the fly in one operation, together with conversion from mono to stereo and vice versa. It heavily simplifies the underlying code as well, reducing bugs and making it easier to add new features.

Another performance related addition is an Audio Buffer Pool system. For any audio clips that are decoded on the fly, the decoded chunks are cached in a memory pool, to avoid the need to re-decode them every time. This can help particularly in cases when a compressed audio clip is played multiple times or when it’s read multiple times (e.g. VolumeMeter).

In related fashion, Neos now also allows multiple ways of loading an audio clip. It can be either fully decoded in memory, which is ideal for shorter sound effects that are played often. The clip can also be decoded on the fly, either from the file on the disk or from memory. This is useful for anything that’s longer, as completely decoding the clip would take a lot of memory, e.g. ambience, music or voice.

By default the choice is automatic, which currently fully decodes anything shorter than 10 seconds and streams from file for everything else, but you can override this behavior if needed. When the clip is fully decoded, Neos can also resample it to the system's output sample rate, which speeds up the reads at the 1.0 playback speed, since they can just copy the raw data.

[h2]Fixed audio issues[/h2]
Reworking the old error-prone code also let us fix some of the long-standing issues. Particularly the issue with short audio clips not playing after they were first played. This actually affected all audio clips, due to the first few hundred samples being skipped. Now all clips should play perfectly every time.

The real-time synchronization streams were reworked and tweaked as well, to better decide when to start playing back data from the circular buffer (filled from the decoded data transmitted over the network). This should help reduce voice stutters due to network jitter and also eliminate some artifacts, where a small chunk of audio data would get skipped.

[h2]Community Highlights[/h2]
[h3]Chao Garden by Sykes[/h3]
Sykes, one of our newest moderators, has recreated the Chao Garden from Sonic Adventure 2! This build was inspired by the game and fully remade in blender, utilizing custom assets and was previously made for Second Life. Come and give it a visit!



[h3]Hallwyl Museum by Enverex[/h3]
The Hallwyl Museum’s photogrammetry model has been repurposed as a world in Neos! The doors act as teleporters so you can visit the many bright and beautiful rooms of this magnificent place. Photos taken in here look almost like real world photographs of the actual museum, so bring a camera and a friend!



------------------------------------------------------

Anyway, that's all for this week! Thank you again for reading this weekly update! All of this work is possible thanks to your help and support and we can’t wait to see what will you make with the new audio capabilities in Neos!