Neos VR News

Another chunk of optimization is done! The system that runs the moved hierarchy events has been reworked to the newer one, utilizing the more efficient hierarchy event framework originally made for dynamic variables. This should help reduce a lot of overhead and wasted cycles, particularly with avatars and worlds where there large parts of hierarchy disabled.

There might be some unexpected side effects of this change. We did some thorough testing and things seem to be running fine, but if you notice anything broken, please let us know!

Also some UIX bugfixes, fixes for the blendshape splitting, locale updates and more!

[h2]Optimizations:[/h2]
- Reworked the Slot Moved Event system to utilize the new hierarchy event system (originally made for the dynamic variable system)
-- This decouples the moved events from other events like Activation or slot changes, preventing the moved events from having to propagate to disabled parts of hierarchy in many cases (this should help with avatars that have a lot of local stuff on the avatar)
-- It also makes propagation a lot more efficient in cases where there are a lot of children under a single parent, but only some of them are listening on moved events
-- It also provides more elegant framework for these events and paves way to more planned future optimizations
- Decoupled Slot world transform events from slot changed events, reducing wasted cycles and unecessary registrations for moved events even when not needed
-- ReferenceNode will now register world moved events only when it's active. As a result it won't cause unecessary LogiX updates on users where it's on locally deactivated parts of hierarchy
- Added event cleanup to relay references on disposal, preventing potential memory leaks and wasted CPU cycles

[h2]Tweaks:[/h2]
- Reworked scheduling of rendering to texture to run after Neos' update loop, rather than immediatelly
-- This could potentially fix some rendering issues and artifacts
- Removed obsolete Menu anchor from avatar creator (reported by @Shifty | Quality Control Lead)

- Merged Spanish locale additions by @Skywind Kitsune
- Merged Japanese locale additions by @Melnus
- Merged German locale additions by @Elektrospy

[h2]Bugfixes:[/h2]
- Fixed UIX GraphicChunk assignement/removal not properly invalidating the layout, resulting in corrupt visuals (e.g. when toggling Mask on and off) (based on relayed report through @Karel | CEO for Neos Classroom)
- Fixed UIX GraphicChunks not getting hidden and disabled correctly in some scenarios when part of hierarchy is disabled
-- This fixes UIX getting corrupted reported by @SmolCookie and @Shifty | Quality Control Lead
- Fixed splitting mesh by blendshapes breaking if the mesh has no blendshapes or if all vertices are affected by blendshapes
- Fixed splitting mesh by blendshapes incorrectly assigning materials to the split parts in some cases (reported by @Shifty | Quality Control Lead)
- Fixed world browser showing incompatible sessions fetched through the contacts (reported by @Gunnar0 and investigated by @Shifty | Quality Control Lead)
- Fixed Interactive camera control Focus visual blocking the camera control buttons underneath (reported by @MirPASEC)
- GrabbableReceiverSurface will now cancel the animation when the object is grabbed or otherwise parented (based on report from @EvawdnousCimota#2730)
- Updated old version of cloud background worker to newer one, fixing issue where storage quota was calculated incorrectly for translators (reported by @Beaned, @jeana through @Shifty | Quality Control Lead)

Pushing a patch to fix broken French locale and a few bonus things! I've added a validator to the build process which checks the locales and catches this error early next time and as a bonus also generates statistics on completeness of each locale and list of missing strings.

The LocaleLoading process is more robust too and will try to load fallback locale if something is wrong with it. Also some tweaks and improvements to the blendshape mesh splitting.

Compatible with current build.

[h2]New Features:[/h2]
- Added locale validation to Neos build process, which will immediatelly alert when a locale file is corrupted
- Added locale statistics generation, which will now update the README.md file with % completeness of each locale as well as generate MISSING.md file which contains list of all keys missing in each locale

[h2]Tweaks:[/h2]
- Splitting skinned mesh to part unaffected by blendshapes will now also strip unecessary bones from the mesh
- Splitting skinned mesh to part unaffected by blendshapes now also setups SimpleAwayIndicator and SimpleAvatarProtection if they're present (based on report by @Gunnar0 and @H3BO3)
- Removed obsolete "Calibrate Hips" and "Calibrate Feet" from Avatar Creator (based on feedback by @Nexulan | Community Manager, @DeliriousJax and others)

- Merged fixed version of French locale by @Archer (reported by @Nicoreda, @H3BO3, @Blaze and others)
- Merged new Japanese, Esperanto and Chinese locale strings by @Melnus
- Added new Czech locale strings

[h2]Bugfixes:[/h2]
- Added an exception guard to LocaleResource loading, which will skip loading of corrupted locales and load the fallbacks instead
- Fixed incorrect Spanish locale code

Some new optimization features for skinned meshes (also regular meshes! If you have a geometry heavy mesh, you can now automatically split it into two parts - one that's affected by blendshapes and the rest that's not. If only small portion (e.g. the face) is affected by blendshapes, this can give a significant performance boost. It comes at cost of extra drawcall(s), but for geometry dense avatars it will result in a net-positive impact.

To optimize your avatar, it's recommended to do following:
1a) Bake all separate meshes into one first where possible
1b) Use "Merge submeshes with same material" to reduce number of submeshes if you're using same material for multiple
2) Run ""Separate parts of mesh unaffected by blendshapes" on it. If the resulting non-blendshape part has much more geometry than the part affected by blendshapes, I recommend using this one. If not, just use the regular one and don't split it

Neos is now also available in Russian and Norwegian thanks to community efforts! There are also some initial Spanish localization strings, but right now it's incomplete. Some improvements for biped rig detection were made too as well as some bugfixes.

[h2]New Features:[/h2]
- Added "Separate parts of mesh unaffected by blendshapes" to SkinnedMeshRenderer
-- This will split the mesh into two parts - parts of geometry affected by any blendshape and parts without any blendshapes
-- This can be used to optimize geometry-heavy avatars which have only parts of them affected by blendshapes, by reducing how much geometry needs to be morphed by the blendshapes (based on report by @Reactant)
- Added "Merge submeshes with same material" to (Skinned)MeshRenderer
-- This will merge submeshes which share the same material on the renderer, reducing number of necessary drawcalls
- "Split into separate meshes by material" now works with skinned meshes (requested by @Turk)
- Added new SkinnedMeshRenderer bounds mode - Proxy
-- This uses local bounds of another skinned mesh renderer, referenced by the ProxyBoundsSource field
-- This is very efficient method (no per-frame calculations), useful in cases you have a big main mesh and you need the small meshes to be linked in visibility with it
-- For this to work correctly, the proxy and the source must be at the same position, orientation and scale in hierarchy

- Added Russian locale by @GhostZ36 and @BrotiwkA
- Added Norwegian locale by @RΞBOЯИ
- Added initial Spanish locale strings by @Skywind Kitsune

[h2]Tweaks:[/h2]
- Improved BipedRig detection heuristics, so it now accepts "feet" for the foot bone (based on report by @Dante)
- Improved search case matching on the backend - it will now check if records contain capitalized version of the search term (based on feedback by @ProbablePrime and others)
-- E.g. searching "avatar" will now match worlds with "Avatar" tag
- MeshRenderer inspector actions are now localizable
- Panner components now have default Repeat range of 1 (suggested by @H3BO3 and @Enverex)
-- This only affects new instances, existing ones are unaffected

- Merged Japanese, Esperanto and Chinese locale updates by @Melnus
- Merged German locale updates by @Elektrospy
- Merged French locale updates by @Archer

[h2]Bugfixes:[/h2]
- Added more diagnostics for errors in streams
- Fixed MeshX submesh splitting not correctly handling vertex colors
- Wrapped clipboard operations in Neosmon with exception guards, so any problems in them don't cause Neos to lock up (particularly when using Neos with Proton, reported by TiagoTiago on GitHub)
- Fixed VRIKAvatar not clearing internal _locomotionController when dequipped, causing incorrect behavior on the dequipped avatar (reported by @Hayden (PolyLogiX - ZyroDesign))
- Fixed ValueGradientDriver getting internally out of order when points are added or removed in some cases and producing invalid values (reported by @Earthmark and @A Monsoon of Babies)
- Fixed being able to undo avatar grabbing after the avatar is equipped, causing it to break (reported by @Hayden (PolyLogiX - ZyroDesign))

Hello and welcome to another weekly update!

This week we have started our work on optimizations, focusing on the low hanging (but still delicious) fruit first. We’ve optimized the Skinned mesh rendering, Viseme Analysis and some other smaller things, getting some pretty decent first performance boosts as a result (e.g. 25 % framerate increase with stress test using many avatars), with a lot more to come.

The Metaverse Maker Competition is also nearing the end and with everyone finishing their submissions we want to make sure you tag your submission properly, so it doesn’t get missed. Details on how are below, make sure to double check if your submission is submitted properly!

Neos is now also available in more languages thanks to our amazing community! Neos is now also available in Finnish (thanks to AdmiralSoap and Napo), French (thanks to Archer and FreakyWaves) and Dutch (thanks to jeana and Beaned).



[h2]Metaverse Maker Competition nearing the end[/h2]
The community-held creative contest by Creator Jam only has a few days left before its end. If you’re participating, make sure you submit your entry before Oct 1, 14:00 EST(UTC 18:00) Thursday. It is recommended to submit early, as you can make updates before the end. However updating the entry after the cutoff is not allowed until the judging finishes.

Also importantly, your entry must have the “MMC” tag so it is properly categorized and filtered, as well as a subcategory tag that’s either “World”, “Avatar” or “Other”. If you’re not sure how to tag your entry, check the following video by Medra:
[previewyoutube][/previewyoutube]

Once the entries are submitted, the timeline is as follows:

Oct 2-4 Audience voting inside Neos.
Oct 4-9 Frooxius, 0utsider, Medra, Jak Wilmot, and Kent Bye will judge the entries.
Oct 10 Winners will be announced live in Neos via twitch stream.

The voting is going to be done through Neos itself, using a voting system created by ProbablePrime. If you’d like to learn more about it, check out his video:
[previewyoutube][/previewyoutube]

We’ll be also adding an official “MMC” category in the “Worlds” menu in Neos in the next build, so you can use it to double check if your entry is properly listed! We wish everyone good luck in the last few days and we can’t wait to see all the amazing stuff you have built during the past month!

[h2]Skinned Mesh Renderer optimizations & culling modes[/h2]
We have started profiling Neos more heavily with various tools in populated sessions to gather information about how different parts of the engine perform and where the bottlenecks lie. We have confirmed many of the known causes of performance issues and found some new ones.

[previewyoutube][/previewyoutube]

One of the big ones that was quick to address were Skinned Mesh Renderers, particularly how they update their bounds and how they’re culled. Neos now supports 4 different modes for updating the bounding boxes:

• Static - bounds are based on the mesh alone and do not update. This is the new default as it has no overhead and works well in most cases
  
• Fast Disjoint Root Approximate - this mode pre-computes approximate spherical bounds for groups of bones (all overlapping bones get merged into a single root one) which are then used to compute the approximate bounding box. This is typically very fast, as most meshes have just one to three bone groups. It produces bounds that are bigger than needed to be, but still offer good culling.
  
• Medium Per-Bone Approximate - this mode uses bounding boxes of each bone of the mesh. This provides more accurate bounds without actually transforming the geometry, but with bone-dense meshes will take a lot more performance.
  
• Slow Realtime Accurate - this actually transforms the geometry every frame and uses that determine visibility. This is significantly slow, but most accurate, taking into account blendshapes as well. Recommended only when absolutely needed.
  

This provides much more flexibility now, rather than a constant forced slow method. Most meshes do well with the “Static” method, with “Fast Disjoint Root Approximate” providing much faster fallback in cases where the meshes get incorrectly culled.

When you’re wearing an avatar, it will automatically use “Fast Disjoint Root Approximate” for yourself, but you might need to switch to it explicitly if other users see parts of your avatar popping off.

However if possible, we strongly recommend baking your avatar meshes into a single one, which will provide a combination of best performance and culling behavior. The in-game Glue tooltip has “Skinned Mesh” baking mode, which preserves and intelligently combines all the bones and blendshapes of your mesh and allows easy way to optimize your avatars without leaving VR.

With this optimization, we have noticed about a 25 % framerate increase in a world with many avatars.

https://www.youtube.com/watch?v=w-NhlyhfVBM
Some behind the scenes of development of these new algorithms

[h2]Viseme Analyzer[/h2]
Another of the low hanging fruits, but significant contributors to performance issues was the VisemeAnalyzer component, which uses the OVRLipSync library internally. Not only is the viseme analysis quite expensive (roughly 1 ms per user per frame), but it’s not properly multithreaded.

This has caused internal thread starvation, as Neos schedules heavy work on its pool of background threads, taking lots of CPU time away from useful work on other background tasks.

To fix this, we have moved the analysis to its own queue with a single background thread to avoid it from starving the thread pool and also importantly added mechanism where visemes from other users are now streamed over network, with your system only responsible for computing your own and any others not associated with an use (e.g. using it for music visualization).

These changes will heavily reduce the CPU usage in populated worlds, freeing up more system resources for other tasks and reduce occurrences of pathological behavior, where UI and other things will start locking up due to not having any background threads to run on.

[h2]Value Streams and others[/h2]
Some other related optimizations involved reducing various overheads. Notably we introduced MultiValueStream, which can stream multiple values with the same settings, rather than using individual ValueStream for each.

This helps significantly for things like hands (or the visemes above), which contain many bones and save some CPU and memory usage, while producing the same results as before.

Many other parts of the engine and its internals have received small optimizations as well, to read more about them, check out the #neos-updates channel on our official Discord or the release notes on Steam.

[h2]Collider User Tracker, Silencing users and more[/h2]
While the focus is on optimizations, we’re adding new features as well. Some of them are designed to allow you better optimize your own creations as well, such as the new ColliderUserTracker component.

This component tracks users inside a collider (setup as a trigger), automatically providing information if any or the local users are inside. This allows you to easily setup culling zones or build things like automatic doors and other behaviors.

Another addition is the ability to Silence users, which helps with self-moderation. Unlike Mute, it will Silence the user for everyone in the session, which can be helpful in many cases, for example users with significant echo. For list of other new features, check out the update notes on Steam or our official Discord.

[h2]Community September Showcase[/h2]
As usual, the last stream of the month is dedicated to you, our community and all the amazing things you have built. If you missed the Friday stream, you can watch the archive here. We take a look at few of the already published MMC entries and other creations:

[previewyoutube][/previewyoutube]

[h2]What’s next?[/h2]
We’re going to be focusing on more optimizations in the upcoming week based on the profiling data, gradually improving performance of Neos as we go.

We have a few smaller, but impactful optimizations planned, like revamping the moved event system, but we’ll be transitioning to bigger architectural reworks and improvements, which will help unlock significant performance gains and implement new optimization features.

To get the latest info, you can keep an eye on the #devlog channel on our official Discord. Thank you for reading and as always, thank you for supporting Neos and making it better every day!

Some more optimizations and related functionality! Skinned Meshes now have new modes for updating bounds, which should help fix the culling issues with some avatars without taking too much performance. If you're experiencing issues with parts of your avatar disappearing at certain angles, do following:

- If your avatar is in multiple pieces, try baking the meshes with the Glue tooltip first if possible. This is optimal for performance and allow the avatar to share a single big bounding box.
- If you can't bake it (e.g. you swap pieces in realtime), switch the affected meshes to "FastDisjointRootApproximate" method. This has an extra performance impact, but is much lower than the SlowRealtimeAccurate method (the previous default)

I've also made changes to Viseme analysis, which is surpringly CPU costly (~1 ms per user per frame) and the internal OVRLipSync library locking was source of major background thread starvation (blocking other computations) with multiple people in the world. Visemes are now computed on each user and streamed to others. There is no explicit mechanism to sync it with voice, but both voice and viseme data are sent as soon as possible and from internal testing this achieves really good latency.

There are some smaller optimizations, like MultiValueStream which significantly reduces overhead of multiple ValueStream objects (network impact is unchanged right now) and more things, I'm going after the most impactful low hanging fruits now and working way to larger reworks and refactoring. More to come!

Also Neos is now available in Dutch thanks to @jeana and @Beaned! I've got the dash and other things to work on Android again, Android/Quest builds incoming!

[h2]New Features:[/h2]
- SkinnedMeshRenderer now has 4 methods to compute bounds and handle culling (big thanks to @DeliriousJax for providing avatar sample for experimentation!):
-- Static (default) - fastest, based on the mesh alone, no realtime computations
-- FastDisjointRootApproximate - merges all bones into disjoint groups (any overlapping bones are merged into a single one) to reduce overall number of bones and uses those to approximate bounds in realtime. Fastest, recommended if part of mesh gets culled
-- MediumPerBoneApproximate - computes mesh bounds from bounds of every single bone. More accurate, but also much slower
-- SlowRealtimeAccurate - uses actual transformed geometry, requiring the skinned mesh to be processed all the time. Very heavy, but will respect things like blendshapes in addition to bones
-- NOTE: When the renderer is in user's hierarchy and set to Static, it'll use FastDisjointRootApproximate method internally, so existing avatars work without changes and don't get culled
- Added "Strip Empty Bones" on SkinnedMeshRenderer, which will remove all empty bones from the mesh
-- You can use this to optimize your meshes/rigs, reducing their CPU, memory and some GPU overhead
-- This only removes them from the mesh, the actual objects representing the bones are still kept in the hierarchy, so it should be generally safe to run this optimization
- Added "Visualize bone bounding boxes" to SkinnedMeshRenderer, which will visualize boudning boxes for each bone affecting the mesh (this includes all vertices that given bone affects)
-- Also added "Remove bone bounding box visuals" to quickly clear them
-- The visuals are non-persitent so won't save
- Added "Visualize approximate merged bone sphere bounds" which will visualize merged bone bounding spheres, representing a fast approximation of disjoint areas of influence of the rig
- Static Mesh "Get Bone List" now prints how many vertices are affected by each bone and shows bones with no bound vertices as red

- Added Dutch locale by @jeana and @Beaned
- Added pluralizer for Polish (requested by @art0007i)

[h2]Optimizations:[/h2]
- Added MultiValueStream, which handles multiple values in a single stream rather than using stream-per-value, significantly reducing associated CPU and memory overhead
-- Switched finger pose estimation to use a single stream for all fingers, rather than individual ValueStream for each bone
- VisemeAnalyzer now streams results of analysis to other users for voice input, rather than having each user compute it on their end independently
-- This should greatly improve CPU load in heavily populated worlds, as analysis takes roughly 1 ms per user per frame (it is done on background thread though, so this will free up lots of background processing resources)
-- As a bonus, Linux users will now see visemes of all other non-Linux users. Windows users will see visemes of all Linux users, because the system will compute them locally
-- Other audio sources (e.g. using it for visualizing music) are still computed locally
-- Currently there's no explicit synchronization mechanism with voice, but the streams are sent out as soon as possible, so they should generally match up close enough (possibly even improving visual latency of the analysis). Let me know if there are issues though
- Fixed VisemeAnalysis causing severe background thread starvation with multiple scheduled analysis due to internal locking mechanism of the OVRlipsync library (this should help some cases of things getting stuck and taking a while to load)
- Reduced CPU and memory usage for LogiX display nodes when they're packed
- Fixed dash projection geometry still being rendered even if it's closed
- Reduced duplicate work with explicit stream encoding (also fixing bug where forced explicit update points wouldn't encode correctly)
- Small low level optimizations when passing in LocaleString values

[h2]Tweaks:[/h2]
- Implemented system to embed the UI locale files in the Android builds for Quest build
- SkinnedMeshRenderer inspector actions are now localizable
- BipedRig inspector actions are now localizable
- Upgraded to Unity 2019.4.11f1 (form 2019.4.10f1)
- Merged French locale new strings by @FreakyWaves Music
- Merged Finnish locale updates by @Nammi and @AdmiralSoap
- Merged "Friends -> Contacts" replacements in English locale by @Alex from Alaska
- Merged "Silence" string localization to Japanese, Esperanto and Chinese by @Melnus
- Merged German locale updates by @Elektrospy

[h2]Bugfixes:[/h2]
- Fixed UserPermissions being misconfigured in some cases, allowing Guest roles to Silence users (reported by @Zane)
- Fixed DynamicReferenceVariableReset having invalid type check, not allowing any proper types (reported by @Earthmark)
- Fixed "Strip Empty Blendshapes" no SkinnedMeshRenderer not releasing read lock on the mesh data
- Fixed "Switch Lomotion Module" not using the locale key when "Exact Match" is enabled (reported by @Shifty | Quality Control Lead)
- Fixed internal UVSphereCapsule procedural generator not setting up triangle correctly when there's pre-existing geometry in the mesh
- Fixed bounding sphere encapsulation producing incorrect result when a smaller sphere is fully contained within another one
- Fixed sending items through messaging system not respecting the item save permissions (reported by @Dante)