Neos VR News

Hello everyone and welcome to another weekly update!

This one is going to be relatively short, as we were busy with the VBLFC over past week, the first official online convention held on our platform. It has broken the record number of concurrent users, reaching 432 peak on Steam.

We also look at some of the optimizations and effects on the cloud infrastructure with the sudden surge in users and how they helped to keep the cloud stable and make it more efficient in long term.

The creator festival Neos Festa is also coming next month for its third installment, held by the Japanese team. If you'd like to have an exhibit, you can find more information at festa.neos.com!



[h2]VBLFC[/h2]
During the last Friday through Sunday, Neos saw its official online convention, the Virtual Biggest Little Furry Convention. This event brought hundreds of new users and broke our all time record for number of concurrent users! We reached 432 peak concurrent users on Steam, with our metrics going slightly above 500 with all accounts considered (including headless).



We’d like to extend a huge thanks to the organizers for choosing Neos, building beautiful worlds and a number of interacted gadgets and accessories for the attendants of the event and bringing a number of guests to serve as entertainment for the program. So again, big thanks to everyone for their hard work making this event happen and thanks for everyone participating!



VBLFC plans to reopen the worlds during the physical convention in October, so if you missed them, you’ll be able to revisit them again! The VBLFC suite world is open to the public already and we’ve already seen many people decorate it and make their own versions of it, so feel free if you haven’t!





[previewyoutube][/previewyoutube]

[h2]Analyzing Cloud Usage & Further Optimizations[/h2]
As many people started joining during the VBLFC Opening Ceremony, we started noticing the cloud response times sharply rising up due to the sudden increase of users. This has resulted in noticeable slowness of much of the cloud functionality.

To compensate, we started scaling up (and out) the infrastructure beyond our default limits, which brought it down to usable levels, but still far from ideal. Based on the live metrics, we began optimizing the biggest culprits that resulted in slow responses and releasing them to the cloud as the event was ongoing.

Overall response time during the first night. The sharp increase is when people join in, the decrease is when the scaling up kicks in. Afterwards the drops and smoothing out are due to optimizations being deployed.

The first bits addressed some quick low hanging fruit, like unnecessary database queries and adding some bits of extra caching. This has smoothed out some of the response times, but still wasn’t sufficient to make it snappy.

Following that, we implemented a quick optimization for the user status update system - the one responsible for updating status of your contacts - whether they’re online, which world they’re in and so on. This system was the second biggest contributor in the cloud load after the messaging system, so it was a prime target for the optimization.

Once this update was pushed out, to our surprise the response times have dropped more than we expected them to, under 50 ms on average, despite about 500 people still being on. This resulted in better response times than before the sudden increase in active users, at the cost of some precision - status of new contacts might take a few minutes to show up.

Impact of the status system optimization

Thanks to this optimization the remainder of the convention was pretty smooth and without any cloud slowness or interruptions. Those optimizations still help even after the event has finished. Based on their effectiveness, we’ll be moving the status updates to SignalR next, similarly to the messaging system, which will make it significantly more responsive and improve the cloud scaling even further.

It is always a bit scary getting a sharp increase in users, as it doesn’t give much room for the infrastructure to adjust, but this event has given us a wealth of information on how it behaves with a quickly rising user base and which parts become bottlenecks and need to be redesigned.

With this, we’ll be better prepared for other big events in the future and just general growth of the Neos community, at least from the cloud perspective.

[h2]New Neos Hands[/h2]
Next time you start Neos, you'll see the update I just pushed for the new default avatar. These new hands were designed to match the headset, and the headset itself was updated with a new Neos logo on the back.

Every time you apply the default avatar, you are normally assigned a random color. However, now you can choose to assign yourself a persistent color that can be used beyond your default avatar using the cloud variable G-Neos.CustomUserColor! Please note that this cloud variable's name and function is subject to change.

The whisper bubble's visuals have also been re-imagined as a ring, to be more light and less isolating.



[h2]Community Highlights[/h2]
Hello everyone, friendly neighborhood Turk here! I hope your week has been wonderful! It’s been jam packed these past 2 weeks mixed with all the stuff that’s been going on! So I hope you have been having fun with all the things going on! This month has all kinds of community events and moments going on so here’s some of them for you to see!

[h3]Firr’s New Homeworld[/h3]
VBLFC has come and gone in the community, and there were some fun times! But for one last hurrah it seems like Firr needs a new homeworld! So everyone quite literally had come into his world and trashed his room. Whether he makes this his homeworld is a thing to be seen, but boy is this headless session/world a treat. There is ALOT of memes.



[h3]Neos Festa 3[/h3]
Neos Festa 3 is soon on the way! Neos Festa is a creator showcase event, where folks can submit content and applications to showcase their works in Neos. Neos Festa has been quite the staple in our community bringing many games, toys, and avatars to the community enriching the community in ways we can’t expect. So if you want people to check our work, and see what people think feel free to submit! Find out more details on the Discord, or in the submission world. You can also find more stuff at the website festa.neos.com.

Thank you to everyone who’s collaborating to make this event possible!



[h2]What's next[/h2]
After the event, we're continuing main focus on BEPUv2 integration. We have already finalized all collider types, getting mesh colliders and convex hulls to also work with the asset variant system, precomputing and caching the data, instead of generating it on the fly every time you load into the world.

There's still quite more to do. We'll also be interweaving some more tasks and updates in between. You can always find the latest progress in the official Discord in the #devlog channel or check out the Roadmaps on GitHub.

And as always, huge thanks for everyone's support! Thanks to your passion and creativity, this place keeps growing and becoming better every day! See you next week!

Just a small build with some bugfixes and security improvements. Compatible with the previous one, so you don't need to update urgently.

[h2]New Features:[/h2]
- Added support for some Japanese MMD naming conventions (suggsted and chart provided by @AshtonSparx)
-- DirectVisemeDriver now detects あ, い, う, え, お and ん
-- Blink bendshape detection now detects まばたき, ウィンク, ウィンク右 and ウィンク２
-- AvatarExpressionDriver now detects 笑い, 敵意, にっこり, ∧, □ and ぺろっ
-- Please let me know if there are any issues with these, I'm relying on a visual chart to map these and I'm not 100 % sure I got the mapping correct

[h2]Security:[/h2]
- Fixed editor components being able to indirectly modify protected fields (based on report by @xLinka and others)
- Improved security of world session configuration
- Added extra diagnostics to changing session configuration
- FileBrowser component will automatically clean itself up in Worldspace (based on report by @xLinka)
-- Note that even when spawned in Worldspace it can't be used as it has internal userspace checks for its methods
- Added logging for permission components being removed to help diagnose cases of them being

[h2]Bugfixes:[/h2]
- Fixed headless prompt not returning after running the restart command (reported by @Glitch)
- Fixed insecure websocket connections not working on headless (reported by @Gawdl3y, @Kulza and @Epsilion)
- Fixed unread messages not being properly marked as read when there's more than 512 (reported by @Lucaedr, @Kodey and @Sox)
- Fixed developer tooltip destroying last selected object with "Destroy Selected" even after it was deselected (reported by @amber)
-- Note that if it's deselected by another user, it will still be destroyed. Dealing with this properly will require the implementation of proper selection management system

Hello everyone and welcome to another weekly update!

This week we have briefly shifted focus from BEPUv2 integration to integrating the SignalR push notification service for our messaging system and other cloud optimizations to improve responsiveness and fix a number of issues, such as message history not loading for certain users.

Thanks to this new system, the load on the cloud infrastructure has significantly dropped. This should keep the cloud responsive even during busy days with lots of people on and keep scaling as the community keeps growing.

It also opens up doors for new upcoming features and abilities, like full real time sync for cloud variables, cloud impulses and optimizing other parts of the cloud infrastructure. We also used it to improve security of the moderation system, ensuring that any account restrictions take immediate effect.

Security and moderation have received some more improvements as well, preventing some cases of ban evasion. You can now also restrict the in-game File Browser to only specific locations, or none at all.

And also in big news, the VBLFC furry con is taking place this weekend! This is the first con held officially in Neos VR and we are proud to host it on our platform. You can find more information at their Discord and Twitter or register at the website here: They have a lot of exciting events planned. We hope everyone will enjoy their time there!



[h2]VBLFC[/h2]
The Virtual BLFC con is taking place this week! This is the first con held officially in Neos VR and the biggest event to date on our platform. The organizers have prepared an awesome set of worlds and other cool things for the attendees. We are honored to host this event and we hope that everyone will have an awesome time!

[previewyoutube][/previewyoutube]

You can find more information at their official website and register for attendance. They have also made this amazing trailer for the event.



[h2]Integrating SignalR for messaging system[/h2]
As our community keeps growing, we have started noticing the cloud infrastructure being heavily taxed during the busy days, causing the cloud service to become less responsive. With some big events coming up, we decided to prioritize integration of SignalR push notification service, which was on the roadmap for a while, to improve how well the system scales and introduce some new capabilities.

The in-game messaging system has been redesigned to use this service. Instead of repeated REST (HTTPs) requests, all the communication is done via WebSockets. This is significantly more efficient for both your connection, as well as on the cloud service - the previous system was responsible for using the majority of the system’s throughput!

Thanks to the new system, the messaging system should be significantly more reliable and robust in the long term (after fixing some initial bugs and issues), decreasing the latency of the messages and cases where they wouldn’t be delivered at all or be slow to respond.

Decreased database usage after optimizations. Shows last 7 days.

The system also enables some new features, like getting message read notifications (can be disabled in settings based on community request) and in the future message writing or voice message recording notifications.

The service is also planned to be used for more as well in the future, notably updating the session status, current profile status and synchronizing cloud variables fully in real time across different clients. You can check out more at our roadmap on GitHub.

[h2]More cloud optimizations[/h2]
When working on the cloud, we found out more underlying causes of issues that were resulting in some slowness, particular slow message loading, due to the database queries being highly suboptimal. Those have now been optimized, resulting in significantly more efficient queries and much quicker loading of message history as a result.

Server usage split by endpoint. The orange line is fetching new messages. You can see the significant decrease after SignalR release, but still some large spikes. Those were addressed near the end of the grpah.

Some other parts of the cloud were tweaked and improved as well to be more robust, improving the rate limiting system with a new solution based on Redis and adding fallbacks in case of transient connectivity issues.

Overall we hope those changes will help resolve a number of issues that you might’ve been experiencing in the past, but we’ll keep monitoring the system and making more improvements as the community keeps growing.


If you run into any problems, let us know either on Discord or our GitHub!

[h2]Security and moderation improvements[/h2]
Our moderation tools have gotten some improvements as well based on some community reports and incidents. Notably we have expanded the ban evasion detection system to cover more loopholes in the system and used the SignalR service so any global bans take effect immediately.

We have also added a configuration option that allows restricting the functionality of the built-in File Browser so it can only access certain paths on your system or none at all. Simply setup an array of paths under pathWhitelist in Config.json file in the Neos installation. Providing an empty array (“pathWhitelist: [ ]”) will disable the File Browser completely.

This can be useful for additional security or when running Neos at public events, schools and other places where you don’t want the visitors/users to access all the files from within VR.

--------------------

Anyway that's all for this week! Also apologies for posting so late, this week has been a bit crazy. We've had some storms and lost power (and internet) for a while, which delayed a few things, but better late than never!

We'll see you next one, keep an eye on our official Discord for info. And as always, huge thanks for everyone for supporting the platform, without your help we wouldn't be here!

Hello everyone!
I've got a new build for you! This is mainly cloud changes, that should greatly improve the responsivness and make the new services more robust! The switch to SignalR ended up a bit bumpier than I hoped for (thanks everyone for understanding despite the inconveniences ^^;), but it should be really worth it in the long run and scale much better as we get more users.

Notably, I found root cause of message history taking really long time to load. In some cases the database queries were very suboptimal, taking several seconds of database time for a single fetch. For example loading message history I used to test consumed around 14000 RUs (Request Units, pretty much database throughput units). After adding extra field to simplify the query and adding composite index (you probably felt that yesterday, as all existing messages had to be re-indexed!), it now takes just 8 RUs!

As a result, the message history should now load super fast across the board and stop hammering the cloud, causing other things to be slow! But please let us know if you still run into some issues.

There are a bunch of improvements as well. The cloud also ran into another issue, where due to Redis server restarting, the rate limiting library wouldn't reinitialize and everything would break! This should be fixed now. The system now also has fallback in case the Redis fail temporarily and will restart itself automatically if it keeps failing.

Anyway, hopefully things should run pretty smoohtly right now. I'll be monitoring them and see if we run into some more issues. There some more things that we need to optimize as well (e.g. world searching/browsing, initial loading of contacts and bunch of others), but this should be a pretty important step forward!

Oh also this build is compatible with the last since most of the changes were in the cloud (but there should be some neat stuff in the build too)!

[h2]Tweaks:[/h2]
- Added optimized index values to messages in the database and optimized message history queries to significantly reduce (orders of magnitude) required database throughput when fetching message history
-- Message history should now load significantly faster and cause less load on the cloud backend (to give example, one fetch would previously consume ~14000 RUs, the same exact fetch now consumes 8 RUs)
-- This should also fix cases of message history not loading at all in somem cases due to the request timing out
-- Additionaly this should help with overall cloud responsiveness, thanks to message history loading not hammering the database like crazy anymore
-- All existing messages have been processed and re-indexed (this was one of the major factors for the slowness yesterday). If you notice any messages not loading or missing, please let us know!
- Improved health check for the cloud API server, to ensure the server gets rebooted in case of CosmosDB or Redis issues
-- This will auto-recover the cloud in case of unexpected connectivity issues that require reinitializing connections
- Tweaked database query batch size when fetching records, to prevent certain operations (e.g. generating record usage report) from resulting in large load on the database, causing cloud slowness
- Record usage report/JSON is now pipelined and rate limited to avoid it from potentially disrupting cloud services and to improve robustness
-- You can only run the command once every 10 minutes and max 8 times a day. This is subject to change, but it's generally not recommended to run this too often
-- The requests are processed in sequence in case of multiple requests to avoid overloading database with too many expensive tasks at once. You will be notified when the generation finishes
- Cloud messaging system now locally tracks sent status of a message
-- This ensures that message is properly colored as sending or failed to sent when the message history is refreshed
- Added in-process rate limiting fallback during Redis connectivity issues
-- This should improve robustness of the cloud services in case of interminnent connectivity issues to the Redis server
- VBLFC badge now shows up in the Session list as well @Delta and @Kulza
- Changed deadzone algorithm for axis movement to be based on magnitude, rather than each individual axis, which results in better diagonal behavior (based on report by @H3BO3)
- If you send a message to user before the history loads, the newly sent message shouldn't disappear anymore

- Merged Russian locale addition by @Shadow Panther [RU/EN, UTC+3]
- Merged Japanese locale additions and tweaks by @Aesc
- Merged Korean locale additions by Holy_Water
- Merged Chinese locale additions and tweaks by Holy_Water
- Merged Czech locale addition by @rampa_3 (UTC +1, DST UTC +2)
- Merged German locale additions and tweaks by @InnocentThief

[h2]Security:[/h2]
- Added extra obfuscation to host user's ID's (based on report by @runtime)

[h2]Bugfixes:[/h2]
- Fixed bug in FireflySoft.RateLimit library that would cause all rate limit checks to fail when the Redis server is unexpectedly rebooted, due to Lua Redis script being cleared
-- This was the primary cause of the cloud issues yesterday
-- Pull request for our fix is available here: https://github.com/bosima/FireflySoft.RateLimit/pull/7
- Fixed full message fetch rate limit not working properly, allowing the full message fetches to be executed too often
- Fixed wrong debug message on SignalR (reported by @Bitman (Neos.js Developer))
- Fixed incorrect URL when fetching records at path with access key (reported by @Bitman (Neos.js Developer))
- Smooth Turn exclusive mode will now also prevent movement when already turning, instead of just preventing turning while moving (reported by @H3BO3 and @Kaptain Krunch)

Do you guys ever get a day where you start working on a thing, then someone needs you to do another thing so you switch focus on that, you find you need to do another thing to do that, then in middle of that someone needs you to do another thing so you focus on that one and as you do another thing comes up. And you finish that one, go back to other things and more things come up and you never even get to the original set of things?

Long story short, no weekly update today, but at least here's a build with a bunch of additions and improvements for the new SignalR message delivery system to solve some initial issues and add a few options based on community feedback. Also some security improvements and bunch of other tweaks and additions! :smile:

[h2]New Features:[/h2]
- Added "Don't send realtime message read status" setting which allows disabling sending message read status to the other user in realtime (requested by @Yellow, @Electronus and @H3BO3)
-- Note that they will still be able to eventually tell that you read them at some point since it's updated in the database, but the status won't change for them the moment you open them
-- When enabled, you don't see any read status on your end either (meaning all messages stay yellow)
- Added SetDateTimeKindNode (implemented by @ProbablePrime | Docs)
-- This allows setting the Kind property on a DateTime instance
- Added VBLFC badge (requested by @Shifty | Quality Control Lead on behalf of @Kulza)

[h2]Security:[/h2]
- Improved ban evasion detection system, preventing users from bypassing public bans by logging out of their accounts (based on reports by the moderation team)
- Added salts to various ID's, to increase difficulty of matching them through a rainbow table (based on report by @runtime)
- Improved the global moderation system to apply account restrictions immediatelly, rather than having to wait for the user to leave and/or rejoin (based on request from @CanadianGit | Moderation Team, @Veer | CMO and rest of the moderation team)
- Added pathWhitelist setting to Config.json which allows restricting (or completely disabling) locations accessible via the built-in File Browser (requested by @Earthmark)
-- This is an array of root paths that can be browsed. E.g. C:\Data\Neos will allow browsing everything in and under that folder, but nowhere else
-- Providing empty array ([ ]) will disable the File Browser functionality completely
-- This can be useful also for events and other situations where you don't want other people browsing everything on your computer

[h2]Tweaks:[/h2]
- When set to Invisible, you automatically don't send realtime message read status to the other user
- Tweaked CosmosDB timeouts and other settings to potentially reduce number of failed requests (based on recent partial outage and community reports)
- Added better REST message API rate limiting mechanism to prevent bad requests from 3rd party applications from causing cloud unresponsiveness
-- If you're making such application, please note that we currently do not officially support this. At very least we ask you to be considerate of how you use the API (e.g. avoid frequently fetching large amounts of messages), otherwise we'll have to tighten the restrictions more or even add a lockout mechanism if too many problematic requests come from your app
- Registration button on auth.neos.com now links to account.neos.com, since the registration on the new website isn't fully implemented yet (based on report by @Levi)
- Removed developer notification message form the Error page on auth.neos.com (based on report by @Levi)
- UserTime node now outputs DateTime with Unspecified kind, rather than UTC (based on report by @I'm Erin., implemented by @ProbablePrime | Docs)

- Merged Japanese locale update by @Aesc
- Merged Russian locale update by @Shadow Panther [RU/EN, UTC+3]
- Merged Czech locale additions and update by @rampa_3 (UTC +1, DST UTC +2)

[h2]Bugfixes:[/h2]
- Fixed Twitch dialog buttons not working after recent change (reported by @LemonCement and @Fuzzy, fixed by @ProbablePrime | Docs)
- Fixed some messages not being properly mark as read on the next login despite reading them, due to the marking as read being processed before the message is fully persisted in the database
- Fixed AdminX not working due to missing some SignalR related libraries