2021.11.7.794 - World saving bugfixes, watchdog mechanism to diagnose lock-ups
Hello everyone! Pushing another maintenance build, mostly with bugfixes and things to help with diagnose some more errors.
Compatible with previous builds.
[h2]New Features:[/h2]
- Added engine watchdog mechanism, which will print out diagnostic information when Neos hardlocks (freezes) (based on reports from @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265)
-- This should significantly help us solving cases of hard freezes and lock-ups
-- If your Neos freezes, leave it running for at least a minute before killing the process and then send us the logs
[h2]Tweaks:[/h2]
- Updated dependencies to latest version (notably SignalR Client)
- Added diagnostics for stream messages to the watchdog mechanism
- The watchdog mechanism will now dump the currently processing sync message when Neos is frozen for longer than 30 seconds for diagnostic purposes
- Stream messages that are older than a few seconds are now thrown away instead of processed, reducing the load on the system during hiccups
- Add stricter validation for stream message versions
-- This might potentially fix the random freezing reported by @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265), at very least we should gather some more data
- Add extra diagnostic information to World to the watchdog mechanism
- Added extra diagnostic information for the watchdog mechanism to help diagnose cases of Neos freezing up (based on reports from @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265)
- Improved stacktrace capture mechanism for the watchdog mechanism
- Implemented proper disposal of threading resources when closing a world to avoid holding on system resources for too long
- Merged Russian locale additions and fixes by @Shadow Panther [RU/EN, UTC+3]
- Merged English locale fix by @Shadow Panther [RU/EN, UTC+3]
- Merged Korean locale additions by @MirPASEC
- Merged Japanese locale additions by @Aesc/あすく
- Merged Czech locale additions by @rampa_3 (UTC +1, DST UTC +2)
[h2]Bugfixes:[/h2]
- World name, description and tags are now snapshotted synchronously during the save process, preventing the world being closed or changed between saving and generating of record breaking the save process (based on report by @Medra, @Creator Jam, @Jeff0SP00K5 and @Rucio, @Shifty | Quality Control Lead, GH #3245)
- Improved error handling when saving worlds, to prevent the save process from getting stuck indefinitely and potentially not logging any save errors (based on report by @Medra, @Creator Jam, @Jeff0SP00K5 and @Rucio, GH #3245)
- Fix last async message encoding throwing exceptions on worlds are just being closed
- Added extra diagnostic information to failed world saves
- Added logging information for errors when resolving UDP Punchthrough and Relay URL's (based on report by @kazu0617 Neos:kazu, GH #3323)
- Fixed world crashing when spawning ToolsFacetPreset in world (reported by @Lexevo, GH #3304)
- Readded accidentally removed Debug button to the Tools Facet Preset (reported by @Lexevo, GH #3305)
Compatible with previous builds.
[h2]New Features:[/h2]
- Added engine watchdog mechanism, which will print out diagnostic information when Neos hardlocks (freezes) (based on reports from @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265)
-- This should significantly help us solving cases of hard freezes and lock-ups
-- If your Neos freezes, leave it running for at least a minute before killing the process and then send us the logs
[h2]Tweaks:[/h2]
- Updated dependencies to latest version (notably SignalR Client)
- Added diagnostics for stream messages to the watchdog mechanism
- The watchdog mechanism will now dump the currently processing sync message when Neos is frozen for longer than 30 seconds for diagnostic purposes
- Stream messages that are older than a few seconds are now thrown away instead of processed, reducing the load on the system during hiccups
- Add stricter validation for stream message versions
-- This might potentially fix the random freezing reported by @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265), at very least we should gather some more data
- Add extra diagnostic information to World to the watchdog mechanism
- Added extra diagnostic information for the watchdog mechanism to help diagnose cases of Neos freezing up (based on reports from @AshtonSparx, @AdmiralSoap, @Alex Thorin, @Party_Grunt, bontebok, @orange, @Nordwig, @Cyro and others, GH #3265)
- Improved stacktrace capture mechanism for the watchdog mechanism
- Implemented proper disposal of threading resources when closing a world to avoid holding on system resources for too long
- Merged Russian locale additions and fixes by @Shadow Panther [RU/EN, UTC+3]
- Merged English locale fix by @Shadow Panther [RU/EN, UTC+3]
- Merged Korean locale additions by @MirPASEC
- Merged Japanese locale additions by @Aesc/あすく
- Merged Czech locale additions by @rampa_3 (UTC +1, DST UTC +2)
[h2]Bugfixes:[/h2]
- World name, description and tags are now snapshotted synchronously during the save process, preventing the world being closed or changed between saving and generating of record breaking the save process (based on report by @Medra, @Creator Jam, @Jeff0SP00K5 and @Rucio, @Shifty | Quality Control Lead, GH #3245)
- Improved error handling when saving worlds, to prevent the save process from getting stuck indefinitely and potentially not logging any save errors (based on report by @Medra, @Creator Jam, @Jeff0SP00K5 and @Rucio, GH #3245)
- Fix last async message encoding throwing exceptions on worlds are just being closed
- Added extra diagnostic information to failed world saves
- Added logging information for errors when resolving UDP Punchthrough and Relay URL's (based on report by @kazu0617 Neos:kazu, GH #3323)
- Fixed world crashing when spawning ToolsFacetPreset in world (reported by @Lexevo, GH #3304)
- Readded accidentally removed Debug button to the Tools Facet Preset (reported by @Lexevo, GH #3305)