2020.12.4.68 - Major cloud SDK upgrade and many bugfixes (inc. asset variants)
Sorry for a smaller build, I ended up going another rabbit hole investigating an issue in the cloud yesterday, that ended up with upgrading the whole cloud infrastructure to new Azure Storage SDK, moving away from the deprecated v11 to new v12 version. This was a major change as the API's aren't compatible, so I had to adjust a lot of existing code, but as bonus the cloud API should now be snappier and use fewer resources, as well as have access to latest features.
During the process I also discovered some long-standing bugs and pathological behavior which are now fixed. One of them that started this had to do with the asset variant system, which resulted in a flood of duplicate asset computation requests (it got to up to 8 million in the queue), which pretty much resulted in the whole system griding down and not processing much (it used to try 6 times to get a valid task and then the processor would wait 5 seconds. Since most of them were invalid, this made it go really REALLY slow).
The queue is now being processed rapidly (it's down to 2.7 million), you might still experience some delays with some assets not being processed and loading in original uncompressed versions, but I think it'll clear up within a day or two. The new system also has measures to prevent this from happening again and clearing any duplicate tasks significantly faster.
This build is compatible with previous one.
[h2]Tweaks:[/h2]
- Added new world category icons by @Coffee | Programmer
- Upgraded Azure storage & table SDK's to the new v12, moving away from the deprecated v11 API and reworked the cloud code onto this new API
-- This should provide some performance benefits for the cloud as well as get the latest bugfixes and features for future development
-- This is also major upgrade as a lot of code had to be adjusted. I have tested things thoroughly before pushing this fully live, but please watch out for any cloud weirdness
- Switched cloud queue message encoding to UTF-8 from Base64 with the new SDK, improving the cloud performance and reducing bandwidth/storage use
- Optimized cloud record preprocessing to reduce overall internal traffic with unecessarily frequent queue processing extensions
- Added a mechanism to prevent flood of duplicate asset variant tasks in the queue in the cloud, causing delays before assets actually get processed and wasting cloud performance
- AdminX now enforces UTF-8 encoding for the console input/output, fixing cases where non-ASCII characters would be displayed/processed incorrectly (based on report from @Karel | CEO)
- Removed MMC world category
- Upgraded to Unity 2019.4.16f1 (from 2019.4.15f1)
- Merged Swedish locale additions by @Swingly
- Merged Russian locale additions by @Shadow Panther [RU/EN, UTC+3]
- Merged Japanese, Esperanto and Chinese locale additions by @Melnus
- Merged Czech locale additions by @rampa_3 (UTC +1, DST UTC +2)
- Merged Korean locale additions by @MirPASEC
[h2]Bugfiixes:[/h2]
- Fixed asset variant processing falling severely behind when there's a flood of duplicate or invalid entries, causing the processing queue to grow faster than it's processeed, despite barely any assets being actually processsed
- Fixed asset variant system requesting invalid metadata type for Cubemaps when a variant is requested, but metadata isn't present
- Fixed cloud dependency removal tokens not being scheduled properly, causing some data not being validated or cleaned up
-- This is mostly internal bugfix, as the issue is compensated by other failsafe subsystems, but this should reduce some unecessary load and errors in cloud processing
- Fixed cloud queue system not deleting processed tasks from the queue properly due to task extension, causing a lot of unecessary reprocessing to occur
- Added generic argument validation to ButtonValueShift to prevent instantiating wrong type and causing crash (reported by @Komdog)
During the process I also discovered some long-standing bugs and pathological behavior which are now fixed. One of them that started this had to do with the asset variant system, which resulted in a flood of duplicate asset computation requests (it got to up to 8 million in the queue), which pretty much resulted in the whole system griding down and not processing much (it used to try 6 times to get a valid task and then the processor would wait 5 seconds. Since most of them were invalid, this made it go really REALLY slow).
The queue is now being processed rapidly (it's down to 2.7 million), you might still experience some delays with some assets not being processed and loading in original uncompressed versions, but I think it'll clear up within a day or two. The new system also has measures to prevent this from happening again and clearing any duplicate tasks significantly faster.
This build is compatible with previous one.
[h2]Tweaks:[/h2]
- Added new world category icons by @Coffee | Programmer
- Upgraded Azure storage & table SDK's to the new v12, moving away from the deprecated v11 API and reworked the cloud code onto this new API
-- This should provide some performance benefits for the cloud as well as get the latest bugfixes and features for future development
-- This is also major upgrade as a lot of code had to be adjusted. I have tested things thoroughly before pushing this fully live, but please watch out for any cloud weirdness
- Switched cloud queue message encoding to UTF-8 from Base64 with the new SDK, improving the cloud performance and reducing bandwidth/storage use
- Optimized cloud record preprocessing to reduce overall internal traffic with unecessarily frequent queue processing extensions
- Added a mechanism to prevent flood of duplicate asset variant tasks in the queue in the cloud, causing delays before assets actually get processed and wasting cloud performance
- AdminX now enforces UTF-8 encoding for the console input/output, fixing cases where non-ASCII characters would be displayed/processed incorrectly (based on report from @Karel | CEO)
- Removed MMC world category
- Upgraded to Unity 2019.4.16f1 (from 2019.4.15f1)
- Merged Swedish locale additions by @Swingly
- Merged Russian locale additions by @Shadow Panther [RU/EN, UTC+3]
- Merged Japanese, Esperanto and Chinese locale additions by @Melnus
- Merged Czech locale additions by @rampa_3 (UTC +1, DST UTC +2)
- Merged Korean locale additions by @MirPASEC
[h2]Bugfiixes:[/h2]
- Fixed asset variant processing falling severely behind when there's a flood of duplicate or invalid entries, causing the processing queue to grow faster than it's processeed, despite barely any assets being actually processsed
- Fixed asset variant system requesting invalid metadata type for Cubemaps when a variant is requested, but metadata isn't present
- Fixed cloud dependency removal tokens not being scheduled properly, causing some data not being validated or cleaned up
-- This is mostly internal bugfix, as the issue is compensated by other failsafe subsystems, but this should reduce some unecessary load and errors in cloud processing
- Fixed cloud queue system not deleting processed tasks from the queue properly due to task extension, causing a lot of unecessary reprocessing to occur
- Added generic argument validation to ButtonValueShift to prevent instantiating wrong type and causing crash (reported by @Komdog)