2021.8.29.1240 - Subtitle import support, reading texture 2D pixels in LogiX...
Hello everyone! I got another build for you with a whole bunch of new goodies and bugfixes.
You can now import Subtitle files natively into Neos! This can be useful for a lot of various workflows, as they get converted into native Neos animation file and can be sampled by time for lots of purposes, syncing them with anything, including custom content.
On related note, LogiX has been expanded a bit with access to some asset types. You can now read the pixels of static 2D textures (either by UV or pixel position)! This could prove useful if you need to encode some 2D spatial information into texture data and use if for your own logic. Similarly you can sample animation tracks from LogiX as well, using time as an input, letting you build cool systems on top of it.
There's also a potentially very important bugfix - turns out that headless wasn't syncing the master clock due to not having any streams of its own. This had a bit of a cascade effect, where a lot of things could drift out of sync on headless sessions, like video/audio playback, LogiX setups and other things, so hopefully this will fixup a lot of issues in one swoop!
I've also merged in latest updates from BEPUv2 upstream and cleaned up everything to now build on the Azure Pipelines rather than local only build so other devs aren't blocked by a missing dependency. This should include some collision edge case improvements too as well as some small optimizations that were missing in our fork.
I'm designing the optimization for dynamic bones on top of BEPUv2 structures, it's going to take a bit more time, so I'm pushing this out before then, but hopefully that should come soon enough too! From some preliminary profiling it should give us another nice performance boost.
[h2]New Features:[/h2]
- Added native subtitle import support (based on request by @Karel | CEO, @Robyn (QueenHidi), RoxyBoxxy, @DovahDoVolom and others, GH #1260)
-- Following formats are currently supported: srt, sub, ssa, ttml, vtt
-- The subtitle is converted in Neos native animation (AnimX) with track with string keyframes. This can be used to sync playback with video or any other playback (including any custom Neos content - they are simply treated as animation with keyframes that are pieces of text)
-- Note that subtitles embedded in video files aren't supported yet
- Added PlaybackSynchronizer, which syncs one playback to another
-- It can optionally sync normalized positions for clips of different duration
-- Note that both playbacks are still individual and can have slight jitter. This doesn't guarantee perfect synchronization, but it should be generally low enough, depending on what's being synchronized
-- You can specify position offset and position rate (this adjusts speed as well)
- Added GetAsset node (under Assets), which will get the asset from input asset provider (e.g. plugging StaticTexture2D will give you Texture2D)
- Added Sample Texture2D UV node (under Assets) which will sample the texture color at given UV coordinate using bilinear interpolation (requested by @BlaXun | Ingo#6920, @Shifty | Quality Control Lead and others, GH #365)
-- Note that the texture must be marked as readable in order to access its data and in format that supports CPU read (e.g. not block compressed)
- Added Texture2D Format node (under assets) which provides basic format and dimensions of input texture
- Added Get Texture2D Pixel node (under Assets), which allows getting specific pixel at specific mip level of a given texture
-- Note that the texture must also be marked as readable for this to work
-- If the position or mipmap are outside of range, clear color will be returned
- Added SampleAnimationTrack node which allows sampling values from imported animations at given time point
-- T must be same type as the corresponding animation track, otherwise it won't work
- Added FindAnimationTrackIndex which finds index of a track for given node and property
-- Returns -1 if no matching track is found
-- It's recommended to cache this value instead of fetching it every time - you can even store it ahead of time if you're always using same animation track - the indexes don't change between loads of the same animation asset
- Added CurrentClockError to VideoTexture, which provides the current time offset of the video playback (as reported by the video playback engine) to the desired playback position from Neos
-- This can be used for diagnostics or to correct other playback components to match the video (you might need to apply some smoothing rather than using raw value due to jitter)
- Added DiscreteAnimationTrack type internally, which allows efficiently holding "Hold" type keyframes that are generally more sparse
[h2]Security:[/h2]
- Removed old ChildrenSynchronizer component (Ticket #145123)
-- This is due to combination of it's half broken state (it's from before inventory was implemented) and newly found security issues that wouldn't be possible to solve without complete redesign
-- Please use inventory or grab things if you want to move them between the worlds, as this gives you more control over what's transfered and is more reliable
- Internal moderation system improvements
Tweaks:
- Merged latest BEPUv2 upstream changes to our fork
-- Has improved corner case handling for triangle-cylinder collisions
-- Fixes for convex hull generation producing invalid hulls
-- Various small(ish) optimizations
- Neos now prints out supported file formats on startup for diagnostics and reference
- Removed workaround for invalid convex hulls thanks to the direct fix from BEPUv2
- Partially reverted input deadzone behavior to original (previously changed based on GH #2613) for turning
-- This fixes forward movement usually causing small amount of turning, since it's impossible to tilt the joystick perfectly straight (reported by @Veer | CMO)
- Merged Spanish locale additions by @Ruz
- Merged Japanese locale tweaks by @Aesc/あすく
[h2]Optimizations:[/h2]
- Small BoundingBox code optimizations
- Fixed forgotten Compound collider test override in BEPUv2, resulting in performance drops in some cases
- Fixed memory streams not being recycled when there are no network streams to encode at given frame, resulting in unecessary memory allocations and additional CPU usage
- Removed unused collider metadata handling
[h2]Bugfixes:[/h2]
- Added a regular stream packet transmission when no other stream packets were sent (typically a headless host), fixing cases where the master clock to drift
-- This should fix numerous time sync issues (reported by @Enverex , @Jolts, @Axius, @AshtonSparx, @Ikani [she/her], @Party_Grunt, GH #2895)
-- E.g. Video, audio, animation playback
-- Audio not playing for some users (reported by @Enverex and @Shadow Panther [RU/EN, UTC+3], GH #2890)
-- Any LogiX using T as the time basis for animating parameters
-- Any time-based driving components - panner, spinner, wiggler and so on
-- Any other time based nodes like Stopwatch or Elapsed Time
- Fixed uninitialized use of data structures in Contact Event Handling
- Fixed disposing of uninitialized data structures in the BEPUv2 integration, potentially causing corruptions
- AudioOutput volume is now clamped before the user volume setting is applied, preventing it from exceeding the user set maximum volume (reported by @AshtonSparx, GH #2894)
- Fixed CharacterEventTrigger & CharcterForceField only responding to Triggers, despite TriggersOnly being false (reported by @Shifty | Quality Control Lead and @Green, GH #2891)
You can now import Subtitle files natively into Neos! This can be useful for a lot of various workflows, as they get converted into native Neos animation file and can be sampled by time for lots of purposes, syncing them with anything, including custom content.
On related note, LogiX has been expanded a bit with access to some asset types. You can now read the pixels of static 2D textures (either by UV or pixel position)! This could prove useful if you need to encode some 2D spatial information into texture data and use if for your own logic. Similarly you can sample animation tracks from LogiX as well, using time as an input, letting you build cool systems on top of it.
There's also a potentially very important bugfix - turns out that headless wasn't syncing the master clock due to not having any streams of its own. This had a bit of a cascade effect, where a lot of things could drift out of sync on headless sessions, like video/audio playback, LogiX setups and other things, so hopefully this will fixup a lot of issues in one swoop!
I've also merged in latest updates from BEPUv2 upstream and cleaned up everything to now build on the Azure Pipelines rather than local only build so other devs aren't blocked by a missing dependency. This should include some collision edge case improvements too as well as some small optimizations that were missing in our fork.
I'm designing the optimization for dynamic bones on top of BEPUv2 structures, it's going to take a bit more time, so I'm pushing this out before then, but hopefully that should come soon enough too! From some preliminary profiling it should give us another nice performance boost.
[h2]New Features:[/h2]
- Added native subtitle import support (based on request by @Karel | CEO, @Robyn (QueenHidi), RoxyBoxxy, @DovahDoVolom and others, GH #1260)
-- Following formats are currently supported: srt, sub, ssa, ttml, vtt
-- The subtitle is converted in Neos native animation (AnimX) with track with string keyframes. This can be used to sync playback with video or any other playback (including any custom Neos content - they are simply treated as animation with keyframes that are pieces of text)
-- Note that subtitles embedded in video files aren't supported yet
- Added PlaybackSynchronizer, which syncs one playback to another
-- It can optionally sync normalized positions for clips of different duration
-- Note that both playbacks are still individual and can have slight jitter. This doesn't guarantee perfect synchronization, but it should be generally low enough, depending on what's being synchronized
-- You can specify position offset and position rate (this adjusts speed as well)
- Added GetAsset node (under Assets), which will get the asset from input asset provider (e.g. plugging StaticTexture2D will give you Texture2D)
- Added Sample Texture2D UV node (under Assets) which will sample the texture color at given UV coordinate using bilinear interpolation (requested by @BlaXun | Ingo#6920, @Shifty | Quality Control Lead and others, GH #365)
-- Note that the texture must be marked as readable in order to access its data and in format that supports CPU read (e.g. not block compressed)
- Added Texture2D Format node (under assets) which provides basic format and dimensions of input texture
- Added Get Texture2D Pixel node (under Assets), which allows getting specific pixel at specific mip level of a given texture
-- Note that the texture must also be marked as readable for this to work
-- If the position or mipmap are outside of range, clear color will be returned
- Added SampleAnimationTrack node which allows sampling values from imported animations at given time point
-- T must be same type as the corresponding animation track, otherwise it won't work
- Added FindAnimationTrackIndex which finds index of a track for given node and property
-- Returns -1 if no matching track is found
-- It's recommended to cache this value instead of fetching it every time - you can even store it ahead of time if you're always using same animation track - the indexes don't change between loads of the same animation asset
- Added CurrentClockError to VideoTexture, which provides the current time offset of the video playback (as reported by the video playback engine) to the desired playback position from Neos
-- This can be used for diagnostics or to correct other playback components to match the video (you might need to apply some smoothing rather than using raw value due to jitter)
- Added DiscreteAnimationTrack type internally, which allows efficiently holding "Hold" type keyframes that are generally more sparse
[h2]Security:[/h2]
- Removed old ChildrenSynchronizer component (Ticket #145123)
-- This is due to combination of it's half broken state (it's from before inventory was implemented) and newly found security issues that wouldn't be possible to solve without complete redesign
-- Please use inventory or grab things if you want to move them between the worlds, as this gives you more control over what's transfered and is more reliable
- Internal moderation system improvements
Tweaks:
- Merged latest BEPUv2 upstream changes to our fork
-- Has improved corner case handling for triangle-cylinder collisions
-- Fixes for convex hull generation producing invalid hulls
-- Various small(ish) optimizations
- Neos now prints out supported file formats on startup for diagnostics and reference
- Removed workaround for invalid convex hulls thanks to the direct fix from BEPUv2
- Partially reverted input deadzone behavior to original (previously changed based on GH #2613) for turning
-- This fixes forward movement usually causing small amount of turning, since it's impossible to tilt the joystick perfectly straight (reported by @Veer | CMO)
- Merged Spanish locale additions by @Ruz
- Merged Japanese locale tweaks by @Aesc/あすく
[h2]Optimizations:[/h2]
- Small BoundingBox code optimizations
- Fixed forgotten Compound collider test override in BEPUv2, resulting in performance drops in some cases
- Fixed memory streams not being recycled when there are no network streams to encode at given frame, resulting in unecessary memory allocations and additional CPU usage
- Removed unused collider metadata handling
[h2]Bugfixes:[/h2]
- Added a regular stream packet transmission when no other stream packets were sent (typically a headless host), fixing cases where the master clock to drift
-- This should fix numerous time sync issues (reported by @Enverex , @Jolts, @Axius, @AshtonSparx, @Ikani [she/her], @Party_Grunt, GH #2895)
-- E.g. Video, audio, animation playback
-- Audio not playing for some users (reported by @Enverex and @Shadow Panther [RU/EN, UTC+3], GH #2890)
-- Any LogiX using T as the time basis for animating parameters
-- Any time-based driving components - panner, spinner, wiggler and so on
-- Any other time based nodes like Stopwatch or Elapsed Time
- Fixed uninitialized use of data structures in Contact Event Handling
- Fixed disposing of uninitialized data structures in the BEPUv2 integration, potentially causing corruptions
- AudioOutput volume is now clamped before the user volume setting is applied, preventing it from exceeding the user set maximum volume (reported by @AshtonSparx, GH #2894)
- Fixed CharacterEventTrigger & CharcterForceField only responding to Triggers, despite TriggersOnly being false (reported by @Shifty | Quality Control Lead and @Green, GH #2891)