24 Oct 25 | Hotfix 2 Status | Server and Stability Statement | Update 18

Hotfix 2 Status - Server and Stability Update - Fri 24th October

[p][/p][p]Today marks a major milestone in our ongoing efforts to resolve the stability and crashing issues that have affected Hell Let Loose since Update 18.[/p][p]After many days of investigation, testing, and community support, we’re pleased to share that the team has successfully identified the root cause of the problem and implemented a fix. This new build has been tested and is almost ready for an rollout as Hotfix 2.[/p][p]We know how disruptive the last week has been for many of you, and we want to thank everyone who has continued to provide reports, feedback, and patience as we worked to get the game back to a stable state. Your input has been instrumental in helping the team reach this point.[/p][p][/p][h3]TLDR Summary[/h3]

[p]The team has identified the root cause of the crashes affecting Hell Let Loose since Update 18.[/p]
[p]The issue was linked to a physics material reference error caused by certain assets, which could result in client and server crashes.[/p]
[p]A new build containing a fix for both client and server crashes has been created, tested, and certified.[/p]
[p]This fix is now available for Steam players and server owners through a new ‘Experimental Branch’.[/p]
[p]The full Hotfix 2 release will roll out on Steam and other platforms early next week following platform approvals.[/p]
[p]Players and server hosts on the main branch or console platforms are advised to remove the following maps from rotation over the weekend:
Carentan, Foy, Omaha, Utah, Remagen, Sainte-Mère-Église (SME), Stalingrad[/p]
[p]The team will continue to monitor for any remaining issues, while also preparing additional asset corrections for Update 19.[/p]
[p]Thank you to everyone who provided reports, feedback, and patience. Your support directly contributed to identifying and fixing the issue.[/p]

[p][/p][h2]Status of the Investigation[/h2][p][/p][h3]Latest Update[/h3][p]After yesterday’s breakthrough in reproducing the crash, Kieran and his team worked late into the night to pinpoint the exact cause. It wasn’t an easy process, but by the end of the evening they had successfully identified the specific issue, allowing work on the fix to begin immediately.[/p][p]This morning, the first new build containing the fix was completed. It also includes an additional correction for another potential fault in the same chain of logic.[/p][p]Since then, our QA team has been rigorously testing the new build, which was successfully certified this afternoon.[/p][p]With this update, we have addressed both client crashes (often seen as mass disconnects) and server crashes (which previously caused players to lose connection).[/p][p]We would like to note that, due to the nature of the issue, there may have been other undefined behaviours linked to it that did not always result in a crash. These should now also be resolved. However, following this hotfix, we will continue monitoring to ensure there are no remaining causes of client disconnections.[/p][p][/p][h3]Details on the Hotfix 2 for Update 18[/h3][p]Thank you all for your patience. We can now confirm that a new build of Hell Let Loose has been created and includes a fix for the crashing issue. Below are the key details to be aware of:[/p][p][/p][h3]1. New Steam Branch:[/h3][p]A new Experimental Branch is now available on Steam. You can access it by navigating to:
Steam → Hell Let Loose → Properties → Betas → Experimental Branch[/p][p][/p][p][/p][p]Head to the Properties section of the Hell Let Loose app on Steam. Do this by right clicking on Hell Let Loose, or hitting the settings cog within the Steam page.[/p][p] [/p][p][/p][p]Go to Betas, and then select ‘experimental_branch’[/p][p] [/p]

[p]The “None” option is the default/main branch that has the current Live 18.0 Hotfix 1 build[/p]
[p]The “experimental_branch” right now has 18.0 Hotfix 2 build[/p]

[p][/p][p]Switching to the new branch in the Steam client is a 3.2GB download. Those that switch, once the patch rolls out globally next week, will need to switch back to the main branch ‘None’. Once back to the default branch, you will be instantly in as it will be the same build at that point.[/p][p]This branch includes the client-side fix, and community servers can now opt to move across. A number of official servers are also live on this build.[/p][p]Please note that this is an opt-in process. Servers and players using the Experimental Branch will not appear in the default server list. This setup is intended for those who want to play on the new build servers and on the new build.[/p][p]A full release of Hotfix 2 will come out on Steam early next week alongside the other platforms.[/p][p] [/p][h3]2. Other Platforms:[/h3][p]Due to approval requirements, this version will not go live on other platforms today. It is currently scheduled for release early next week.[/p][p]In the meantime, for all other platforms (and for those on Steam who have already begun seeding on the main branch) we have shared a list of maps that have been identified as potential triggers for the crash:[/p]

[p]Carentan[/p]
[p]Foy[/p]
[p]Omaha[/p]
[p]Utah[/p]
[p]Remagen[/p]
[p]Sainte-Mère-Église (SME)[/p]
[p]Stalingrad[/p]

[p]These maps have the highest likelihood of triggering the crash. We strongly recommend removing them from rotation if you are on the main Steam branch or any other platform (Epic, WinStore, Xbox, or PS5).[/p][p]Thank you again for your continued patience and support. It has been invaluable throughout this process, and we deeply appreciate everyone in the community for working with us as we’ve resolved the issue.[/p][p][/p][h2]From the Dev Team: Understanding the Issue[/h2][p]Yesterday, Kieran, Studio Technical Director at Expression Games, shared a detailed breakdown of the work happening behind the scenes to uncover the root cause of the crashes, and why this issue only appeared in the live environment. Following the breakthrough we had last night, we were able to identify the cause of the crashing issue and deliver a fix.[/p][p]With this in mind, we wanted to continue our commitment to transparency by providing a deeper explanation of what went wrong and how it was resolved.[/p][p][/p][h3]How The Crash Presents Itself[/h3][p]The crash can manifest on any client or the game server. In both cases, one of the following functions would typically appear near the top of the call stack:[/p][p][/p][p]The crash appeared to happen randomly and didn’t have a 100% reproduction rate, which made it extremely difficult to track down.[/p][p][/p][h3]How The Core Issue Was Discovered[/h3][p]When the issue first began to appear in community reports post-Update 18, the team reviewed crashes captured through Backtrace in the development environment. We later confirmed that the issue had existed since 28 July 2025, with only four crashes recorded during that time (between 28 July - 16 October): two server-side and two client-side. This immediately showed how hard it was to reproduce under normal development conditions.[/p][p]At first, it was unclear whether the cause was related to server type (GPortal versus community servers), differences in server tools like CRCON, whether it required a full server populations, or specific gameplay situations that you you seen in a public game environment.[/p][p]To narrow things down, we explored all these possibilities in parallel. We collaborated closely with the community, server owners, and War Correspondents to gather reports and eliminate potential causes.[/p][p]Because the issue occurred so rarely during testing, it became clear that the only way to properly diagnose it was to continue investigating while the bug remained live in the game. If we had rolled back Update 18, there was a real chance we might never reproduce it during internal testing, which would have prevented us from finding and fixing the issue.[/p][p]On Wednesday 22 October, the first successful reproduction of the crash occurred on an live official server, ruling out both the server provider and external tools (e.g. CRCON) as contributing factors. During this time, we identified that the Foy map had the highest rate of reproductions, thanks to reports from the server owners and War Correspondents.[/p][p]The next day (Thursday 23 October), QA focused on Foy specifically. At that point, there was a theory that the new artillery strike commander ability might be related. While that turned out to be true, although not in the way initially suspected, it was during these tests that we got lucky.[/p][p]A QA tester experienced a crash after entering a building. One by one, other testers entered the same building and also crashed. That gave us our 100% reproduction case.[/p][p][/p][h3]What The Problem Was[/h3][p]When performing physics tests in Hell Let Loose, the game often requests the physical material of the object being interacted with. This information determines what visual effects, particles, and sounds are used when bullets or projectiles hit that surface.[/p][p]With Update 18, some assets were re-imported to resolve texture issues, but this process unintentionally affected their Level of Detail (LOD) setup. As a result, certain objects were referencing one fewer material than they should have.[/p][p]When the physics system interacted with these assets, it sometimes attempted to read a material from an invalid memory address. Depending on what that memory address pointed to, what happens next is down to chance, with one of two outcomes occuring:[/p]

[p]The game continued running, unknowingly using corrupted data.[/p]
[p]The game crashed immediately in one of the previously listed functions.[/p]

[p]This randomness explained why reproducing the crash was so difficult; it depended on chance whether the invalid pointer caused an instant crash or not.[/p][p]For reference, the three most likely causes of the crash, in order, are:[/p]

[p]Simulating a bullet trajectory, and it hitting a “bad” object[/p]
[p]Simulating a projectile (e.g. artillery), and it hitting a “bad” object[/p]
[p]Walking on a “bad” object[/p]

[p][/p][h3]What Was The Fix[/h3][p]Using the PhysX API directly, the team implemented a validation step to ensure that when retrieving a material index, it matches a valid material within the asset. If the index does not map correctly, the system safely exits that process without returning a physical material, preventing the crash.[/p][p]In practice, this means that for “bad” assets, the correct visual or sound effect may occasionally not appear. However, since this issue has existed for months without being noticed, the impact will be minimal, and most importantly, the game and servers will no longer crash.[/p][p]Over the coming weeks, the team will continue identifying and repairing these affected assets to ensure they are corrected for Update 19.[/p][p][/p][h3]How Will We Ensure That This Specific Issue This Doesn’t Happen Again[/h3][p]In future non-shipping builds, whenever this issue is detected, a red on-screen message will appear that reads:[/p][p][/p][p]This system allows QA to easily identify and report any problematic assets. Since the message clearly shows the actor and component names, the art team can quickly locate and fix the issue.[/p][p]Because this error can occur through any kind of physics interaction (shooting, bombing, or walking on an affected surface) QA will naturally encounter and resolve these assets during standard gameplay testing.[/p][p] [/p]