Hell Let Loose News

[p]Update on our investigations into the server and stability issues since Update 18.[/p][p][/p][p]Thank you to everyone who has continued to support us and share crash reports over the past few days. Your feedback and detailed reports have played a huge role in helping us make real progress.[/p][p]Today, we’re pleased to share that we’ve taken a major step forward in our investigation. The team has been able to consistently reproduce the crash, which brings us much closer to identifying the exact cause. We know how important stability is to your experience, and we remain fully committed to resolving this issue as quickly as possible.[/p][p][/p][h2]TLDR Summary[/h2]

• [p]The team has made a major breakthrough in the investigation.[/p]
• [p]We have now found a consistent way to reproduce the crash, allowing us to focus on a much narrower area of the game’s systems.[/p]
• [p]This issue has been extremely difficult to track down, as reproducing it in a QA or development environment was nearly impossible until now.[/p]
• [p]With this progress, the team is closing in on the root cause and continuing to work around the clock toward a fix.[/p]
• [p]We will share more information before the weekend so stay tuned.[/p]

[p][/p][h2]Status of the Investigation[/h2][p][/p][h2]Latest Update[/h2][p]Today marks a significant step forward in resolving the current crashing issues.[/p][p]As of this afternoon, the team has discovered a consistent way to trigger a crash. This breakthrough allows us to focus on a much smaller area of the game’s systems to pinpoint the exact cause.[/p][p]This particular issue has been a needle in a haystack to uncover. The conditions required to reproduce it are so specific that encountering it within a QA or development environment is almost impossible. Finding a reliable way to make it happen marks a major turning point in our investigation.[/p][p]The team continues to work around the clock to determine the root issue, and have been focused on getting this resolved as soon as possible.[/p][p]We will be sharing another update tomorrow and hope to have more information available before the weekend.[/p][p][/p][h2]From the Dev Team: Understanding the Issue[/h2][p]As part of our update yesterday, we shared more behind-the-scenes details about the process of tracking down the cause of the recent crashes. The response from the community has been incredibly supportive, and it’s been great to see players taking such an interest in how we’re approaching our investigation. With that in mind, we wanted to expand a little further on how we got here and what we’ve been doing to resolve the issue.[/p][p][/p][p]Here is an explanation from Kieran, Studio Technical Director at Expression Games:[/p]

[p]“The release build for Update 18 was starting to be finalised from 10 September, over a month ago. This means the underlying issue responsible for the current crashes has likely existed since that point, and possibly even earlier within our development environment.[/p][p]Prior to yesterday afternoon (22 October), we had only observed a handful of unrelated crashes internally, and none provided indicators of the issue that has since appeared in the live environment.[/p][p]The challenge we face is that this specific crash occurs when something has already gone wrong before the actual crash point. As a result, the crash dumps we collect, both from our internal testing and from the community, show where the crash happens within PhysX but provide little insight into how the physics engine reached that unstable state.[/p][p]Our next step is to fully understand what conditions lead to the crash so we can reliably reproduce it. Once we can reproduce it consistently, we can verify a fix and roll it out confidently to everyone.[/p][p]The good news is that on Wednesday 22 October, we successfully triggered the crash on a live server outside of the development environment for the first time. This discovery allowed us to focus our search on what causes the crash. By attaching development tools to clients playing on live servers, we were then able to collect detailed debugging information whenever a crash occurs, helping us better understand its cause.[/p][p]Using this approach, as of this afternoon we have now been able to reproduce the crash within the development environment. This marks a major step forward, and we are closing in on the root cause.[/p][p]We would like to thank everyone who has been submitting crash logs and reports during this process. Your contributions have been an invaluable resource in our investigation. We’re confident that once we’ve confirmed the exact issue, we’ll be able to prepare and deliver a fix as soon as possible. Your support means a great deal to me and to the wider development team.”[/p]

[h2]Server and Stability Update - Wed 22nd October[/h2][p]Update on our investigations into the server and stability issues since Update 18.[/p][p][/p][p]Thank you to everyone in the community who has been sending us crash reports since our last post. These have been incredibly helpful for the team and are allowing us to narrow down the cause of the crashes.[/p][p][/p][h2]TLDR Summary[/h2]

• [p]The team has identified a likely cause of the crashes related to the PhysX physics engine.[/p]
• [p]These crashes can affect both servers and players, often resulting in multiple players disconnecting at once.[/p]
• [p]During our testing today, we have seen a client crash ourselves when connected to a live server, allowing for more targeted investigation.[/p]
• [p]Some additional crashes may be related to DX12 compatibility issues. Updating drivers or switching to DX11 may help.[/p]
• [p]The team continues to investigate and will provide further updates as progress is made.[/p]
• [p]If you can reproduce the crash, then please reach out to us.[/p]

[p]If any members of the community have been able to reproduce crashes consistently, we would greatly appreciate you sharing the details with us. Knowing how to trigger the issue would significantly speed up our efforts to identify and fix it.[/p][p][/p][h2]Status of the Investigation[/h2][p]From our investigations so far, we have determined that the problem is likely related to a complication within the PhysX physics engine used in Hell Let Loose.[/p][p]There appears to be a crash within the PhysX library that can manifest in several ways. It can cause both servers and connected clients to crash. When this happens, multiple players may crash simultaneously, which explains the instances where groups of players drop out of matches at the same time.[/p][p]This issue has not occurred in our development environment, which makes it more challenging to diagnose. We are working hard to understand what is causing it to occur in the live game, where there are many more variables than in testing.[/p][p]During our testing today, we have seen a client crash ourselves when connected to a live server. Now that we have seen this, we are able to get a number of developers to attempt the same with development tools attached to analyse what in-game scenario triggers the crash within the PhysX library.[/p][p]We hope this breakdown helps provide some clarity on the situation and where we currently stand in our investigation, and as we asked at the start, if anyone has found ways to reproduce this crash consistently do reach out and let us know.[/p][p][/p][h2]Other Crashing Reports - DX12[/h2][p]While reviewing crash reports, we have also seen a number of reports related to DirectX 12 (DX12).[/p][p]In Patch 17.1 the game was changed to use DX12 by default, where it previously used DX11. For players with compatible graphics cards, this provides a performance improvement.[/p][p]However, some graphics card drivers and certain GPUs may experience instability when running DX12, which can lead to crashes.[/p][p]If you are experiencing crashes, please first ensure that you are using the latest graphics card drivers for your system.[/p][p]If crashes continue, you can switch the game to DX11 mode by going to the Game Options menu and changing the “Renderer” setting at the bottom of the Video section to DX11.[/p][p][/p][p][/p][h3]Tue 21 Oct Update:[/h3][h3][dynamiclink][/dynamiclink][/h3]

[h2]Server and Stability Update - U18 Statement[/h2][p]We want to acknowledge the ongoing issues some players are experiencing, including server instability and client-side crashes following Update 18 and its recent hotfix. We recognise how frustrating this can be and want to assure everyone that we are fully focused on resolving these problems.[/p][p][/p][p]At this time, our investigations at this time indicate that these issues are not related to any of our server hosting partners. We’re working closely with QONZER and other providers, who continue to support us as we dig deeper into the root causes. Our team is actively investigating and testing potential solutions.[/p][p][/p][p]As soon as we have a more comprehensive understanding of the situation, we’ll share an update with the community. You can expect to hear more from us as our investigation progresses.[/p][p][/p][h3]Crash Forum[/h3][p]That being said, support from the community would really aid our investigations. We ask that if your game crashes, we ask you to go to either visit the new⁠ crashing reports forum within the Hell Let Loose Discord, or reply in the comments below, and let us know what was happening when your game crashed. This will be invaluable to our investigations.[/p][p]Please can we ask that if your game crashes, we ask you to go to this forum and let us know what was happening when you crashed. This will be invaluable to our investigations [/p][p][/p][p]Info needed in the feedback post should include[/p]

• [p]Description of what you were doing when it crashed.,[/p]

[p]Please see examples of what to include in the description below:[/p]

• [p]Were you in game at the time of the crash,[/p]
• [p]Were you shooting at something,[/p]
• [p]What weapon were you using,[/p]
• [p]Were you in a vehicle,[/p]
• [p]etc.,[/p]

• [p]Number of players in game at the time of crash,[/p]
• [p]What map were you on,[/p]
• [p]Were tanks being actively used in the game at the time of the crash?,[/p]
• [p]What server were you playing on?,[/p]
• [p]Please include anything else you think would be useful,[/p]
• [p]Feel free to add any images or videos of the crash/report[/p]

[p][/p][p][/p][p]Finally, we’d like to thank everyone who has jumped back in to play since U18 and the hotfix, and for all your continued support so far in addressing these issues.[/p][p][/p]

[p]We want to thank everyone for your patience and understanding while we worked to get this hotfix ready. Your feedback since the launch of Update 18 has been invaluable and continues to help us improve the experience.[/p][p][/p][p]Read the full list of fixes and changes below:[/p][p][/p][p][/p][h2]Changelog - Hotfix 1[/h2][p][/p][h3]General Bug Fixes[/h3]

• [p]Enemy flares can be seen on the tac-map by the opposing team[/p]
• [p]All Recon Tanks and Half-Truck wheels can not be repaired with a blowtorch[/p]
• [p]Killed In Action screen shows an issue when killed by environmental fire[/p]
• [p]Kubanka hat has incorrect ribbon model and colouring[/p]
• [p]Panther tank has a very difficult time dealing with terrain variance[/p]
  • [p]Addressed reports of collision inconsistencies and reduced performance with the Panther. Adjustments have been made to the vehicle’s suspension behaviour and physics collision system to align its performance with its original specifications.[/p]
• [p]Artillery Strike can trigger the “Rain Hell” achievement[/p]
• [p]Tank skin preview can be seen through geometry to all players[/p]
• [p]The recoil on the Luchs is excessive, making it difficult to aim as an autocannon[/p]
• [p]Following the implementation of the new tank knockback, the Luchs experienced excessive recoil due to its high rate of fire. Adjustments have been made to the recoil and knockback force to restore proper control during rapid fire.[/p]

[p][/p][h3]Artillery Strike Balance Adjustments[/h3]

• [p]The commander artillery strike ability has had several balance changes:[/p]
  • [p]The strike zone radius has been slightly increased, making the shots less accurate[/p]
  • [p]The number of shells that fall has been decreased from 16 to 12[/p]
  • [p]The munition cost per placement has been increased to 125[/p]
  • [p]The cooldown time has been increased to 430 seconds[/p]
• [p]On top of this, the model used has been adjusted and the smoke trail removed[/p]
• [p]Artillery Strike ability can take out spawn points[/p]
  • [p]The commander artillery strike ability can no longer destroy garrisons[/p]

[p][/p][h3]Armor Tank Balance[/h3]

• [p]Following playtesting and community feedback, it was determined that tank handling was not performing as intended, particularly in higher gear ranges. The balance between applied braking and thrust during steering has been adjusted to deliver a smoother and more responsive experience.[/p]
• [p]To continue the Armor update, all tanks have undergone suspension and physics collision updates. These adjustments aim to reduce issues with vehicles becoming immobilized on uneven terrain or smaller objects, particularly those located directly beneath the tank. This change is a part of an ongoing effort to enhance Armor and provide a more consistent driving experience for future updates.[/p]

[p][/p][h3]Gameplay Balance[/h3]

• [p]Following player feedback and testing, several gameplay balance refinements have been made:[/p]
  • [p]Blowtorch Repair Speed: Increased slightly, improving repair efficiency on all tank components[/p]
  • [p]Anti-Tank Launcher Damage: Increased from 325 to 380. This adjustment preserves the same number of required hits to destroy tanks while heightening the threat level to Armor units[/p]
  • [p]Anti-Tank Gun Damage: Increased from 350 to 400. This adjustment makes these weapons more potent against all tanks; particularly heavy tanks which can now be destroyed in fewer hits compared to the anti-tank launchers[/p]

[p][/p][p]We’re also aware of other issues reported by the community, including client-side crashes, and our team is actively investigating them. Please continue to share any problems you encounter with the support team at [/p][p]https://support.team17.com/hell-let-loose/en/[/p][p][/p]

[p]❗️AWS & EOS Outage, & Hotfix Update[/p][p]We're aware of a major AWS (Amazon Web Services) outage, alongside the EOS outage, which is causing players to not be able to access games & other web-based services.[/p][p]
[/p][p]We're actively monitoring the situations with both parties and we are hoping for a resolution to the problems as soon as possible.[/p][p]
[/p][p]We are still aiming to deliver the Update 18 Hotfix 1 today, however we will keep you updated as to the status of the AWS and EOS issues which may delay release. Thank you for your patience and we hope to deliver the hotfix as soon as we can.[/p]