Patch 4.2.1 - Server Issues Fixed & Many Old Crashes Fixed
Hey everyone its been over 550 Days since the last patch! Unfortunately the servers started having issues so I had to do development on Valgrave for the first time in a year and a half.
First off, I want to apologize for the downtime. This was the first major downtime the game has experienced to my memory and that sucks. To make matters work it took me over a week to solve it. For that I am truly sorry. My increased knowledge, better documentation and logging for my future self, and more flexible build environment will reduce downtimes greatly in the future.
Secondly, I want to be clear that we won't be resuming active development on Valgrave but do intend to support it as long as we can even if it takes us time to fix like it did this time. I know that it sucks there aren't a ton of players playing but thank everyone who has continued to enjoy it regardless. I hope you have all been well. We'll still be working on our second game, Lodestar, which is a sci-fi co-op adventure game similar to a mix between Zelda and Subnautica with a bit of Satisfactory-lite. Early Access for this is set to begin in June.
Thirdly, I'm not 100% sure this patch is super stable but I'll be looking at it over the next couple of days for problems. It seems to be working with my rigorous testing and logging. Regardless if you run into any issues please hit up the discord to let me know and I'll probably be able to resolve it much faster with your help.
Thank you for your patience during this and hope you enjoy your time on the Immortal Plains!
Finally, for those who want a more detailed explanation of what happened its below and somewhat technical.
1) Our source control had a security vulnerability that got exploited many months ago. This caused us to turn off the server in AWS and put it in archived storage for when we needed it in the future.
2) We didn't take the time to stand-up a new copy which meant a couple days of trying to remember what was required to get source control back up and to do it safely. Additionally I am by no means an expert in AWS so I had to learn about a bunch of different features I had not used before.
3) The next problem was that we noticed the server was no longer able to get public access to the internet for Gamelift and Steam to communicate over (or so it seemed). This initially led us to believe a breaking change had occurred with Steam or Gamelift and we submitted tickets to each of them asking if perhaps a change had taken place. Eventually, after adding a ton of logging, we were able to deduce that one of the plugins we were using required special google servers to stay up and not change IP addresses. Yet after 550D they had and it was causing the GameServer to wait indefinitely on the last step of setup. We've moved over to a public API with a longer track record of success so it should not happen for many many years.
4) Then we hit the next problem that UWorks was no longer registering itself as the defacto OnlineSubsystem. So when players connected it would not know they were connecting from Steam and thus be unable to parse their player information. Resulting in a crash on the server during authentication. We're not sure exactly WHY this occurred but believe it may be due to either an updated OS on Gamelift or an updated VS compiling the project differently. This one took a bit longer to figure out but delaying the loading a very specific part of the code helped resolve this. (delaying all of UWorks led to more crashes)
5) Finally after shipping about 15-20 test builds over the course of a week we had resolved everything and were able to deliver this patch to you.
First off, I want to apologize for the downtime. This was the first major downtime the game has experienced to my memory and that sucks. To make matters work it took me over a week to solve it. For that I am truly sorry. My increased knowledge, better documentation and logging for my future self, and more flexible build environment will reduce downtimes greatly in the future.
Secondly, I want to be clear that we won't be resuming active development on Valgrave but do intend to support it as long as we can even if it takes us time to fix like it did this time. I know that it sucks there aren't a ton of players playing but thank everyone who has continued to enjoy it regardless. I hope you have all been well. We'll still be working on our second game, Lodestar, which is a sci-fi co-op adventure game similar to a mix between Zelda and Subnautica with a bit of Satisfactory-lite. Early Access for this is set to begin in June.
Thirdly, I'm not 100% sure this patch is super stable but I'll be looking at it over the next couple of days for problems. It seems to be working with my rigorous testing and logging. Regardless if you run into any issues please hit up the discord to let me know and I'll probably be able to resolve it much faster with your help.
Thank you for your patience during this and hope you enjoy your time on the Immortal Plains!
Finally, for those who want a more detailed explanation of what happened its below and somewhat technical.
1) Our source control had a security vulnerability that got exploited many months ago. This caused us to turn off the server in AWS and put it in archived storage for when we needed it in the future.
2) We didn't take the time to stand-up a new copy which meant a couple days of trying to remember what was required to get source control back up and to do it safely. Additionally I am by no means an expert in AWS so I had to learn about a bunch of different features I had not used before.
3) The next problem was that we noticed the server was no longer able to get public access to the internet for Gamelift and Steam to communicate over (or so it seemed). This initially led us to believe a breaking change had occurred with Steam or Gamelift and we submitted tickets to each of them asking if perhaps a change had taken place. Eventually, after adding a ton of logging, we were able to deduce that one of the plugins we were using required special google servers to stay up and not change IP addresses. Yet after 550D they had and it was causing the GameServer to wait indefinitely on the last step of setup. We've moved over to a public API with a longer track record of success so it should not happen for many many years.
4) Then we hit the next problem that UWorks was no longer registering itself as the defacto OnlineSubsystem. So when players connected it would not know they were connecting from Steam and thus be unable to parse their player information. Resulting in a crash on the server during authentication. We're not sure exactly WHY this occurred but believe it may be due to either an updated OS on Gamelift or an updated VS compiling the project differently. This one took a bit longer to figure out but delaying the loading a very specific part of the code helped resolve this. (delaying all of UWorks led to more crashes)
5) Finally after shipping about 15-20 test builds over the course of a week we had resolved everything and were able to deliver this patch to you.