1. Last Oasis
  2. News

Last Oasis News

Good news about the server status

Test has been going very well so far! With the exception of some small issues that affected individual players or servers, which we're investigating and will be fixing soon, everything is stable and working as intended.

Seeing that many of you are enjoying the game, we've decided to keep the servers running overnight, and if no major issues appear during that time, we will not be wiping your progress or shutting servers down anymore.

Thank you everyone for helping us with the test and all the support you've shown over the last week!



UPDATE

We're working on a content patch, which should be ready next week, and we'll be adding an ability to change your character's name and appearance soon. We will also be reworking the character deletion feature, but it will be coming later, as we'd like to make it a stable as possible. Unfortunately, it was one of the major reasons for the backend issues in the past.

To some of you who selected different starting zones than your friends, you can already take care of it by clicking Respawn in the menu, switching to World Map in the respawning screen, and selecting an oasis you'd like to respawn on. You can choose to respawn on any oasis (respawning on other cradles is only possible if you haven't left one). You will lose your inventory and will have to leave your walker behind, but you'll be instantly teleported to your friends. If you cannot see your friends' oasis, it might be because it's too far away North/South (over 50 tiles), but you can do multiple respawns to get there.

Stress Test

As we announced this morning, we are going to bring the servers up for a public test for some time. The stress test will start today at 8 PM CET / 11 AM PST.

To join, all you have to do is wait for the update on Steam and launch the game as you would normally.

Due to the data corruption that occurred in our database, the game will start fresh with a new world and a full wipe.

You can expect some longer queue times, as we have changed things so it doesn't overload our systems. We'll be addressing the waiting times later, but the game should run normally otherwise. For testing purposes, there might be restarts every so often. We will keep you all updated as frequently as possible, and will consider the options we have throughout the test.

The purpose of this test is to verify that the fixes we applied throughout the week will allow us to re-launch the game into Early Access.

We'll be observing the performance of the test closely and decide afterwards how to proceed. If things run well, our plan is to restart the servers tomorrow. Should we encounter more issues, we'll have to take a closer look again.



[h2]What happened and what did we do about it?[/h2]

A lot of things went wrong at launch, and while we could've tackled them one by one, they all happened at the same time. We felt prepared for the launch and wouldn't have released yet otherwise. We ran a beta for over a year and we've been continuously fixing all the issues we've come across. On top of that, we did many large-scale load tests throughout the last year, which showed no problems in any of our systems.

As many of you saw, the first hours after the release went smooth. After some time, we started getting occasional reports of characters being stuck in the world map. Engineers proceeded to look into it, but it didn't seem like a widespread problem at the time, considering a very small number of those reports. Our original plan for the first days after release consisted of splitting the team into a dayshift and a nightshift, fixing problems quickly as they pop up, and introducing updates 24/7 until things ran stable. Soon after, however, we noticed that the number of reports were increasing, the backend became more and more unstable, and that the situation was getting progressively worse.

More and more people failed to connect and a large number of players were stuck in queues. We were pushing quick fixes out, but in such a short period of time, we couldn't have truly understood the problems without proper debugging.

In the middle of it all, the most crucial problem started occurring — servers were continuously shutting off.

To explain what that means, we'd like to dive a bit deeper into how our world system works.
In our game, every oasis represents one server. There is a massive database that makes sure that the world is consistent - a player that leaves one server has to successfully arrive on another server for this consistency to be guaranteed. Because of how the system works, the servers shutting down wasn't as much of a bug on its own as it was a database failsafe. It meant that something happened in the world that interrupted that consistency and whenever that happens, the server shuts down and reboots to make sure the last correct state is resumed (what many of you mostly referred to as a rollback).

At some point, the database became so slow with all the connections that the gameservers couldn't verify their consistency state on time anymore, forcing them to all shut down simultaneously. Again, it was a failsafe to make sure that the world doesn't become inconsistent and breaks the logic of the game in the long run.

The continuous shutdowns revealed another two errors that were previously unnoticed. First, our lobby and joining queues weren't optimized enough for tens of thousands of players connecting simultaneously, as, normally, players join in smaller waves over a period of time. Clients were sending massive numbers of requests to our backend to check for the status, essentially spamming it. Doing so shouldn't have overloaded the gameservers with the verification process, but the sheer amount of those requests did. The system in place couldn't handle the load. On top of that, with so many people stuck in the queue due to another bug, the clients were continuing to send those requests indefinitely.

Our systems were essentially stuck in a loop with multiple issues affecting each other. As all the servers were shutting down and restarting, over 20k people were trying to rejoin at the same time, leading to our queue system failing, which then kept overloading the master server, letting only a few people to join a time until the master server would shut off again and take all servers down with it. And the cycle would continue.

A lot of you were understandably angry about this, so we were trying to get the systems to a working state and fix underlying problems at the same time. It felt like rebuilding a house of cards in the middle of a hurricane. We were hoping to find that magical solution that would interrupt the cycle at the time, waiting to see if the next patch fixes the main issue. Taking a lot of pressure and stress into account mixed with worsened communication due to the quarantine situation made things even more difficult.

At some point, we had to accept the reality that it isn't just one fix away. Too many systems failed simultaneously, which proved to be a much bigger problem for the release than we initially thought. A lot of you brought up that the decision to shut down the servers on Sunday came too late, and we agree. It was very difficult to pick the right time for the most nuclear option, as with so many patches coming in, we were asking ourselves: "What if we will have fixed it all in the next hour?".

After we announced the servers going down for maintenance, most of the team went to get some sleep after days of working non-stop and woke up to something we did not expect: a lot of you expressing understanding and support. We can't tell you how much of a morale boost that was for everyone on the team. It helped us to get our shit together: organize properly, analyze logs and code and fully check what went wrong. Over the last few days, we can easily say we found a lot. We've been refactoring large parts of the code that we've seen cause problems, changing the database structure that was failing, and fixing various integrity issues that appeared. As far as we can tell, all the known issues have been fixed.

The scary part for us, and we'd like to be as upfront as we can, is that we can't guarantee yet that it's fully solved. With so many issues at once causing related issues to appear, we know which bugs happened, but not which exact ones started the chain that led to the terminal problem. To stick with the house of cards example: all we see is that it collapsed, and now we need to reconstruct which card fell first, which we’ll only be able to start on once real players join the servers.

This is why we decided to start a public stress test today and verify all the fixes we did. We've also added additional server logging to make sure that we have better data to analyze the problems should new ones come up.

Some functionalities that we feel need more time to be properly refactored were temporarily disabled for now. We believe they were a large part of the source of the problems. Specifically, server decay, time-based structure packing, and deletion of your own character.
Once things are running smoothly again, we'll address those features as quickly as possible and bring them back in a more stable form.

The immediate goal, however, is to let you all play and enjoy Last Oasis again as soon as we can.

Regarding Server Issues

[previewyoutube][/previewyoutube]

Our team of engineers has been working incredibly hard to resolve the bottleneck with our master server, which prevents people from logging into the game properly. The team is doing their best to implement solutions as quickly as possible, but as you have seen yourselves, the connection problems continue to appear as players join.

Therefore, we decided to take all servers offline to give our engineers time to investigate without new pressure for about 7 days.

If you are disappointed by this, we completely understand and ask you to request a refund from Steam - all refunds will be approved for the next 3 days, regardless of the time you spent in-game.

We hope to get back on track as soon as possible to allow everyone to enjoy Last Oasis. We'll keep you updated on the situation and announce the good news once we confirm the game is stable. We'd like to apologize once again for the rough launch and thank everyone who offered their support.

Service updates 101

With such a large number of players trying to play simultaneously, our current Master Server (backend/database) is unable to hold all the load at this moment. We are working around the clock to expand it and fix the known issues, but this may take some time.

You might experience crashes, bugs with traveling and safe logout, or other problems. We promise you that all of those issues will be resolved as quickly as they can be.

We'll share more information regarding this situation within the next 8 hours.


To ensure the quickest updates on the service status of our server, we created a dedicated Twitter account for live updates straight from our backend team.

[h2]Server Status Updates[/h2] https://twitter.com/lastoasishelp

Furthermore, we have established two platforms for putting you in direct contact with the support team.

[h2]Technical issues support[/h2] http://lastoasis.zendesk.com

(for all the issues related to client crashes, error messages, and so on)

[h2]Bug reporting[/h2] http://feedback.lastoasis.gg

(for letting us know about weird things you have experienced during the game)


Thank you for your patience and support,
the Donkey Crew team


Last Oasis now available on Steam Early Access!

Welcome to Last Oasis!

The Earth has stopped rotating on its axis, but not around the Sun. The only way to stay alive is to keep moving (quite ironic, given the times we live in). That’s how the world of Last Oasis is: a remote post-apocalyptic future in which humanity has become a nomadic society that relies on giant wooden walking machines to stay mobile and survive.

[previewyoutube][/previewyoutube]

Before you jump into the game, we wanted to take this opportunity to share some thoughts.

[h2]Where are we now and where are we going next[/h2]
For the first weeks, our focus is to keep working hard to solidify the core mechanics of Last Oasis. This means fixing balances where needed and kill any bugs whenever we may find them and ensure the stability of our server infrastructure. But this is just the beginning.

Last Oasis is a community-driven MMO experience. While we have our own vision on what we want our game to become, the whole development process has been constantly influenced by the wishes and suggestions of our players. For us, the Early Access release is just another step into the production of our Nomadic Survival MMO.

We are planning to stay in Early Access for up to two years while we keep expanding the game with loads of new biomes, creatures, walkers and mechanics. We want our game to be suitable for multiple survival-gaming playstyles, and we will continue to lead you on this journey and will always listen closely to whatever feedback you may have for us - after all, we are making this game for you.

Last but not least, we want to thank our beta testers: it’s been a long ride since we revealed Last Oasis to the public at the beginning of 2019, and we wouldn’t have made it this far without the help of thousands of passionate players that kept supporting the development of this title.

We often hang out with our passionate Discord community, and we would love to see more and more players joining it and helping to shape the game they want. You become part of it by clicking the link at the bottom of this post.

[h3]The time has come Nomads, see you in Last Oasis!
[/h3]