Last Oasis News

So the game works now. We managed to stabilize the backend and optimize our database, and everything has been running smoothly for the last week. We’re still doing changes and improvements, especially to accommodate private servers (more below!) without overloading the systems again. We’ll continue to work on it on the side while preparing our next large content update.

[previewyoutube][/previewyoutube]

[h2]New content update on the way[/h2]

With that update, you can expect a brand new Walker - Mollusc. It’s a small Walker that might not look like much at first, but it’s the first kind of a vehicle with a closed cabin that you could use to store chests or crafting stations, and even build Steering Levers to control your Walker from safety, while sacrificing visibility.

We will also be adding new varieties of maps for you to explore and conquer, overhauls to creature behaviors, and lots of other things. The Sun will also start obliterating oases, so get ready to travel East if you want to survive. In addition to that, it might be a good idea to look up occasionally, in case a giant rock is falling in the sky. When they collide with Earth, they create craters rich in iron and other minerals required for the most advanced technologies known to Nomads. Expeditions to craters might be the most dangerous events to date, so if you’re not sure you’re ready for them, it might be best to buy metals from more hardy Nomads until you’re prepared.

We’re hard at work on the update right now and will keep you posted with more information about it next week.



[h2]On the zerging problem[/h2]

In addition to preparing the large patch, we’re looking into the most reported issue so far - zerging, kill on sight, whatever you want to call it. Obviously, we’ve noticed how some clans choose to treat interactions with small groups and solos, hunting them down for a little loot or just fun. We believe a major cause for this behavior was the lack of high-tier clan interactions, locked in Hard maps until recently. Now that clans have access to claiming, we see that the interactions between clans and small groups/solos slowly get less hostile over time.

Since players inside a claim bring harvesting and trading bonuses only if they’re safe and unharmed, it encourages getting rid of pirates and not killing peaceful farmers and traders. Also, as clan wars over territories start to emerge, the targets shift, and attacking random nomads for some wood and fiber becomes unimportant.

But that’s not enough.

While we expect the interactions to become less randomly hostile with more and more clans transitioning from being looters to a “vassal” role, passively gaining wealth while fighting wars against other clans, we want to continue working towards encouraging clans to protect small groups and solo players. So far, we’ve designed the game to have really fun large-scale combat and that’s where we want it to shine for those large clans.

We’re currently collecting feedback and brainstorming ideas on how to iterate on an environment that’s fun to play in for everyone - be it large or small clans, groups or solo players. If you have suggestions on this, we’d love to see them on our forums or Discord!

As mentioned above, we’re also working on improving stability for when we launch private servers, which will be very soon. You’ll be able to own your own separate realm to play on together with friends or any other players if you decide to make it public. We’ll be announcing private servers going live as soon as we have a definitive date set. At first, you’ll be able to get your own realm set up with Nitrado, but we’ll also be working on adding more solutions for hosting later on.

Meanwhile, we’ve been releasing smaller patches and hotfixes, tweaking some things and getting rid of annoying bugs you’ve reported. We’ll continue doing so in-between larger content updates to make sure the game runs as smoothly as possible, but we wouldn’t be able to do it without your help. Thanks to everyone who reports issues, we’re able to quickly get rid of them and improve the game. If something is broken or even mildly annoying, please let us know and we’ll do our best to fix it!

TECHNICAL

• Improved server persistence to reduce rollbacks; mostly when restarting or patching.
  

BALANCE & TWEAKS

• Increased Barrier Plank's HP from 100 to 12000; actually useful now.
  
• Restricted building beds on walkers; too op bypassing respawn timer.
  
• Buffed general repairing value by 2; was too slow.
  
• Buffed walker water health repairing value by 5; was also slow.
  
• Improved oasis danger calculation; was possible to cheat the system.
  

**FIXES**

• Fixed Forager Module not harvesting foliage; finally.
  
• Fixed ability to sell Tablets at trading stations.
  
• Fixed unpacking sometimes being blocked because of base height.
  
• Fixed ammo displaying issues when manning weapons that have ammo racks.
  
• Fixed keybinding changes sometimes not applying to building mode.
  
• Fixed base rotation not being properly set when unpacking.
  
• Fixed containers sometimes failing clan ownership check.
  
• Fixed Torque value sometimes glitching when sprinting.
  
• Fixed display issues with claimed tiles on world map.
  
• Fixed server crash related to harvesting foliage with invalid data.
  
• Fixed client crash in offmap lobby when crew data wouldn't load properly.
  
• Fixed client crash when combat system would get invalid action.
  

[previewyoutube][/previewyoutube]

A thick layer of ice is melting, uncovering lush terrains full of unique vegetation and strange wildlife. Are you prepared to risk your life and possessions for a chance to craft rare Tablets, harvest some cotton or explore ancient ruins?

With a new type of oasis, there are many new dangers and opportunities waiting to be discovered. Make sure you’re ready for it, as there are new monsters lurking around and many dangerous pirates will surely try to be the first to rob the new lands off of their resources.

Hard difficulty oases will be of interest to anyone who wants to unlock technology of higher tier - more walkers, better gear, more efficient harvesting tools, and access to territory claiming. This will finally provide long-term goals to clans, create reasons to declare wars, and encourage protecting solo players and small groups with harvesting and tax bonus mechanics.

You will also be able to find various ancient ruins and structures to explore. Some allow crafting Tablets out of fragments, some have mazes to investigate with a promise of generous rewards, and some are large old cities attracting many nomads to fight over loot. New Rupu camps and loot sites to raid can also be found in those oases.

Not all resources can be found in new oases, so travel will continue to be essential. New kinds of vegetation and rocks, however, will allow for crafting more advanced equipment and complex machines. Some sturdy foliage requires high tier tools to be harvested, so make sure to explore your tech tree and unlock required ones.

New hard maps will start appearing in the world sometime tomorrow! In addition to a new type of oases, the patch brings fixes to some of the critical issues you’ve reported.

IMPROVEMENTS

- Improved usability of World Map.
- Made kill feed only show clan kills.
- Forbid packing Rupu Slings.


FIXES

- Fixed harvesting from Phemke and Nurr.
- Fixed streamer name anonymization sometimes showing real names.
- Fixed some foliage being difficult to harvest with many interactions nearby; cactus fruit, etc.
- Fixed subtitles not displaying if two voice lines are played in sequence.
- Fixed being able to flip enemy walkers.
- Fixed incorrect height adjustment when unpacking bases.
- Fixed building key hints not updating when keybind changes.
- Fixed building key hints not displaying when unpacking.

- Fixed crash related to packing while initiating player is missing.

[h2]What’s coming next - new contents and private servers[/h2]


We’re also working on a large content patch to include many gameplay improvements and even a new type of a walker.

As we’ve been set back with private servers by the much-needed work on the backend, it’s taking a little more time than previously estimated. We’re currently optimizing the system to make sure everyone is able to host a server and enjoy the benefits of owning your own oasis.


[h2]Community contest - #screenshotsaturday[/h2]

Starting this week, we want to invite you to participate in our permanent #screenshotsaturday contest!

Every Saturday we will select one or more of the best screenshots provided by the community and post them on our main Twitter account.

How to enter:

To participate, users have to post a screenshot publicly using one of the following methods:

*Steam Community: the screenshot must contain the hashtag #screenshotsaturday

*Twitter: the screenshot must tag our main page @oasisnomad and contain the hashtags #LastOasis and #screenshotsaturday

*Imgur: the screenshot must contain the tags “last oasis” (https://imgur.com/t/last%20oasis) and “screenshotsaturday” (https://imgur.com/t/screenshotsaturday)

Test has been going very well so far! With the exception of some small issues that affected individual players or servers, which we're investigating and will be fixing soon, everything is stable and working as intended.

Seeing that many of you are enjoying the game, we've decided to keep the servers running overnight, and if no major issues appear during that time, we will not be wiping your progress or shutting servers down anymore.

Thank you everyone for helping us with the test and all the support you've shown over the last week!



UPDATE

We're working on a content patch, which should be ready next week, and we'll be adding an ability to change your character's name and appearance soon. We will also be reworking the character deletion feature, but it will be coming later, as we'd like to make it a stable as possible. Unfortunately, it was one of the major reasons for the backend issues in the past.

To some of you who selected different starting zones than your friends, you can already take care of it by clicking Respawn in the menu, switching to World Map in the respawning screen, and selecting an oasis you'd like to respawn on. You can choose to respawn on any oasis (respawning on other cradles is only possible if you haven't left one). You will lose your inventory and will have to leave your walker behind, but you'll be instantly teleported to your friends. If you cannot see your friends' oasis, it might be because it's too far away North/South (over 50 tiles), but you can do multiple respawns to get there.

As we announced this morning, we are going to bring the servers up for a public test for some time. The stress test will start today at 8 PM CET / 11 AM PST.

To join, all you have to do is wait for the update on Steam and launch the game as you would normally.

Due to the data corruption that occurred in our database, the game will start fresh with a new world and a full wipe.

You can expect some longer queue times, as we have changed things so it doesn't overload our systems. We'll be addressing the waiting times later, but the game should run normally otherwise. For testing purposes, there might be restarts every so often. We will keep you all updated as frequently as possible, and will consider the options we have throughout the test.

The purpose of this test is to verify that the fixes we applied throughout the week will allow us to re-launch the game into Early Access.

We'll be observing the performance of the test closely and decide afterwards how to proceed. If things run well, our plan is to restart the servers tomorrow. Should we encounter more issues, we'll have to take a closer look again.



[h2]What happened and what did we do about it?[/h2]

A lot of things went wrong at launch, and while we could've tackled them one by one, they all happened at the same time. We felt prepared for the launch and wouldn't have released yet otherwise. We ran a beta for over a year and we've been continuously fixing all the issues we've come across. On top of that, we did many large-scale load tests throughout the last year, which showed no problems in any of our systems.

As many of you saw, the first hours after the release went smooth. After some time, we started getting occasional reports of characters being stuck in the world map. Engineers proceeded to look into it, but it didn't seem like a widespread problem at the time, considering a very small number of those reports. Our original plan for the first days after release consisted of splitting the team into a dayshift and a nightshift, fixing problems quickly as they pop up, and introducing updates 24/7 until things ran stable. Soon after, however, we noticed that the number of reports were increasing, the backend became more and more unstable, and that the situation was getting progressively worse.

More and more people failed to connect and a large number of players were stuck in queues. We were pushing quick fixes out, but in such a short period of time, we couldn't have truly understood the problems without proper debugging.

In the middle of it all, the most crucial problem started occurring — servers were continuously shutting off.

To explain what that means, we'd like to dive a bit deeper into how our world system works.
In our game, every oasis represents one server. There is a massive database that makes sure that the world is consistent - a player that leaves one server has to successfully arrive on another server for this consistency to be guaranteed. Because of how the system works, the servers shutting down wasn't as much of a bug on its own as it was a database failsafe. It meant that something happened in the world that interrupted that consistency and whenever that happens, the server shuts down and reboots to make sure the last correct state is resumed (what many of you mostly referred to as a rollback).

At some point, the database became so slow with all the connections that the gameservers couldn't verify their consistency state on time anymore, forcing them to all shut down simultaneously. Again, it was a failsafe to make sure that the world doesn't become inconsistent and breaks the logic of the game in the long run.

The continuous shutdowns revealed another two errors that were previously unnoticed. First, our lobby and joining queues weren't optimized enough for tens of thousands of players connecting simultaneously, as, normally, players join in smaller waves over a period of time. Clients were sending massive numbers of requests to our backend to check for the status, essentially spamming it. Doing so shouldn't have overloaded the gameservers with the verification process, but the sheer amount of those requests did. The system in place couldn't handle the load. On top of that, with so many people stuck in the queue due to another bug, the clients were continuing to send those requests indefinitely.

Our systems were essentially stuck in a loop with multiple issues affecting each other. As all the servers were shutting down and restarting, over 20k people were trying to rejoin at the same time, leading to our queue system failing, which then kept overloading the master server, letting only a few people to join a time until the master server would shut off again and take all servers down with it. And the cycle would continue.

A lot of you were understandably angry about this, so we were trying to get the systems to a working state and fix underlying problems at the same time. It felt like rebuilding a house of cards in the middle of a hurricane. We were hoping to find that magical solution that would interrupt the cycle at the time, waiting to see if the next patch fixes the main issue. Taking a lot of pressure and stress into account mixed with worsened communication due to the quarantine situation made things even more difficult.

At some point, we had to accept the reality that it isn't just one fix away. Too many systems failed simultaneously, which proved to be a much bigger problem for the release than we initially thought. A lot of you brought up that the decision to shut down the servers on Sunday came too late, and we agree. It was very difficult to pick the right time for the most nuclear option, as with so many patches coming in, we were asking ourselves: "What if we will have fixed it all in the next hour?".

After we announced the servers going down for maintenance, most of the team went to get some sleep after days of working non-stop and woke up to something we did not expect: a lot of you expressing understanding and support. We can't tell you how much of a morale boost that was for everyone on the team. It helped us to get our shit together: organize properly, analyze logs and code and fully check what went wrong. Over the last few days, we can easily say we found a lot. We've been refactoring large parts of the code that we've seen cause problems, changing the database structure that was failing, and fixing various integrity issues that appeared. As far as we can tell, all the known issues have been fixed.

The scary part for us, and we'd like to be as upfront as we can, is that we can't guarantee yet that it's fully solved. With so many issues at once causing related issues to appear, we know which bugs happened, but not which exact ones started the chain that led to the terminal problem. To stick with the house of cards example: all we see is that it collapsed, and now we need to reconstruct which card fell first, which we’ll only be able to start on once real players join the servers.

This is why we decided to start a public stress test today and verify all the fixes we did. We've also added additional server logging to make sure that we have better data to analyze the problems should new ones come up.

Some functionalities that we feel need more time to be properly refactored were temporarily disabled for now. We believe they were a large part of the source of the problems. Specifically, server decay, time-based structure packing, and deletion of your own character.
Once things are running smoothly again, we'll address those features as quickly as possible and bring them back in a more stable form.

The immediate goal, however, is to let you all play and enjoy Last Oasis again as soon as we can.