We're happy to share with you another devlog by
Rapha, this time about the recently implemented big save rework, why it was necessary and what went into it! Hope you enjoy reading it.
[h2]The Perils of Binary[/h2]
In the realm of video game development, preserving a player's progress is paramount. This is where game saves come in, capturing the state of a game so it can be resumed later. However, the methods used to store these save games can significantly impact their susceptibility to corruption, be it by external or internal factors.
[h2]The problem
[/h2]
100% Orange Juice since time immemorial used binary storage, it used to be the go to for this kind of data when processing power and storage space weren't plenty available. It's more efficient to write, to read, and uses less space in disk. But this efficiency comes at a cost, it's easy to mess up. It's why we've added more and more ways the game backups your saves at various points over time, to make sure players have some backup to return to if their save with years of progress gets corrupted.
Now let's imagine a theoretical game,
Poppo Saves The World! Let's imagine it as a beat-em-up, where you go around fighting enemies, one after the other, until you win and save the world. Lets supposed you have the following section of data that needs to be saved:
- Wins
- Points
- Last stage cleared
Lets supposed you have 10 victories, 393939 points and the last stage cleared as stage 7. Let's imagine you used 64 bit values for everything. So:
|Data | Explanation|
|--|--|
|0A 00 00 00 00 00 00 00 | 10 in hex padded to 64 bits|
|D3 02 06 00 00 00 00 00 | 393939 in hex padded to 64 bits |
|07 00 00 00 00 00 00 00 | 07 in hex padded to 64 bits |
In the file would be:
0A 00 00 00 00 00 00 00 D3 02 06 00 00 00 00 00 07 00 00 00 00 00 00 00
But then you introduce abilities to the game, you decide to store it as a single byte, that will keep if you have each of the 8 abilities available unlocked, one in each bit.
|8|7|6|5|4|3|2|1|
|--|--|--|--|--|--|--|--|
|0|0|0|0|0|0|0|0|
Let's suppose your player unlocked only the first one so it's a 1, so in binary:
|8|7|6|5|4|3|2|1|
|--|--|--|--|--|--|--|--|
|0|0|0|0|0|0|0|1|
You inadvertently decided to write it between wins and points, so you would have the new binary:
|Data | Explanation|
|--|--|
|0A 00 00 00 00 00 00 00 | 10 in hex padded to 64 bits|
|01 | 01 in hex, it's a single byte |
|D3 02 06 00 00 00 00 00 | 393939 in hex padded to 64 bits |
|07 00 00 00 00 00 00 00 | 07 in hex padded to 64 bits |
In the file would be:
0A 00 00 00 00 00 00 00 01 D3 02 06 00 00 00 00 00 07 00 00 00 00 00 00 00
And as logically implied, you would change your reading code to expect that too! But then comes the problem! Your old save doesn't agree. Let's see what your game would read from an old save.
0A 00 00 00 00 00 00 00 D3 02 06 00 00 00 00 00 07 00 00 00 00 00 00 00
It expects 8 bytes for the Wins, so it reads 8 bytes:
0A 00 00 00 00 00 00 00
That reads to 10! Seems ok! Up until now everything have gone correctly.
But then it tries to read the byte for abilities...
D3
D3 is byte, your game doesn't know if it is valid or not, it's binary, it believes in the binary. Let's look at it for a bit, in binary:
|8|7|6|5|4|3|2|1|
|--|--|--|--|--|--|--|--|
|1|1|0|1|0|0|1|1|
So not only the first, but the second, fifth, seventh and eighth abilities are unlocked?!? Wow! That's a problem.
Let's continue, more 8 bytes:
02 06 00 00 00 00 00 07
That's
504403158265497090 points... That's definitely not what was written originally. By reading that byte that wasn't there, you misaligned the whole thing, and now your player has more points than literally the universe has seconds of life. And then you continue, last 8 bytes:
00 00 00 00 00 00 00
But that's not 8, you only got 7! If you really try to read that, you will get an unexpected behavior. What happens depends on what system, what processor and what contingencies are in place. In some systems you may read a byte from memory that has nothing to do with your save, in others you may read some leftover on the memory bus, in some you will crash... The ideal behavior would be the crashing.
But lets supposed it didn't. It loaded.
Now you have:
10 victories
5 abilities unlocked
504403158265497090 points
And never cleared a single stageIf this happens to your user, they won't understand a single bit of what's happening, will they?
[h2]The conclusion[/h2]
You must be thinking now, "Well, that's easy, just don't mess up". Yeah, but that's easier said than done, with less than a dozen of information to save, and total control over it you can't commit this mistake.
But what about having 200000 different bits of data that need to be read and written correctly, by code written over 15 years, written by different people, with different designs in mind, written by functions in different files, some keeping proper spacing between data, with spare data for future use, some not. That was our situation! Years of updates adding numerous incongruities that we didn't account for, despite fighting against it all day.
Yeah, that's a design flaw, and that's why we are fixing it, starting with this step! 100% Orange Juice is a bigger game than one might imagine, and such huge code base being kept alive by less than 10 people is nothing less than a miracle driven by passion and willpower over the years.
[h2]The solution[/h2]
We won't enter into detail, because our new save system not only is 100% reliable but has a good number of protection layers against people editing it. But firstly, if your save was changed externally, it won't load.
This is to avoid the game crashing if your storage is faulty and some bit flipped. That's why the backups are now more important than ever, if your main save corrupted, it won't load, no one will be able to restore it, its gone forever. The game won't even try to read it.
But not only that, now generally speaking, each bit of data inside the save games has a proper name to address them.
Let's bring the example from before:
- Wins
- Points
- Last stage cleared
Now we build a database in memory, naming each value in a proper structure before writing it to the save. So it would be something like this:
wins: 10, points: 393939, last_stage_cleared: 7
For reading, you simply search for the value name and loads the value attributed to it: wins → 10, points → 393939, and it goes on.
If you add a new value:
wins: 10, abilities: 1, points: 393939, last_stage_cleared: 7
It doesn't matter, you aren't reading in sequence, you have a proper system to encode and decode the values and search them by name. If abilities doesn't exist on the save, you load the default value and go on.
This basically clears the future from unwanted data changes, but nothing in life is a perfect solution. We still need to load the old save, and if that's corrupted, the new one will be too, as the old save system trusted above everything else what was written, in the exact order it was.
The game also may after some update write something wrong, that's a bug, it can happen, it doesn't mean the save is corrupted stricto sensu, but in lato sensu the data was handled wrongly, so that's still a bug, that's still a thing that needs to be reported.
So if after the save conversion on the last big update, you got more dice than the universe has seconds of life, the best we can suggest is contacting us! We will do the best in our power to fix your old corrupted save. But be aware there may be some times that it's impossible (mainly if you didn't play for the last 7~10 years, the amount of uncounted problems can be beyond any repair), but even in this horrific situation, talk with us, say what you hold dear about your old save, and we might be able to find a path that will make you happy.
