Weekly Update: Feb 14, 2025
There was another minor patch this week to fix a couple of issues in the latest update, improve the new thermal visualization system, and add volunteer improvements to Russian and Ukrainian language translations.
Apart from that, I've been investigating a reported memory leak. Several users had reported increasing memory consumption previously, but this week a user described a specific scenario they identified as reproducible.
A memory leak is a situation where a program continuously allocates memory (i.e., creates new objects) over time without properly releasing memory that is no longer being used, eventually exhausting system resources.
Debugging memory usage is a bit tricky. For one thing, memory management works differently in the Unity Editor compared to a build, and this leak didn't seem to happen in the Editor so I had to create a new build to test each new hypothesis. For another, memory usage fluctuates under normal usage both from game related processes as well as the engine unloading resources that have not been accessed in a while.
Here is a snapshot of all objects at a point in time using the memory profiler:
Or rather, the first few dozen objects out of almost half a million.
There are a huge number of objects in memory at all times representing textures, models, vfx, strings, numbers, etc. Basically any data the game is using. (Sometimes data is used briefly and then no longer needed, in which case it should be "recycled" by the garbage collector. There is some performance cost associated with this, so generally one prefers recycling objects to repeated create and remove, but garbage creation is not the same thing as a memory leak as its memory is automatically recovered.)
Looking at the above snapshot, I have no way of immediately recognizing which data might be part of a leak. I did at least have some clues. First, it was probably either something that made big allocations infrequently, or small allocations very often, probably every frame at least. Big allocations every frame would have been identified before, and small allocations infrequently would never exhaust system resources, even if left running for days.
The user had said they had left the game open in the station, which eliminated a large number of possibilities, because most of the game's logic is paused while in the station. This meant that I might
be able to find it just by opening the station, taking a memory snapshot, leave the game running, then capturing another snapshot later and comparing the new objects.
I quickly identified one possible memory leak from screenshot thumbnail captures that occurred during automatic game saves, but this was too small and infrequent to be the culprit of a noticeable leak.
After some further investigation I identified some allocations that were the result of an operation that only occurred on the Operations screen (no pun intended), and was complex enough to potentially be hiding some bugs:
In order to display the command staff status and whether their "call button" should highlight, for each actor the conversation system needs to check if they have an active and valid conversation topic, whether it is marked "attract", and whether the player has seen it. This is moderately expensive, but since the Operations Panel is not a performance critical game state, I hadn't spent time optimizing it. I could see no obvious reason why this should leak memory, since all of its memory should be freed after each check. The calls do rely on the Lua interpreter, which is a third party library, so perhaps there was either a bug or interaction that I didn't fully understand.
Without knowing exactly where the memory leak was, I could not fix it, and I'd already spent many hours investigating it. However if it was somewhere within the "call button" check, I could at least mitigate it. There is not a strong need to check the actor button status literally every frame, since it changes infrequently. So I added a cached value and check the state only when the Operations Panel is entered, then every half second thereafter. At 60 frames per second, this would reduce the calls by 97% (and the leak by a corresponding amount if it is the culprit).
Some additional investigation suggests that this leak doesn't always occur on the Operations Panel. It's possible it only happens if the active conversation check has to invoke Lua, or maybe with some specific Lua callback function.
While I like to get to the root cause of bugs, having spent several days on this I may have to satisfy myself in minimizing its harm.
Until next week!
- Kevin
Apart from that, I've been investigating a reported memory leak. Several users had reported increasing memory consumption previously, but this week a user described a specific scenario they identified as reproducible.
A memory leak is a situation where a program continuously allocates memory (i.e., creates new objects) over time without properly releasing memory that is no longer being used, eventually exhausting system resources.
Debugging memory usage is a bit tricky. For one thing, memory management works differently in the Unity Editor compared to a build, and this leak didn't seem to happen in the Editor so I had to create a new build to test each new hypothesis. For another, memory usage fluctuates under normal usage both from game related processes as well as the engine unloading resources that have not been accessed in a while.
Here is a snapshot of all objects at a point in time using the memory profiler:
Or rather, the first few dozen objects out of almost half a million.
There are a huge number of objects in memory at all times representing textures, models, vfx, strings, numbers, etc. Basically any data the game is using. (Sometimes data is used briefly and then no longer needed, in which case it should be "recycled" by the garbage collector. There is some performance cost associated with this, so generally one prefers recycling objects to repeated create and remove, but garbage creation is not the same thing as a memory leak as its memory is automatically recovered.)
Looking at the above snapshot, I have no way of immediately recognizing which data might be part of a leak. I did at least have some clues. First, it was probably either something that made big allocations infrequently, or small allocations very often, probably every frame at least. Big allocations every frame would have been identified before, and small allocations infrequently would never exhaust system resources, even if left running for days.
The user had said they had left the game open in the station, which eliminated a large number of possibilities, because most of the game's logic is paused while in the station. This meant that I might
be able to find it just by opening the station, taking a memory snapshot, leave the game running, then capturing another snapshot later and comparing the new objects.
I quickly identified one possible memory leak from screenshot thumbnail captures that occurred during automatic game saves, but this was too small and infrequent to be the culprit of a noticeable leak.
After some further investigation I identified some allocations that were the result of an operation that only occurred on the Operations screen (no pun intended), and was complex enough to potentially be hiding some bugs:
In order to display the command staff status and whether their "call button" should highlight, for each actor the conversation system needs to check if they have an active and valid conversation topic, whether it is marked "attract", and whether the player has seen it. This is moderately expensive, but since the Operations Panel is not a performance critical game state, I hadn't spent time optimizing it. I could see no obvious reason why this should leak memory, since all of its memory should be freed after each check. The calls do rely on the Lua interpreter, which is a third party library, so perhaps there was either a bug or interaction that I didn't fully understand.
Without knowing exactly where the memory leak was, I could not fix it, and I'd already spent many hours investigating it. However if it was somewhere within the "call button" check, I could at least mitigate it. There is not a strong need to check the actor button status literally every frame, since it changes infrequently. So I added a cached value and check the state only when the Operations Panel is entered, then every half second thereafter. At 60 frames per second, this would reduce the calls by 97% (and the leak by a corresponding amount if it is the culprit).
Some additional investigation suggests that this leak doesn't always occur on the Operations Panel. It's possible it only happens if the active conversation check has to invoke Lua, or maybe with some specific Lua callback function.
While I like to get to the root cause of bugs, having spent several days on this I may have to satisfy myself in minimizing its harm.
Until next week!
- Kevin
