Atmospherics System Refactor
Cross learning and tech sharing between our projects brings you this update
Contrary to what has been incorrectly suggested in some reviews, the multi-team nature of our studio works towards better products, and this update is an excellent example of that. After technical review of our unreleased train tycoon game (Art of the Rail) by ICARUS technical team members, we embarked on a new approach to multi-threading for some core systems that took better advantage of multiple cores, and reduced thread concurrency dependency. What this means is we were able to squeeze better performance and reduce technical complexity (think: opportunity for really hard to fix bugs) at the same time.
We are really proud of how we have built a studio that has different teams, but are still able to share the lessons and grow better overall - and we hope you see the results of that too. Our lessons, both good and bad, from ICARUS and Stationeers are flowing between our teams and projects as well as the unreleased games we are working on. Even if these unreleased games never came out, they prove valuable in us being able to test our systems and approaches that we can port to games like Stationeers.
It's a shame that the top review for Stationeers, draws the conclusion that multiple projects and teams in a studio is bad for any of the games when, as this update demonstrates, the reality is quite the opposite.
Highlights
- Localization is fixed
- Some versions of Linux could not run the dedicated server due to missing DLLs, this should now be fixed
- The atmospheric tick has been highly optimized and experiences much less slow down. On many saves the atmos tick was running only about once per second or worse. Now it will be running every 500ms. On top of this the room generation and atmos tick are now in a fixed sequential order.
- Fixed large IC scripts not being synced in multiplayer
[h3]Want to run a dedicated server? Much has changed! Checkout this guide on my github for more information[/h3]
Background on Threading
Threading, in a programming sense, can be thought as of making your computer multi-task something. That is obviously a good thing, especially so for a systems game like Stationeers, because your processor (CPU) has a lot of things to do. You'll hear us talk about a "frame", which is the time (typically measured in milliseconds or "ms" for short) which you have to construct the image that will be sent to the users monitor for display. If you take to long, or even do so at constantly shifting speeds, the game "lags". There are multiple reasons that this can happen which you will hear people talk about on forums, such as being "CPU bound" or "GPU bound". GPU is the graphics card (or integrated processor), a device heavily focused on doing enormous numbers of math calculations concurrently (at the same time). When I say enormous, I really do mean enormous. Modern GPUs conduct math operations at a truly staggering rate.
While it is obvious that multi-threading is good for games, it is typically also very bad for them as well. This is because when you make an operation concurrent, it is very difficult to get that working together with the other threads going on. Especially so with the "main thread" - which is the one your game is using to put everything together. Modern game engines such as Unity or Unreal nearly always have some multi-threading out of the box, such as physics (at least some parts of it), audio (again, some), and some file operations. The rest is up to developers.
Sequential Operation of Worker Threads in Art of the Rail
We're pretty proud that Stationeers has always delved deep into unlocking the potential of all those CPU cores on your computer. But in many cases, we would make one thread for one group of tasks. Take atmospherics, which operated on one thread but constantly calculated what needed to happen. In Art of the Rail, I had a similar approach to vehicles. I had a companion thread that would move vehicles along their road or rail constantly, at a regular tick rate. This however caused some eventual scale issues and also left me the constant headache of the main and vehicle threads (as well as other ones) tripping over each other.
Unity offers an out-of-the-box approach to solve this called ECS, but it has a lot of restrictions. So with Art of the Rail, I took the general approach and rolled a custom system. What it involved was rolling up a variable number (based on detected core numbers, settable by player in settings) of "workers", assigning the tasks between these workers, and executing all the workers at the same time, but only applying their results at the start of the next frame.
What this means is that we get to do things sequentially, avoiding the problems of trying to keep data congruency between the state of things on varying threads. Work happens while the game is running, but it is all reconciled at the beginning of a frame, so all threads always agree on the state of the game.
https://store.steampowered.com/app/1286190/Art_of_the_Rail/
Applying this to Stationeers
We were having tremendous difficulty fixing bugs associated with the Room Manager thread in Stationeers. The purpose of this thread was to generate rooms, but it caused a lot of weird state issues due to concurrency and data trampling with the atmospherics manager. Not to mention it would sometimes take an enormous amount of time to run.
There wasn't a clean way to solve this, so I proposed we try the Vehicle Worker implementation from Art of the Rail we discussed above. The situation is much more complex in Stationeers, however. In Art of the Rail the vehicles are very independent making them an excellent candidate for truly concurrent handling. Stationeers has multiple dependencies, so it ended up being a much more complex process.
Broadly the following tasks are threaded, and handled using a mix of dedicated threads or concurrent worker threads:
- Calculating the neighbors to mix with
- Caching all the data for the main thread
- Fire and burning
- Any new rooms that need to be calculated
- Atmosphere mixing between neighbors and networks
- Internal reactions in Atmospheres
- Pipe Atmospheres are handle specially
Thing atmosphere reactions are done on the main thread, because they touch a lot of different things.
[h3]The result is about four time faster than it was before, and it solved a whole lot of bugs associated with room creation issues.[/h3]
Order of operations for Game Logic is now
This will likely have impacts on your IC code, if you are tightly dependent on some of the order of operations and timing.
1. Room Generation
2. Atmospherics Simulation
3. Power Tick
4. Logic Chips/Devices Tick
5. IC Chips Tick
Dedicated Server development
A fair few fixes and changes have come through, mainly focused on Linux dedicated servers, but with some limited success due to issues with our build server. It is still not very pretty, and we are still working on this, but it is much better than before. We have also reached out to many of the Game Server Providers (GSPs) directly, to work with them to help better support our servers. This will be an ongoing focus for us. We plan to devote significant resources to this over the coming months.
Like what you see? Help out!
We're super excited about where to take Stationeers in 2022. If you are as excited as we are please consider leaving us a positive review - it means a lot to us. Also our DLC was made as a way for the community to support us, so consider grabbing that as well.
https://store.steampowered.com/app/1038400/Stationeers_Zrilian_Species_Pack/
https://store.steampowered.com/app/1038500/Stationeers_HEM_Droid_Species_Pack/
Changelog v0.2.3392.16643
- Fixed Respawning after dieing in an explosion stops respawned character from being able to move. (beta bug introduced rev.16602)
- Fixed Error spam thrown by Atmospherics thread when trying to calculate Temperature of a Zero volume atmosphere. Zero volume atmospheres are now deregistered if they accidently get created.
- Fixed Terrain Manipulator mining terrain rather than adding it (bug on beta branch only, introduced rev.16557).
- Fixed Rooms not being deregistered when a wall of a room is completely deconstructed. (introduced rev16566)
- Fixed Error thrown by TerrainLod.
- Fixed Server hangs when printing a Programming motherboard.
- Fixed case where smelting ore could cause occlusion thread to crash, causing disappearing pipes bug.
- Fixed Atmospheric Tablet reading incorrect values (due to occlusion thread not updating its local atmosphere).
- Fixed a number of errors thrown by atmospheric interactions happening when Main thread was trying to modify atmosphere data. These now use AtmosphericEventInstance system to handle these interactions.
- Fixed Error thrown by Audio Concurrency system when an ore was smelted while another ore was playing an impact sound.
- Removed GetAtmosphereWorld function. Use GetAtmosphereLocal() or SampleGlobalAtmosphere() for Reading on main thread. Use CloneGlobalAtmosphere() or GetAtmosphereLocal() for writing atmospherics data the atmospherics thread.
- Added new AtmosphericEventInstance functions for Grid operations (where an atmos does not yet exist) CreateEmpty, CloneGlobal, CloneGlobalAddGasMix, CloneGlobalRemoveEnergy. Also added Reset AtmosphericEventInstance for clearing a gas mix.
- Load game button no longer throws exceptions if there are no saves
- Stop world descriptions from disappearing when you leave the current game
- Fixed bug where certain values would cause multiple exceptions when hovering over pipe analyser
- Fixed Clients could sometimes fall through terrain when joining.
- Fixed Helmets offset on clients.
- Fixed Locked game at 60fps during initial scene loading.
- Changed Forced Terrain generations to run synchronously
- Fixed FrameRate Counter now runs in pause menu.
- Fixed FrameRate limiter no longer accepts out of bounds values.
- Fixed Simple, Good and Beautiful Graphics pre-sets turning on v-sync.
- Removed double logging in console window
- Removed Cell.CellLookup Dictionary, Was duplicating GridManager.GridCells.
- Fixed Bug where terrain renderers being turned off at incorrect time when playing as server.
- Fixed World atmospheres that are outside of rooms not being destroyed in a timely fashion.
- Fixed Some world atmospheres not having a reference to their cell.
- Fixed bug where languages were not showing and could not be selected
- Cleaned up help logging in console, made it oneline. Still a bit messy. Will make it better soon.
- Fixed Atmospheres in rooms in terrain having a volume less than 8000 when all terrain voxels appear to be mined out.
- Fixed exception thrown when intialising console overrides in ConsoleWindow
- Fixed DuctTape not fixing solar panels (and other things)
- Added clearing console once dedicated server is ready to receive input
- Fixed submitting large IC scripts in Multiplayer as server or client throws error spam and/or crashes the game.
- Added small optimisation for room generation
- Added interception of unity logs by our logging system so that we can deal with them individually
- Fixed -settingspath not working if file already exists. Removed existence check. File is created if not.
- Fixed Atmospherics and Logic Tick not pausing when in pause menu.
- Removed IMGUI completely from server builds.
- Fixed Terrain does not hold atmospheres. Rooms can now be reliably generated using terrain.
- Refactored Atmospherics, Power/Logic & Room Generation to run sequentially on a new GameTick locked at 500ms. Previously these different systems were running concurrently and at increasingly varying rates as bases grew in size.
- Order of operations for Game Logic is now: Room Generation > Atmospherics Simulation > Power Tick > Logic Chips/Devices Tick > IC Chips Tick.
- Added new worker thread system.
- Fixed World atmosphere being created when mining. Mining into terrain will now create an empty atmospheric cell that will pull in neighbour gas.
- World Atmosphere sometimes being spawned on loading a game in rooms created by terrain.
- Fixed slowdown of atmospherics processing in larger bases.
- Refactored Atmospherics Processing to allow for parallel processing of most atmospheric functions. Atmospherics runtime is now about 4x faster.
- Fixed FrameRate spikes when Clients were Reading and Processing Atmospherics data. This work is run on a separate thread.
- Fixed Moved Jetpack emissions handling to Atmospherics thread.
- Fixed submitting large IC scripts in Multiplayer as server or client throws error spam and/or crashes the game.
- Fixed Air conditioner buttons not interactable for clients.
- Fixed Explosion FX and sounds not playing.
- Added sounds to remote detonator.
- Added beeping sound to armed explosives.
- Fixed Explosions not mining terrain.
- Fixed Remote Detonator not working for Clients.
- Fixed Error when an explosion triggered.
- Fixed Explosions not removing clutter.
- Changed increased the radius of terrain mined by explosives to match the radius at which force/damage is applied.
