Devlog 027 - Improving the Simulation
Hey everyone!
We got a lot of feedback on the performance on of shapez 2 since the Early Access launch. There are a couple of areas where significant improvements could be made. A portion of these – including multithreading – will arrive with Update 0.0.9, which is set to go live after a short period on the experimental branch. If you're interesting in being part of this process, be sure to join our Discord to try it right now!
For today, Nicko will walk you through how the simulation in shapez 2 currently works and some of the improvements we're working on! It's quite a technical blog, but hopefully the images will help you visualize everything. But first, some news!
[h2]News[/h2]
[h3]Steam Awards[/h3]
The Autumn Sale brought along the annual Steam Awards! Be sure to vote for your favorite games from this year, though we'd like to recommend shapez 2 for no particular reason.
You can vote for shapez 2 as many times as you want, in any category besides Labor of Love. The Sit Back and Relax reward is a great fit, alongside Game of the Year of course!
[h3]→ Cast your votes here! ←[/h3]
[h3]Bundle Sale[/h3]
We've partnered up with Youthcat Studio and Gamirror Games to bring you the Automation Pipeline Master bundle, giving you a 10% discount on shapez 2 and Dyson Sphere Program! Whether you already own one of the games or still have both of them on your wishlist, you can expand your library and try two different takes on factory building in space.
Keep in mind, the 10% bundle discount stacks with any other discounts! During the Autumn Sale, shapez 2 is 25% off, Dyson Sphere Program is 20% off and you get an additional 10% off through the bundle, marking the lowest price yet for both games.
The bundle is available until December 4th, 2024!
https://store.steampowered.com/bundle/46765/Automation_Pipeline_Master/
[h3]Disclaimer[/h3]
As per usual, everything in this devlog is subject to change and things may end up working out differently than expected!
[hr][/hr]
[h2]Devlog 027[/h2]
In shapez 2, the core gameplay is quite simple: The player places a building and then it starts working. However, quite some effort is necessary to make all the buildings that are placed work together as the player expects.
Besides the pure functionality of a factory, performance is a key element for shapez 2. The goal has always been to enable the player to build massive factories, in an enjoyable way. From a technical point of view, this means that shapez 2 must be capable of simulation millions of buildings while maintaining a stable framerate and must react fluently when the players make changes to their factories.
We will first have a look at the state of development when the shapez 2 Demo was released, then the state of development of the current shapez 2 Early Access version, and finally the improvements with the upcoming 0.0.9 and 0.1.0 release. Let’s go!
[h2]Past[/h2]
During the development of the shapez 2 Demo, we focused on precise functionality and a sustainable way to create new buildings quickly. This time was not about scale, but to implement a solid basis which we can easily improve later.
[h3]Basics[/h3]
All buildings are assembled from common components like belts, pipes, and fluid containers. In addition, most buildings also have some custom simulation logic.
For example, the Rotator consists of three successive belt lanes. The input lane on the left and the output lane on the right are visualized as conveyors and will just move shapes forward. The processing lane at the center is visualized as a circle. It has a length of zero and does not move the shape. Instead, it contains the custom simulation logic that performs the rotation of the shape.
[h3]Connections[/h3]
Each building also defines input and output connectors. For example, the entry to the Rotator’s first belt lane is defined as its input connector, and the exit of its last belt lane is defined as its output connector.
Whenever a building is placed, we check if adjacent buildings have compatible connectors at the same edge. If we find a compatible connector, we connect the two buildings.
In the example, the incoming Conveyor on the left will hand over shapes to the Rotator, as soon as they reach the end of the conveyor’s belt lane. The Rotator will hand over shapes to the outgoing Conveyor when they reach the end of the Rotator’s output belt lane.
[h3]Update Order[/h3]
When the simulation is updated, it will move forward all shapes on any belt lanes in the factory. To move a shape forward, there must be enough space in front. However, on a full belt, there is no logical space between the shapes, even though we render the shapes with a gap for visual clarity.
In the example, you see five shapes and their respective, logical size. None of the shapes could be moved forward because there no space in front of them.
To be able to move the shapes forward anyway, we must update the buildings and their components in the right order. Therefore we start updating at the end and finish updating at the start of a shape’s path. This usually means that we move the shapes at the entrance to the Vortex first and the shapes at the Extractors last.
For instance, the Conveyor at Rotator’s output is updated before the Rotator. The Rotator then updates its output, processing, and input lanes - in this order. Finally, the Conveyor at the Rotator’s input is updated after the Rotator.
With these requirements, we can compute an optimal update order for any setup of buildings.
If you're interested in learning more about our early optimization efforts, be sure to check out Devlog 011!
https://steamcommunity.com/games/2162800/announcements/detail/3710460746137286715
[h2]Present[/h2]
During the development of the shapes 2 Early Access version we focused on scale. We had to improve the performance of a running factory and improve the performance when players make changes to their factory. We also had to find solutions for some special requirements.
At the release of the shapez 2 Demo, controlling the Belt Launchers and Belt Catchers was one of the biggest issues. Unlike the other buildings, they had to behave differently depending on the 'constellation' – the relative positioning of the buildings. Binding the custom simulation logic directly to a building did not work out well in these cases.
[h3]Pattern Matching[/h3]
For the Early Access, we introduced pattern matching to decide which simulation logic is applied to each building. This means, that whenever a building is placed, we can first check the building’s surroundings and then decide on one of multiple available custom simulation logic.
We can also aggregate multiple buildings into a single simulation logic. This grants precise control of building behaviors in different constellations and opportunities for performance optimizations.
As an example, a Belt Launcher without a corresponding Belt Catcher results in a simulation logic that blocks incoming shapes. A Belt catcher without a corresponding Belt Launcher results in a simulation logic that doesn’t do anything. Only a Belt Launcher with a matching Belt Catcher results in a single simulation logic for both buildings together, capable of throwing shapes from one building to the other.
We can also make use of pattern matching to improve performance. For example, we now aggregate all directly connected Conveyors into a single simulation logic. This reduces the required computations dramatically.
[h3]Update Graph[/h3]
Whenever buildings are added or removed, the update order has to be recomputed. This is a very expensive process. So we searched for a way to reduce this effort.
For Early Access, we no longer compute the update order directly. Instead, we maintain a directed graph of simulation logic that determines the update order. This is much faster, as we usually only need to attach or detach a single node to the graph. More expensive computations are necessary only when the player makes big changes to the graph, like placing a big blueprint.
In the example, you can see how the connections between the simulation logic of of all placed buildings create a graph.
[h3]Clusters[/h3]
Another benefit of a graph is that we can identify isolated subgraphs. These are parts of the graph that are not connected to other parts of the graph.
In the example, you can see that the Update Graph consists of three isolated subgraphs (blue, red, and green). To visualize a subgraph while playing, select a building and press the “Select Connected” hotkey “O”.
For shapez 2, we move these subgraphs into structures we call clusters. We can benefit from these Clusters in multiple ways.
First, even if we need to recompute an Update Order completely, we only need to do it for one cluster ignoring all other clusters. This especially improves the flow of the game when you place bigger blueprints. We can now also define an individual update behavior for each single cluster.
Update Frequency
Before introducing the clusters, we updated each simulation logic in every frame.
With the introduction of clusters, we don’t do this anymore. Clusters that are far away receive an update only every few frames. And clusters that are out of view are updated only three times per second.
During the endgame of shapez 2, players currently easily place up to 500.000 buildings in their factories, resulting in several thousand clusters. Only the handful of buildings in view must be updated every frame, allowing the Early Access version to support about 20 times bigger factories than in the Demo version.
[h2]Future[/h2]
Heading towards shapez 2 Update 1, we will further improve the performance of the game to finally support truly massive factories. The 0.0.9 release – set to come very soon – will give you a taste of the future scale of shapez 2. Here are some of the things included in the update:
In the general settings menu, you will find two additional settings in the simulation section that enable new performance features.
[h3]Simulation Threads[/h3]
Clusters don’t interfere with other clusters during their update. This means we can make use of all available CPU cores and update multiple clusters at the same time.
In the example above, you can see how using a second core already improves the simulation performance by 100%. Depending on your hardware and the number of other processes running on your computer while playing shapez 2, this can speed up the simulation update by up to 2000%. AMD CPUs especially should see big improvements, as they tend to have a lot of cores. Again, join our Discord if you'd like to try this update earlier ;)
[h3]Parallel Rendering[/h3]
During the game’s update loop, we need to do several things. By far the most expensive are the simulation update and the rendering.
Already during the development of the shapez 2 Demo, we decoupled the actual rendering from the gathering of all the information for the rendering. Therefore, the rendering is already decoupled from the simulation update. This made it a relatively small step to do the simulation update in parallel to the rendering. Depending on your hardware setup, this may double your performance as well.
[hr][/hr]
We hope you enjoyed this devlog and maybe even learned something new! Soon, some of these changes will go live and hopefully give you a significant boost in performance. See you then!
~ Nicko & the shapez 2 Team
[h3]Join the community:[/h3]
Discord Reddit Suggestions Portal X / Twitter YouTube
https://store.steampowered.com/app/2162800/shapez_2/
We got a lot of feedback on the performance on of shapez 2 since the Early Access launch. There are a couple of areas where significant improvements could be made. A portion of these – including multithreading – will arrive with Update 0.0.9, which is set to go live after a short period on the experimental branch. If you're interesting in being part of this process, be sure to join our Discord to try it right now!
For today, Nicko will walk you through how the simulation in shapez 2 currently works and some of the improvements we're working on! It's quite a technical blog, but hopefully the images will help you visualize everything. But first, some news!
[h2]News[/h2]
[h3]Steam Awards[/h3]
The Autumn Sale brought along the annual Steam Awards! Be sure to vote for your favorite games from this year, though we'd like to recommend shapez 2 for no particular reason.
You can vote for shapez 2 as many times as you want, in any category besides Labor of Love. The Sit Back and Relax reward is a great fit, alongside Game of the Year of course!
[h3]→ Cast your votes here! ←[/h3]
[h3]Bundle Sale[/h3]
We've partnered up with Youthcat Studio and Gamirror Games to bring you the Automation Pipeline Master bundle, giving you a 10% discount on shapez 2 and Dyson Sphere Program! Whether you already own one of the games or still have both of them on your wishlist, you can expand your library and try two different takes on factory building in space.
Keep in mind, the 10% bundle discount stacks with any other discounts! During the Autumn Sale, shapez 2 is 25% off, Dyson Sphere Program is 20% off and you get an additional 10% off through the bundle, marking the lowest price yet for both games.
The bundle is available until December 4th, 2024!
https://store.steampowered.com/bundle/46765/Automation_Pipeline_Master/
[h3]Disclaimer[/h3]
As per usual, everything in this devlog is subject to change and things may end up working out differently than expected!
[hr][/hr]
[h2]Devlog 027[/h2]
In shapez 2, the core gameplay is quite simple: The player places a building and then it starts working. However, quite some effort is necessary to make all the buildings that are placed work together as the player expects.
Besides the pure functionality of a factory, performance is a key element for shapez 2. The goal has always been to enable the player to build massive factories, in an enjoyable way. From a technical point of view, this means that shapez 2 must be capable of simulation millions of buildings while maintaining a stable framerate and must react fluently when the players make changes to their factories.
We will first have a look at the state of development when the shapez 2 Demo was released, then the state of development of the current shapez 2 Early Access version, and finally the improvements with the upcoming 0.0.9 and 0.1.0 release. Let’s go!
[h2]Past[/h2]
During the development of the shapez 2 Demo, we focused on precise functionality and a sustainable way to create new buildings quickly. This time was not about scale, but to implement a solid basis which we can easily improve later.
[h3]Basics[/h3]
All buildings are assembled from common components like belts, pipes, and fluid containers. In addition, most buildings also have some custom simulation logic.
For example, the Rotator consists of three successive belt lanes. The input lane on the left and the output lane on the right are visualized as conveyors and will just move shapes forward. The processing lane at the center is visualized as a circle. It has a length of zero and does not move the shape. Instead, it contains the custom simulation logic that performs the rotation of the shape.
[h3]Connections[/h3]
Each building also defines input and output connectors. For example, the entry to the Rotator’s first belt lane is defined as its input connector, and the exit of its last belt lane is defined as its output connector.
Whenever a building is placed, we check if adjacent buildings have compatible connectors at the same edge. If we find a compatible connector, we connect the two buildings.
In the example, the incoming Conveyor on the left will hand over shapes to the Rotator, as soon as they reach the end of the conveyor’s belt lane. The Rotator will hand over shapes to the outgoing Conveyor when they reach the end of the Rotator’s output belt lane.
[h3]Update Order[/h3]
When the simulation is updated, it will move forward all shapes on any belt lanes in the factory. To move a shape forward, there must be enough space in front. However, on a full belt, there is no logical space between the shapes, even though we render the shapes with a gap for visual clarity.
In the example, you see five shapes and their respective, logical size. None of the shapes could be moved forward because there no space in front of them.
To be able to move the shapes forward anyway, we must update the buildings and their components in the right order. Therefore we start updating at the end and finish updating at the start of a shape’s path. This usually means that we move the shapes at the entrance to the Vortex first and the shapes at the Extractors last.
For instance, the Conveyor at Rotator’s output is updated before the Rotator. The Rotator then updates its output, processing, and input lanes - in this order. Finally, the Conveyor at the Rotator’s input is updated after the Rotator.
With these requirements, we can compute an optimal update order for any setup of buildings.
If you're interested in learning more about our early optimization efforts, be sure to check out Devlog 011!
https://steamcommunity.com/games/2162800/announcements/detail/3710460746137286715
[h2]Present[/h2]
During the development of the shapes 2 Early Access version we focused on scale. We had to improve the performance of a running factory and improve the performance when players make changes to their factory. We also had to find solutions for some special requirements.
At the release of the shapez 2 Demo, controlling the Belt Launchers and Belt Catchers was one of the biggest issues. Unlike the other buildings, they had to behave differently depending on the 'constellation' – the relative positioning of the buildings. Binding the custom simulation logic directly to a building did not work out well in these cases.
[h3]Pattern Matching[/h3]
For the Early Access, we introduced pattern matching to decide which simulation logic is applied to each building. This means, that whenever a building is placed, we can first check the building’s surroundings and then decide on one of multiple available custom simulation logic.
We can also aggregate multiple buildings into a single simulation logic. This grants precise control of building behaviors in different constellations and opportunities for performance optimizations.
As an example, a Belt Launcher without a corresponding Belt Catcher results in a simulation logic that blocks incoming shapes. A Belt catcher without a corresponding Belt Launcher results in a simulation logic that doesn’t do anything. Only a Belt Launcher with a matching Belt Catcher results in a single simulation logic for both buildings together, capable of throwing shapes from one building to the other.
We can also make use of pattern matching to improve performance. For example, we now aggregate all directly connected Conveyors into a single simulation logic. This reduces the required computations dramatically.
[h3]Update Graph[/h3]
Whenever buildings are added or removed, the update order has to be recomputed. This is a very expensive process. So we searched for a way to reduce this effort.
For Early Access, we no longer compute the update order directly. Instead, we maintain a directed graph of simulation logic that determines the update order. This is much faster, as we usually only need to attach or detach a single node to the graph. More expensive computations are necessary only when the player makes big changes to the graph, like placing a big blueprint.
In the example, you can see how the connections between the simulation logic of of all placed buildings create a graph.
[h3]Clusters[/h3]
Another benefit of a graph is that we can identify isolated subgraphs. These are parts of the graph that are not connected to other parts of the graph.
In the example, you can see that the Update Graph consists of three isolated subgraphs (blue, red, and green). To visualize a subgraph while playing, select a building and press the “Select Connected” hotkey “O”.
For shapez 2, we move these subgraphs into structures we call clusters. We can benefit from these Clusters in multiple ways.
First, even if we need to recompute an Update Order completely, we only need to do it for one cluster ignoring all other clusters. This especially improves the flow of the game when you place bigger blueprints. We can now also define an individual update behavior for each single cluster.
Update Frequency
Before introducing the clusters, we updated each simulation logic in every frame.
With the introduction of clusters, we don’t do this anymore. Clusters that are far away receive an update only every few frames. And clusters that are out of view are updated only three times per second.
During the endgame of shapez 2, players currently easily place up to 500.000 buildings in their factories, resulting in several thousand clusters. Only the handful of buildings in view must be updated every frame, allowing the Early Access version to support about 20 times bigger factories than in the Demo version.
[h2]Future[/h2]
Heading towards shapez 2 Update 1, we will further improve the performance of the game to finally support truly massive factories. The 0.0.9 release – set to come very soon – will give you a taste of the future scale of shapez 2. Here are some of the things included in the update:
In the general settings menu, you will find two additional settings in the simulation section that enable new performance features.
[h3]Simulation Threads[/h3]
Clusters don’t interfere with other clusters during their update. This means we can make use of all available CPU cores and update multiple clusters at the same time.
In the example above, you can see how using a second core already improves the simulation performance by 100%. Depending on your hardware and the number of other processes running on your computer while playing shapez 2, this can speed up the simulation update by up to 2000%. AMD CPUs especially should see big improvements, as they tend to have a lot of cores. Again, join our Discord if you'd like to try this update earlier ;)
[h3]Parallel Rendering[/h3]
During the game’s update loop, we need to do several things. By far the most expensive are the simulation update and the rendering.
Already during the development of the shapez 2 Demo, we decoupled the actual rendering from the gathering of all the information for the rendering. Therefore, the rendering is already decoupled from the simulation update. This made it a relatively small step to do the simulation update in parallel to the rendering. Depending on your hardware setup, this may double your performance as well.
[hr][/hr]
We hope you enjoyed this devlog and maybe even learned something new! Soon, some of these changes will go live and hopefully give you a significant boost in performance. See you then!
~ Nicko & the shapez 2 Team
[h3]Join the community:[/h3]
Discord Reddit Suggestions Portal X / Twitter YouTube
https://store.steampowered.com/app/2162800/shapez_2/
Tier 1 Converter
Imagine the red circle part to be a shape part that you can not produce otherwise.
Tier 4 Converter
Tier 1, 4, 6 & 7 Converters