DevBlog #2: Input to actions architecture 🐧
[h3]Hey Pingheros,[/h3]
Today, I want to share with you how we implemented our input handling stack using Unity and Photon Quantum for our online multiplayer brawler game Los Pingheros at Hectiq. Photon Quantum is a deterministic simulation engine that handles rollback, which allows to only send inputs over the network.
Even though the description and examples will be explained with Unity and Photon Quantum, most of the concepts are generalizable and help decouple inputs from your game logic.
[h2]Global architecture[/h2]
From pressing a button to seeing the character move in our game, the inputs go through different transformations.
If you follow the path from pressing a button to having the character do either a slap or a grab, it looks like this:
- Pressing Button X.
- Remapping Button X to Slap InputAction.
- Transforming Slap InputAction to Slap SimInput.
- Transforming Slap SimInput to a Slap CharacterAction or a Grab CharacterAction depending on the current state of the character as we plan to use the same input to trigger different actions.
Here is a bit of colourful animations of the slap and slide actions so you have a better idea of what is going on in the game.
[h2]Dynamic remapping (raw inputs to input actions)[/h2]
Unity “new” (well, not that new anymore…) input system allows you to bind raw inputs to actions. It allows to map several different buttons from different devices to the same callback.
In our example, raw inputs are directly mapped to input actions here. This helps iterating on your mappings quite easily as you do not have to write/edit code while you test different mappings. It also allows you to handle some low-level stuff such as dead-zones, quick button press vs hold, & so.
We did not do dynamic remapping yet, but if you plan to do it, that would be where you are supposed to plug it. Actions in Unity would become rawer, such as South, and you would then map this raw action to the associated input action such as Slide using a remapping table or something similar.
However, this leads in you needing to handle low level logic by yourself (e.g. the press vs. hold logic) as it is probably action dependant, not input dependant.
[h2]Input ingestion (InputAction to SimInput)[/h2]
The way input are ingested with Photon Quantum simulation is through a callback which is called at the beginning of each simulation frame. You fill up an input structure composed of buttons & Vectors that are send to each other clients.
On the Unity side, you hold an input struct that you fill up with every Unity’s input event. This struct is then polled from the simulation and reset for gathering inputs coming after.
Buttons are a bit more than booleans, they are accessed in the simulation through IsDown, WasPressed, and WasReleased properties. If you need some hold/quick press logic on the simulation side, you have to do it yourself. Even though filling the input struct is pretty straightforward, it comes with some important tricks that you might find useful.
[h3]Device specifics:[/h3]
We decided to add a boolean specifying if the input comes from a pointer type (mouse, trackpad, &so.). More information about this in the interpretation section.
[h3]Sub-frame press-release:[/h3]
There is a huge chance that the rate of your simulation is fixed, meaning that the input gathering rate from Unity and from your simulation are out of sync. This might lead to issues where you receive several raw inputs impacting the same sim-input in contradictory ways. The most impactful example of this we encountered is when you press, then release an input between two simulation input polling.
By pressing the slap input, you fill up your buffer input structure setting up the slap button to the “down” state, then you release it, putting the button back to the “up” state. This means that you should not use IsDown only when checking for inputs inside the simulation, as that would lead to you dropping the input entirely. More information in how to handle this in the interpretation section.
[h3]Sub-frame position precision:[/h3]
If your game and your player base is competitive, it is possible that they develop really good aim-click coordination (this is even more true with mouse players), and thus, the need to retain the click position of the exact time they clicked. Otherwise, you could end up with something like this:
Player clicks, leading to the slap input to be triggered, then continue to move their mouse, registering the current cursor position at a rate way higher that your simulation rate. You would then send a position input associated with the last recorded position, and you end up with the feeling that the button press is lagging.
To counteract this, we decided to separate the ActionDirection vector from the AimVector. The former represent the direction when the last action was triggered, while the later represents the latest recorded aiming position. This gives you the best of precision for both the action and your orientation.
[h2]Interpretation (Simulation inputs to character actions)[/h2]
The last step is to convert the inputs received by the simulation every frame into actions for your character to execute. This goes with problematics such as buffering inputs, or transforming the same input into different actions depending on the context.
In our game, a CharacterAction looks like this:
[h3]State based input transformation[/h3]
Those actions are decorrelated from the inputs and are as meaningful as possible for the character itself. The two main reasons for this are:
In our game, as often, the character has a state machine controlling what it is doing and what it can or cannot do. As a consequence, it appears important to take the current state of the character into consideration when interpreting inputs.
Here is an example to illustrate this: somebody grabbed you, so you cannot slap. In that case, you want to interpret the input as a mash action to help breaking free from your foe.
If we did the separation between slap & mash outside of the simulation, we would have used a state of the character at the time of the button press, that might not be the state of the character at interpretation time because of rollback or network latency. Thus, making our input drop.
All our states in the character state machine implement the InputsToCharacterAction method to interpret inputs depending on the context.
The following paragraphs are exemple on how we leverage this method to solve different problems.
[h3]Handling “Hold” inputs[/h3]
As stated previously in the Sub-frame press-release paragraph, it is not advised to use the IsDown property only as it can lead to inputs being missed. You would need to use it in combination with WasPressed or WasReleased this way.
Our slap action do not have any logic when keeping the button pressed. Thus, only using WasPressed works like a charm.
Our slide action though is continuous, and need the slide input to be held. As a result, we protect it from sub frame press-release or release-press by keeping the state of the previous input.
[h3]Device type branching[/h3]
Maybe you noticed that the Sim inputs have a MoveVector and an ActionDirection and that the AimVector from sim inputs has been laid off.
This is because during the interpretation phase, input vectors are becoming actions vectors with a logic that depends on the state and on the type of input device used.
Basically, with a controller, the MoveVector is controlled with the left stick and the AimVector is controlled with the right stick while ActionDirection is the AimVector at the moment where we pressed another input.
With a Keyboard/Mouse device, the MoveVector is controlled with WASD and the AimVector is controlled with the mouse while ActionDirection is determined by the mouse position at the exact moment we pressed another input.
But, when implementing certain features, we figured out that keyboard/mouse users sometimes use the mouse for aiming purposes, and sometimes for movement purposes. Sometimes, we also want the MoveVector to trigger another action only with the controller because it would feel weird with the mouse.
Thus, depending on the state, we may have something like this:
This allow to use the `MoveVector` action to control the slide independently of what actually triggered the action.
[h3]Handling input buffer[/h3]
In dynamic games, it is quite frustrating when one of your input is skipped because you pressed a button a frame or 2 before your character become activable. As we emphasis the decision making more than the frame-perfect execution, we implemented an input buffer system.
We implemented a serializable queue to store the inputs that are not consumed. It is up to you how you decide to implement that, though we might share how we did it on another snowy day.
We start by adding the last incoming input to the queue, and we dequeue inputs until we find what we call a meaningful input. If that is the case, we dequeue all inputs up to our meaningful input.
If your attention was sharp, you realized that the InputsToCharacterAction method presented previously was returning a boolean. This is is the way for the current state of the character state machine to tell if there is a meaningful input to be consumed.
This is an example from our Slide state. You can see that Slap or Slideinput would be consumed instantly, while another input such as Grab would not be meaningful for that state as we cannot grab something while sliding, and would thus be kept in the input queue if no meaningful inputs are found afterward in the queue. If we find a meaningful input later in the queue, the Grab input would be dropped.
Another subtility is that the MoveVector is always the interpretation of the last incoming input (this is done in the DirInputToCharacterActions method call before the inputQueue loop). And filling up the MoveVector for character actions is never going to be considered as a meaningful input. This way, buffering an input for an action has no effect on your movement.
[h2]Conclusion[/h2]
Coming up with this architecture for our inputs was not something that came in a day, but this gives us a lot of flexibility in how we develop our actions and how we control them. We are interested in what you think about our implementation, or in how you did architecture your input flow.
Don’t hesitate to leave us comments on what topic you might want to see on this devlog for the future.
Also, if you are interested in our project, you can follow us on social media, or come play with us on LosPingheros on Steam.
Don’t freeze your brains up ‘til next time, and meet us on the Mexican ice of Los Pingheros !
Romain GROSSE
Hectiq
Today, I want to share with you how we implemented our input handling stack using Unity and Photon Quantum for our online multiplayer brawler game Los Pingheros at Hectiq. Photon Quantum is a deterministic simulation engine that handles rollback, which allows to only send inputs over the network.
Even though the description and examples will be explained with Unity and Photon Quantum, most of the concepts are generalizable and help decouple inputs from your game logic.
[h2]Global architecture[/h2]
From pressing a button to seeing the character move in our game, the inputs go through different transformations.
If you follow the path from pressing a button to having the character do either a slap or a grab, it looks like this:
- Pressing Button X.
- Remapping Button X to Slap InputAction.
- Transforming Slap InputAction to Slap SimInput.
- Transforming Slap SimInput to a Slap CharacterAction or a Grab CharacterAction depending on the current state of the character as we plan to use the same input to trigger different actions.
Here is a bit of colourful animations of the slap and slide actions so you have a better idea of what is going on in the game.
[h2]Dynamic remapping (raw inputs to input actions)[/h2]
Unity “new” (well, not that new anymore…) input system allows you to bind raw inputs to actions. It allows to map several different buttons from different devices to the same callback.
In our example, raw inputs are directly mapped to input actions here. This helps iterating on your mappings quite easily as you do not have to write/edit code while you test different mappings. It also allows you to handle some low-level stuff such as dead-zones, quick button press vs hold, & so.
We did not do dynamic remapping yet, but if you plan to do it, that would be where you are supposed to plug it. Actions in Unity would become rawer, such as South, and you would then map this raw action to the associated input action such as Slide using a remapping table or something similar.
However, this leads in you needing to handle low level logic by yourself (e.g. the press vs. hold logic) as it is probably action dependant, not input dependant.
[h2]Input ingestion (InputAction to SimInput)[/h2]
The way input are ingested with Photon Quantum simulation is through a callback which is called at the beginning of each simulation frame. You fill up an input structure composed of buttons & Vectors that are send to each other clients.
input
{
bool IsPointer;
button Slap;
button Slide;
button SlideCharge; // Keyboard only
button Grab;
FPVector2 Move;
FPVector2 Aim;
FPVector2 ActionDirection;
...
}
On the Unity side, you hold an input struct that you fill up with every Unity’s input event. This struct is then polled from the simulation and reset for gathering inputs coming after.
Buttons are a bit more than booleans, they are accessed in the simulation through IsDown, WasPressed, and WasReleased properties. If you need some hold/quick press logic on the simulation side, you have to do it yourself. Even though filling the input struct is pretty straightforward, it comes with some important tricks that you might find useful.
[h3]Device specifics:[/h3]
We decided to add a boolean specifying if the input comes from a pointer type (mouse, trackpad, &so.). More information about this in the interpretation section.
[h3]Sub-frame press-release:[/h3]
There is a huge chance that the rate of your simulation is fixed, meaning that the input gathering rate from Unity and from your simulation are out of sync. This might lead to issues where you receive several raw inputs impacting the same sim-input in contradictory ways. The most impactful example of this we encountered is when you press, then release an input between two simulation input polling.
By pressing the slap input, you fill up your buffer input structure setting up the slap button to the “down” state, then you release it, putting the button back to the “up” state. This means that you should not use IsDown only when checking for inputs inside the simulation, as that would lead to you dropping the input entirely. More information in how to handle this in the interpretation section.
[h3]Sub-frame position precision:[/h3]
If your game and your player base is competitive, it is possible that they develop really good aim-click coordination (this is even more true with mouse players), and thus, the need to retain the click position of the exact time they clicked. Otherwise, you could end up with something like this:
Player clicks, leading to the slap input to be triggered, then continue to move their mouse, registering the current cursor position at a rate way higher that your simulation rate. You would then send a position input associated with the last recorded position, and you end up with the feeling that the button press is lagging.
To counteract this, we decided to separate the ActionDirection vector from the AimVector. The former represent the direction when the last action was triggered, while the later represents the latest recorded aiming position. This gives you the best of precision for both the action and your orientation.
[h2]Interpretation (Simulation inputs to character actions)[/h2]
The last step is to convert the inputs received by the simulation every frame into actions for your character to execute. This goes with problematics such as buffering inputs, or transforming the same input into different actions depending on the context.
In our game, a CharacterAction looks like this:
component CharacterActions
{
FPVector3 MoveVector;
FPVector3 ActionDirection;
bool Slide;
bool SlideCharge;
bool Slap;
bool Grab;
bool Mash;
...
}
[h3]State based input transformation[/h3]
Those actions are decorrelated from the inputs and are as meaningful as possible for the character itself. The two main reasons for this are:
to simplify the logic in the state machine.
to simplify the interfacing with the AI we use for bots. Indeed our bot system use the exact same character as the players, except for the logic that creates character actions every frame. Having meaningful character actions helps the logic understanding of AI development and make it resilient to changes linked to how we translate inputs to character actions.
In our game, as often, the character has a state machine controlling what it is doing and what it can or cannot do. As a consequence, it appears important to take the current state of the character into consideration when interpreting inputs.
Here is an example to illustrate this: somebody grabbed you, so you cannot slap. In that case, you want to interpret the input as a mash action to help breaking free from your foe.
If we did the separation between slap & mash outside of the simulation, we would have used a state of the character at the time of the button press, that might not be the state of the character at interpretation time because of rollback or network latency. Thus, making our input drop.
All our states in the character state machine implement the InputsToCharacterAction method to interpret inputs depending on the context.
public unsafe bool InputsToCharacterAction(Frame f, in Input input, ref CharacterActions characterActions, EntityRef e){}
The following paragraphs are exemple on how we leverage this method to solve different problems.
[h3]Handling “Hold” inputs[/h3]
As stated previously in the Sub-frame press-release paragraph, it is not advised to use the IsDown property only as it can lead to inputs being missed. You would need to use it in combination with WasPressed or WasReleased this way.
characterActions.Slap = input.Slap.WasPressed;
if (previousInput.Slide.IsDown)
characterActions.Slide = input.Slide.IsDown && !input.Slide.WasReleased;
else
characterActions.Slide = input.Slide.IsDown || input.Slide.WasPressed;
Our slap action do not have any logic when keeping the button pressed. Thus, only using WasPressed works like a charm.
Our slide action though is continuous, and need the slide input to be held. As a result, we protect it from sub frame press-release or release-press by keeping the state of the previous input.
[h3]Device type branching[/h3]
Maybe you noticed that the Sim inputs have a MoveVector and an ActionDirection and that the AimVector from sim inputs has been laid off.
This is because during the interpretation phase, input vectors are becoming actions vectors with a logic that depends on the state and on the type of input device used.
Basically, with a controller, the MoveVector is controlled with the left stick and the AimVector is controlled with the right stick while ActionDirection is the AimVector at the moment where we pressed another input.
With a Keyboard/Mouse device, the MoveVector is controlled with WASD and the AimVector is controlled with the mouse while ActionDirection is determined by the mouse position at the exact moment we pressed another input.
But, when implementing certain features, we figured out that keyboard/mouse users sometimes use the mouse for aiming purposes, and sometimes for movement purposes. Sometimes, we also want the MoveVector to trigger another action only with the controller because it would feel weird with the mouse.
Thus, depending on the state, we may have something like this:
if (input.Slide.IsDown && input.IsPointer)
if (input.ActionDirection != FPVector2.Zero)
characterActions.MoveVector = input.ActionDirection.XOY;
else
characterActions.MoveVector = input.Aim.XOY;
else
characterActions.MoveVector = input.Move.XOY;
This allow to use the `MoveVector` action to control the slide independently of what actually triggered the action.
[h3]Handling input buffer[/h3]
In dynamic games, it is quite frustrating when one of your input is skipped because you pressed a button a frame or 2 before your character become activable. As we emphasis the decision making more than the frame-perfect execution, we implemented an input buffer system.
component BufferedInputs
{
SerializableQueueMetadata QueueMetadata;
array[6] QueueStorage;
}
We implemented a serializable queue to store the inputs that are not consumed. It is up to you how you decide to implement that, though we might share how we did it on another snowy day.
var inputQueue = new QueueRef(bufferedInputs->QueueStorage, &bufferedInputs->QueueMetadata);
if (inputQueue.Count == inputQueue.Capacity)
inputQueue.Dequeue();
inputQueue.Enqueue(input);
CharacterActions outCharacterAction = default;
CharacterActions.DirInputToCharacterActions(ref input, ref outCharacterAction, in *filter.pCharacterStateMachine, in *filter.pCharacterStats);
for (int i = 0; i < inputQueue.Count; i++)
{
if (filter.pCharacterStateMachine->InputsToCharacterAction(f, in inputQueue[ i ], ref outCharacterAction, filter.Entity))
{
for (int ii = 0; ii < i + 1; ii++)
inputQueue.Dequeue();
//Quantum.Log.Info($"Used buffered inputs from {inputQueue.Count} frames ago.");
break;
}
}
We start by adding the last incoming input to the queue, and we dequeue inputs until we find what we call a meaningful input. If that is the case, we dequeue all inputs up to our meaningful input.
If your attention was sharp, you realized that the InputsToCharacterAction method presented previously was returning a boolean. This is is the way for the current state of the character state machine to tell if there is a meaningful input to be consumed.
public unsafe bool InputsToCharacterAction(Frame f, in Input input, ref CharacterActions characterActions, EntityRef entityRef)
{
bool bIsMeaningfulInput = false;
characterActions.Slide = input.Slide.IsDown;
bIsMeaningfulInput |= characterActions.Slide;
characterActions.Slap = input.Slap.WasPressed;
bIsMeaningfulInput |= characterActions.Slap;
return bIsMeaningfulInput;
}
}
This is an example from our Slide state. You can see that Slap or Slideinput would be consumed instantly, while another input such as Grab would not be meaningful for that state as we cannot grab something while sliding, and would thus be kept in the input queue if no meaningful inputs are found afterward in the queue. If we find a meaningful input later in the queue, the Grab input would be dropped.
Another subtility is that the MoveVector is always the interpretation of the last incoming input (this is done in the DirInputToCharacterActions method call before the inputQueue loop). And filling up the MoveVector for character actions is never going to be considered as a meaningful input. This way, buffering an input for an action has no effect on your movement.
[h2]Conclusion[/h2]
Coming up with this architecture for our inputs was not something that came in a day, but this gives us a lot of flexibility in how we develop our actions and how we control them. We are interested in what you think about our implementation, or in how you did architecture your input flow.
Don’t hesitate to leave us comments on what topic you might want to see on this devlog for the future.
Also, if you are interested in our project, you can follow us on social media, or come play with us on LosPingheros on Steam.
Don’t freeze your brains up ‘til next time, and meet us on the Mexican ice of Los Pingheros !
Romain GROSSE
Hectiq
[h2]Join our playtest[/h2]