Building a state system for RLBot in C#
25 Feb 2020
"State machines are pretty cool."
~ Me
Bots can have a variety of different states, like saving the net, taking a shot, and picking up boost. Bots also need a way to handle these states and select between them. That's what we'll be building in this post.
For the state handler in this post, we'll have a list of states in order of importance. The state handler will then pick the first state in that list that is viable. It will run that state until it finishes/expires.
For example, let's say we have these states in our list (in order of importance):
SaveNet
- Viable only when the ball is predicted to go into the agent's netTakeShot
- Viable only when the agent has a good path to hit the ball into the enemy's netGetBoost
- Viable only when the agent has less than 80 boostChaseBall
- Viable when all of the above are not viable
Let's imagine that the bot is going to roll into the net. The state handler selects the SaveNet
state and keeps running it until the state signals that it has finished. After SaveNet
finishes, let's say that the agent doesn't need to save anything and it can't take a shot, but it has low boost. Now, the state handler selects the GetBoost
state. After the agent retrieves the boost, the state handler once again selects a state to run.
Now that you know what you're in for, let's get to the code!
BaseState
We're going to need a common interface between states for our state handler. To do this, let's create an abstract class
called BaseState
.
/// <summary>
/// Base class that all states should inherit from.
/// </summary>
public abstract class BaseState
{
/// <summary>
/// Determines whether the state can run at the moment.
/// </summary>
public abstract bool IsViable(Bot agent, Packet packet);
/// <summary>
/// Gets called every frame by the StepHandler until it returns null.
/// </summary>
/// <returns>
/// Returns the Controller outputs that the state should use.<br />
/// Returns null if the state has finished.
/// </returns>
public abstract Controller? GetOutput(Bot agent, Packet packet);
}
Note: You should make your Bot
class public
.
To see how to use this BaseState
class, let's make an example GetBoost
state.
public class GetBoost : BaseState
{
public override bool IsViable(Bot agent, Packet packet)
{
return packet.Players[agent.index].Boost < 80;
}
public override Controller? GetOutput(Bot agent, Packet packet)
{
// Code here
if (hasPickedUpBoost)
return null;
else
return new Controller {Throttle = 1, Steer = steer}
}
}
Once the agent picks up the boost, it can return a null
to signal that it has finished.
StateHandler
Once we have a common interface between our states, it's time to choose between them. We'll have a class called StateHandler
that does this. Let's see what the structure of this class is going to look like.
/// <summary>
/// Responsible for running and selecting the states for the bot.
/// </summary>
public class StateHandler
{
private readonly Bot agent;
private BaseState currentState;
private (int, int) prevFrameScore = (0, 0);
public StateHandler(Bot agent)
{
this.agent = agent;
}
/// <summary>
/// Chooses the first viable state (determined from IsViable).
/// </summary>
private BaseState SelectState(Packet packet) {...}
/// <summary>
/// Returns a tuple of (blue team goals, orange team goals).
/// </summary>
private static (int, int) GetGoalScore(Packet packet) {...}
/// <summary>
/// Returns the output from the current state, and selects a
/// new state when the current state finishes.
/// </summary>
public Controller GetOutput(Packet packet) {...}
}
We'll be using prevFrameScore
to know if the goal score changed between this frame (or GetOutput
call) and the last frame. If it has changed, we should end our current state because it wouldn't make sense to continue a state into a new "round". Note that this variable is a tuple of (int, int)
. You'll need to add the System.ValueTuple NuGet package to use it.
The GetGoalScore
method is pretty simple.
private static (int, int) GetGoalScore(Packet packet)
{
return (packet.Teams[0].Score, packet.Teams[1].Score);
}
This is how SelectState
is going to work.
private BaseState SelectState(Packet packet)
{
// States in order of importance
BaseState[] states =
{
new SaveNet(),
new TakeShot(),
new GetBoost()
};
foreach (BaseState state in states)
if (state.IsViable(agent, packet))
return state;
// If none of those states are viable,
// we can return a state that will always be viable.
return new ChaseBall();
}
It's important that the state that's always viable (which is ChaseBall
in the code snippet above) finishes quickly. For example, ChaseBall
should only run for a second before finishing (sending null
). Otherwise, the agent will be unable to choose any other state and will be stuck on that state forever.
Now let's tackle GetOutput
for StateHandler
.
/// <summary>
/// Returns the output from the current state, and selects a
/// new state when the current state finishes.
/// </summary>
public Controller GetOutput(Packet packet)
{
// Reset currentState if a goal has been scored.
// We don't want to continue the state into a new "round".
(int, int) currentFrameScore = GetGoalScore(packet);
if (currentFrameScore != prevFrameScore)
{
currentState = null;
prevFrameScore = currentFrameScore;
}
if (currentState == null)
currentState = SelectState(packet);
Controller? stateOutput = currentState.GetOutput(agent, packet);
// Return the Controller if the state is still running.
if (stateOutput.HasValue)
return stateOutput.Value;
// Reset currentState if it finished and select a new state.
currentState = null;
return GetOutput(packet);
}
The logic should be pretty easy to follow. It resets the current state if a goal is scored. It selects a new state if there's no state at the moment. If the current state has a Controller
output, it returns that. Otherwise, the method recurses to select a new state and returns that state's output.
Putting it in the agent class
Phew. We're done with all the hard stuff. Now all that's left to do is run it in the agent's class. I've modified the Bot
class from the example bot so that it runs our StateHandler
.
public class Bot : RLBotDotNet.Bot
{
private readonly StateHandler stateHandler;
public Bot(string name, int team, int index) : base(name, team, index)
{
stateHandler = new StateHandler(this);
}
public override Controller GetOutput(rlbot.flat.GameTickPacket gameTickPacket)
{
Packet packet = new Packet(gameTickPacket);
Controller output = stateHandler.GetOutput(packet);
return output;
}
}
Aaaaand that's it! You've just made your very own state system for a bot! But don't stop here, because there's a whole world of stuff that you could add! You could do all this:
- Add rendering in
StateHandler
- it makes it easier to understand what state your bot is in - Add "urgent states" that can override normal states - right now the agent can't suddenly go for a save if it's retrieving boost, which is a massive MonkaS
- Use a discriminant union instead of
Controller?
- it makes your code cleaner and gives you more options like adding anAbort
output - Implement higher orders like Self-driving car - it makes code much easier to understand since it acts a bit like imperative code
- Add a
Start
method that only runs once - it's useful if you want state initialisation that requires the packet
We'll be covering some of these in following posts.
Did you see any errors? Any suggestions? Let me know. ✨