Building a state system for RLBot in C#

25 Feb 2020

"State machines are pretty cool."
~ Me

Bots can have a variety of different states, like saving the net, taking a shot, and picking up boost. Bots also need a way to handle these states and select between them. That's what we'll be building in this post.

For the state handler in this post, we'll have a list of states in order of importance. The state handler will then pick the first state in that list that is viable. It will run that state until it finishes/expires.

For example, let's say we have these states in our list (in order of importance):

Let's imagine that the bot is going to roll into the net. The state handler selects the SaveNet state and keeps running it until the state signals that it has finished. After SaveNet finishes, let's say that the agent doesn't need to save anything and it can't take a shot, but it has low boost. Now, the state handler selects the GetBoost state. After the agent retrieves the boost, the state handler once again selects a state to run.

Now that you know what you're in for, let's get to the code!

BaseState

We're going to need a common interface between states for our state handler. To do this, let's create an abstract class called BaseState.

/// <summary>
/// Base class that all states should inherit from.
/// </summary>
public abstract class BaseState
{
    /// <summary>
    /// Determines whether the state can run at the moment.
    /// </summary>
    public abstract bool IsViable(Bot agent, Packet packet);

    /// <summary>
    /// Gets called every frame by the StepHandler until it returns null.
    /// </summary>
    /// <returns>
    /// Returns the Controller outputs that the state should use.<br />
    /// Returns null if the state has finished.
    /// </returns>
    public abstract Controller? GetOutput(Bot agent, Packet packet);
}

Note: You should make your Bot class public.

To see how to use this BaseState class, let's make an example GetBoost state.

public class GetBoost : BaseState
{
    public override bool IsViable(Bot agent, Packet packet)
    {
        return packet.Players[agent.index].Boost < 80;
    }

    public override Controller? GetOutput(Bot agent, Packet packet)
    {
        // Code here

        if (hasPickedUpBoost)
            return null;
        else
            return new Controller {Throttle = 1, Steer = steer}
    }
}

Once the agent picks up the boost, it can return a null to signal that it has finished.

StateHandler

Once we have a common interface between our states, it's time to choose between them. We'll have a class called StateHandler that does this. Let's see what the structure of this class is going to look like.

/// <summary>
/// Responsible for running and selecting the states for the bot.
/// </summary>
public class StateHandler
{
    private readonly Bot agent;
    private BaseState currentState;
    private (int, int) prevFrameScore = (0, 0);

    public StateHandler(Bot agent)
    {
        this.agent = agent;
    }

    /// <summary>
    /// Chooses the first viable state (determined from IsViable).
    /// </summary>
    private BaseState SelectState(Packet packet) {...}

    /// <summary>
    /// Returns a tuple of (blue team goals, orange team goals).
    /// </summary>
    private static (int, int) GetGoalScore(Packet packet) {...}

    /// <summary>
    /// Returns the output from the current state, and selects a
    /// new state when the current state finishes.
    /// </summary>
    public Controller GetOutput(Packet packet) {...}
}

We'll be using prevFrameScore to know if the goal score changed between this frame (or GetOutput call) and the last frame. If it has changed, we should end our current state because it wouldn't make sense to continue a state into a new "round". Note that this variable is a tuple of (int, int). You'll need to add the System.ValueTuple NuGet package to use it.

The GetGoalScore method is pretty simple.

private static (int, int) GetGoalScore(Packet packet)
{
    return (packet.Teams[0].Score, packet.Teams[1].Score);
}

This is how SelectState is going to work.

private BaseState SelectState(Packet packet)
{
    // States in order of importance
    BaseState[] states =
    {
        new SaveNet(),
        new TakeShot(),
        new GetBoost()
    };

    foreach (BaseState state in states)
        if (state.IsViable(agent, packet))
            return state;

    // If none of those states are viable,
    // we can return a state that will always be viable.
    return new ChaseBall();
}

It's important that the state that's always viable (which is ChaseBall in the code snippet above) finishes quickly. For example, ChaseBall should only run for a second before finishing (sending null). Otherwise, the agent will be unable to choose any other state and will be stuck on that state forever.

Now let's tackle GetOutput for StateHandler.

/// <summary>
/// Returns the output from the current state, and selects a
/// new state when the current state finishes.
/// </summary>
public Controller GetOutput(Packet packet)
{
    // Reset currentState if a goal has been scored.
    // We don't want to continue the state into a new "round".
    (int, int) currentFrameScore = GetGoalScore(packet);
    if (currentFrameScore != prevFrameScore)
    {
        currentState = null;
        prevFrameScore = currentFrameScore;
    }

    if (currentState == null)
        currentState = SelectState(packet);

    Controller? stateOutput = currentState.GetOutput(agent, packet);

    // Return the Controller if the state is still running.
    if (stateOutput.HasValue)
        return stateOutput.Value;

    // Reset currentState if it finished and select a new state.
    currentState = null;
    return GetOutput(packet);
}

The logic should be pretty easy to follow. It resets the current state if a goal is scored. It selects a new state if there's no state at the moment. If the current state has a Controller output, it returns that. Otherwise, the method recurses to select a new state and returns that state's output.

Putting it in the agent class

Phew. We're done with all the hard stuff. Now all that's left to do is run it in the agent's class. I've modified the Bot class from the example bot so that it runs our StateHandler.

public class Bot : RLBotDotNet.Bot
{
    private readonly StateHandler stateHandler;

    public Bot(string name, int team, int index) : base(name, team, index)
    {
        stateHandler = new StateHandler(this);
    }

    public override Controller GetOutput(rlbot.flat.GameTickPacket gameTickPacket)
    {
        Packet packet = new Packet(gameTickPacket);
        Controller output = stateHandler.GetOutput(packet);

        return output;
    }
}

Aaaaand that's it! You've just made your very own state system for a bot! But don't stop here, because there's a whole world of stuff that you could add! You could do all this:

We'll be covering some of these in following posts.

Did you see any errors? Any suggestions? Let me know. ✨