Setting Up an Experiment

The EvolutionControlActor

The EvolutionControlActor is the invisible actor that will control how EvolvingBehavior works in your experiment/game. To begin using the plugin, first add an EvolvingControlActor to your scene. Currently, only one EvolutionControlActor per scene is supported - make sure you don't add multiple copies.

The EvolutionControlActor contains two components that control the settings and run the experiment. These are the Life Cycle Component and the Evolution Manager Component.

If you need access to parts of these components, such as the Population Manager and the Fitness Tracker, you can retrieve the EvolutionControlActor from the scene and access its components. We provide a helper method that can be called from C++ or Blueprints (defined in EvolutionControlActor.h), called GetEvolutionControlActors, to easily retrieve the evolution control actor from the scene.

Managing the Experiment's Life Cycle

The LifeCycle Component controls how the evolution progresses: its timing, fitness measurements, and so forth. It will be automatically created on the EvolutionControlActor, but you need to set several properties on it.

Life Cycle Properties

An example of setting properties for the Life Cycle component.

You can set various properties to control the evolutionary life cycle.

Trial Length (in seconds): This determines the time limit for each generation of the evolution. When this time expires, the scene will be restarted with a new generation of behavior trees.

Total Trials: The number of generations over which to run the evolution. After this many generations, the experiment will end.

Trial Size: The number of behavior trees to generate in each generation. This should ideally be the same as the number of AI agents who will be spawned in the level to test each individual, generated behavior tree.

Game Speed Modifier: This multiplies the game speed. You can set this higher to accelerate the experiment, but note that setting it too high may cause strange behavior in the physics system.

Fitness Calculator: This section allows you to specify the Fitness calculations that determine which behavior trees are performing well, according to your goals. Please see Determining Fitness for details.

Determining Fitness

Fitness calculation is important for guiding the evolution of your behavior tree. You will want to try to define fitness functions that determine the degree to which each behavior tree's actions meet your goals. It can be difficult to capture all of your goals in numeric calculations, however, so you may need to iterate on your fitness functions and try various approaches. Inspecting and interacting with the resulting agents may also help you determine how well the resulting agents meet your needs, even if the fitness functions are not perfect.

You will need to specify a Fitness Calculator in the Life Cycle component. The Fitness Calculator controls how all the different fitness calculations are combined to determine the final fitness for the agent. Usually, you can use the built-in Linear Weight Fitness Calculator, which adds together all fitness calculations. If that does not meet your needs - for instance, if you want differently-curved fitness calculations - you could write a new Fitness Calculator.

To set up a new Fitness calculation, you need two things:

• A property entry in the FitnessCalculator on the Life Cycle component, set up with details for each fitness key. The key name will be used by the code to update the fitness value for that agent. See Linear Weight Fitness Calculator for more details on setting up the standard calculator.
• C++ or Blueprint code that calls the "BroadcastFitnessUpdated" event on a FitnessUpdater.

In C++, this code should find the FitnessTracker for the population, register the FitnessUpdater (which must implement IFitnessUpdater), then call BroadcastFitnessUpdated:

UFitnessTracker* fitnessTracker = populationManager->GetTracker();
fitnessTracker->RegisterFitnessUpdater( this );
 
// Adds 1 to the current fitness value for the "Alive" fitness key.
// You will need the ID of your NPC's behavior tree, which you will get and save when you Register the use of that behavior tree from the population (see below section on retrieving behavior trees).
BroadcastFitnessUpdated( behaviorTreeID, FString( "Alive" ), 1, false );

Linear Weight Fitness Calculator

The Linear Weight Fitness Calculator is a built-in, simple fitness calculator. It multiplies each fitness key's value by a corresponding "weight," then adds the result together.

For example, if you had two keys and weight values, "MovementDistance: 2.0" and "PointsScored: 3.0", and the result for the agent was 10 movement distance and 4 points scored, the final fitness for that agent would be: 2 * 10 + 3 * 4 = 32.

You should set up the keys and their corresponding weights in the "Id to Weight Map" property field.

The Evolution Manager Component

The next step is to set up the BTEvolution Manager component. It will also be automatically created on the EvolutionControlActor.

The BTEvolutionManager component handles the evolution process of the "population" of Behavior Trees evolved from your initial, hand-designed tree.

You will need to specify the following sets of properties:

• The Template Collection containing the starting behavior tree and extra nodes.
• The Generated Template Library containing the information to generate completely new nodes of various kinds.
• The Reproducer for creating new behavior trees out of old ones.
• The Parent Selector for choosing which behavior trees to use as parents, based on their fitness, for creating the next generation.
• The Initial Population settings for randomly mutating your hand-designed behavior tree, if desired, to create more variety in the starting population.

Template Collection

An example of setting properties for the Template Collection.

The Template Collection contains the manually-designed behavior tree(s) and the extra library(s) of additional behavior tree nodes you would like the evolutionary process to work with.

You can specify one or more starting behavior trees in the "Initial Population Templates" list. These will be the behavior trees used for the first generation of agents, or, if you specify a Initial Population mutation/generation process, they will be used to start the random mutations that generate the starting population.

The "Extra Templates" are additional behavior trees, but the structure of these trees will be ignored - they will be used as sources of additional pre-set nodes that can be swapped into the behavior trees during certain types of mutation. In this way, these "extra templates" act as "libraries" of nodes to try to place in the real behavior trees.

Generated Template Library

The Generated Template Library contains the information needed to randomly create completely new behavior tree nodes with various properties.

You could use this to allow the evolution to add, for instance, a Wait node that has a random time between 0.5 and 3 seconds, and which may mutate over time to have various values in that range, in later generations.

The nodes are separated by type (Task, Decorator, or Service). In each list, you can add multiple nodes of the appropriate Generated node type, with the following properties:

• Name: The node's name, used to show the node's information and help you recognize it.
• Class Type: The actual Behavior Tree node class that should be generated.
• Property Dict: The properties that should be generated inside the node.

The Property Dict has a set of Generated Properties of various types (float, int, bool, etc.) The "name" of each property must match the name of a real property on the Behavior Tree node type that is being generated (no spaces). The properties, depending on type, may also have a "value" and/or "lower bound," "upper bound," or set of possible values to fill in, to specify what reasonable values may be when randomly mutating them.

Reproducer

An example of setting properties for a Mutating Reproducer.

The Reproducer controls how specific Behavior Trees (usually a pair, selected by the Parent Selector) are reproduced and mutated to create a new Behavior Tree for the next generation. Usually, a "Mutating Reproducer" will be flexible enough for most purposes. However, it is also possible to write new types of Reproducer in C++.

Mutating Reproducer

In a Mutating Reproducer, you can add any number of Mutators and give them specific probabilities of acting to change the child that is created.

The Crossover mutator is a common choice. It picks a random point in each of the two parent Behavior Trees, and replaces the primary parent's sub-tree with the branch selected from the secondary parent, essentially grafting the branch to create a child that is a hybrid of the two parents. The Depth Biased Crossover mutator works similarly, but tends to select nodes closer to the leaves of the tree with a particular probability bias you can set using a property. This could be useful if the standard Crossover mutator is making too many large changes to the tree by selecting central trunk nodes.

Additionally, there are many mutators available that Add, Remove, or Replace random nodes (of particular types) in the child. These are called "point" mutations, because they act on a single node instead of an entire branch.

Some mutators always use nodes from the "Extra Templates," the hand-created nodes in the Template Collection described above. These have "Mapped" in their name. Others always use nodes from the Generated Template Library, generating properties based on the rules there. These have "Generated" in their name. Finally, some mutators will randomly select between Mapped or Generated, with a probability you can set.

There will also be mutators for modifying properties of existing nodes, rather than replacing the nodes entirely. These are a planned feature to be released in a future version of the tool, but they are not yet available at the time of writing.

Parent Selector

An example of setting properties for the Tournament Selector parent selection algorithm.

The Parent Selector chooses the specific chromosomes to be used as parents for the next generation, and these parents are sent in batches (usually pairs) to the Reproducer.

Selector

The most common Parent Selector is the Tournament Selector. This parent selector randomly selects some number (we'll call it K) of possible nodes from the population. Then, it takes only the highest-fitness node of that small subset. This gives almost every member of the population a chance to be selected, but over time, the more fit parents are more likely to win each small tournament. The larger the size of the tournament (K), the more likely that a high-fitness chromosome will be in the list. Therefore, high values of K will tend towards always selecting high-fitness parents, and lower values will allow more variety and mix in a larger number of less-fit parents. Often, 4 is suggested as a good tournament size (K) to try first.

Initial Population

An example of setting properties for the Initial Population.

The Initial Population properties provide a way to randomly mutate the input Behavior Trees before starting the experiment, creating more random variety in the initial chromosomes used for the experiment (making variations of the original behavior tree randomly, without regard for fitness).

The __"Num Initial Population Generations"__ property determines how many generations of random children to create before starting to test fitness.

The __"Initial Population Reproducer"__ property allows you to configure a separate set of mutators with different probabilities, so you can add more random mutation, or put more limits on the changes, relative to the rest of the experiment generations.

If you set the number of initial population generations very high, the behavior trees may be very different from your original behavior tree. This might be good for initial prototyping and experimenting, but less helpful if you are doing minor tuning on a behavior tree. If you find that all your behavior trees are too similar at the start of the experiment and you end up with the same results all the time, try adding more probability to the mutators (or more mutators), and/or increase the number of initial generations.

Making NPCs Use Behavior Trees From The Population

You will need to test each Behavior Tree generated as part of the Population, in order to find out how fit each evolved Behavior Tree is.

To do this, when you spawn an NPC (or you could switch the behavior tree running on an existing NPC), you can retrieve the next available Behavior Tree from the current Population generation, and set that Behavior Tree to run on the NPC.

Example C++ code to retrieve the behavior tree and set it on the AI Controller for the NPC:

// Find the EvolutionControlActor and get the Population data.
AEvolutionControlActor* evolutionControlActor = AEvolutionControlActor::GetEvolutionControlActors(GetWorld());
UPopulationManager* populationManager = evolutionControlActor->lifeCycleComponent->GetPopulationManager();
 
// Check that there is a behavior tree available, and register that you are now using the next available one.
FPopulationReg btRegistration;
if( nullptr == populationManager || !populationManager->IsTrialReady() || !population->Register(reg))
{   
 // No behavior tree is available - you could run fallback code to insert a standard behavior tree, or register for the event populationManager->OnTrialPopulationReady() and wait for the callback.
 return;
}
 
// You will need to write code to keep track of which NPC is registered to this behavior tree, so you can send fitness updates with the correct ID.
// For instance, you could have a Pawn subclass that keeps a variable to store this ID.
StoreNPCBehaviorTreeID(this, btRegistration.id);
 
UBehaviorTree* newTree = btRegistration.bt;
 
// This is only necessary if you need to stop an existing tree that is running on your NPC.
if( BehaviorComp->TreeHasBeenStarted() )
{
    BehaviorComp->StopTree( EBTStopMode::Safe );
}
 
// Set up the new tree's Blackboard.
if( newTree->BlackboardAsset )
{
    BlackboardComp->InitializeBlackboard( *newTree->BlackboardAsset );
}
 
// Run the new tree on your NPC!
BehaviorComp->StartTree( *newTree );

Next Steps

Now that you have started an experiment, you can play against the evolving NPCs (or create opponent AIs to play against them), and then save the results of the experiment.

To learn how to save and look at the results of your experiment, continue reading:

Next Page: Experiment Results: Saving, Loading, Evaluating.