Right now, I am involved in the development of a couple of games. One of them is "Just Let Me Go," the horror game I've mentioned before, which is being developed in Unreal Engine 4; and the other one is an unannounced top down dungeon crawler developed in Unity.
One of the biggest differences between Unreal Engine and Unity is the way they handle AI: while UE lets you use behavior trees to make your AI, Unity requires you to do everything using code.
In the Unity Asset Store, you can find assets that add behavior trees to Unity, but I have never tried one and I don't plan to use them any time soon, so I decided to see if I could make a similar functionality using PlayMaker.
The first thing I did was go ahead and create a few different state machines for my enemy. As you can see below, I have a base behavior tree state machine, an actions state machine and a sight state machine. In the past I've also written about using PlayMaker to make NPCs (look for my "How to create NPCs with Navigation Meshes and PlayMaker" series), but, after using UE4's AI tools I realized using behavior trees is a lot easier and more organized than using the method I used before, where I had a big state machine that became too hard to maintain. Another thing I need to mention is that, while this AI I am making right now is pretty simple, you should be able to use this basic principle to make more complex AI.
The first state machine is the behavior tree. This is a pretty basic state machine that runs every 0.1 seconds (for the most part) and checks if the player is seen or not. To make sure the behavior tree is constantly running from the start, I add a "Send Event" action with "RESTART" as the event to be sent. If you don't do this, the behavior tree stops at the last node. To make things easier to manage, you can add the same "Send Event" action to every end node, with different waiting times.
Every node at the end of the tree has a different action to be called (these actions are found in the "actions" FSM). For example, the "attack" node calls for the "ATTACK" event in the "actions" FSM. Right off the bat, this makes things more manageable, since the behavior tree is the one in charge of deciding which action to perform, and I don't need to really worry about "exit conditions" or "overlapping conditions" for the different actions. For example, in my previous method, if the enemy was performing an action but the game conditions change, I had to add exit conditions to all the nodes of that action to make sure the AI would react accordingly. Using this method, the behavior tree simply forces the new action to happen, regardless of what the AI was doing before.
Since this is a very simple game, where combat happens in 4-wall rooms, I decided to make a simple AI, and the only "stimulus" I am using is sight. To keep things simple, I only created two flags, one for distance to player (this checks if the player is close enough to be "seen" by the AI) and the other to check if the player is within a view angle. I downloaded an extra action from the PlayMaker Ecosystem, called "Float Compare With Bools" which performs the comparison and returns true or false, without the need of sending an event (the default float compare forces you to send an event).
I perform two of these "Float Compare With Bools" actions, one to check if the distance to player is less than the required distance, and the other one to check if the angle to player is less than the required angle. Then I use a "Bools All True" to set the "is_seen" variable as true. This "is_seen" variable is the one checked by the behavior tree.
Another thing I can do, is add a raycast, to check if the player is behind a piece of architecture (a wall, a column). I will add that next. Another thing you notice is that the "Bools All True" is sending an event. I did this because I wanted to have a delay between every check. However, it's not needed.
Lastly, the "actions" FSM is shown below. One thing to keep in mind is that every action is launched with a global transition (the dark boxes above every action network) because I need these actions to be launched on demand, regardless of what the AI is doing. For example, if the AI is patrolling, the NPC should be able to enter the CHASE state regardless of what node of the PATROL state is running. This lets me focus just on the actions themselves, rather than having to think about possible exit conditions.
Also, there is nothing special about my "PATROL" or "ATTACK" states, so you can simply make your own as you see fit. In my case, the "PATROL" state simply makes the AI move from side to side, and the "ATTACK" simply makes the AI shoot. The really important stuff is above, in the behavior tree. The only thing I added to the "ATTACK" state was a "Send Event" action that would reset the behavior tree. However, if you do what I mentioned above, and simply add the "Send Event" actions to the behavior tree itself, with different waiting times, you won't have to do this. I would advise you to do this second option, so you don't have to worry about sending events back to the behavior tree.
This is a very basic AI, and there is still room for improvement, but the concept itself makes things a lot easier to manage. Another thing to consider is how you could use FSM templates to make the actions, rather than adding an extra FSM to your enemy, and that's definitely something worth looking into.
On top of that, you can also add other FSMs to read other aspects of the game and environment. For example, you could add another FSM that "listens" to the player, or that is constantly checking on other kinds of environmental situations. In this case, FSM templates can also be useful because you won't need to add different FSM components for every "stimulus" but rather you'll just need to add a single FSM component that runs all the stimulus from FSM templates.