Sorry to possibly throw a wrench in your plans.

I have exactly 0 experience in making an AI, but have been reading about various tactics in implementing things, as they look interesting.

I know FSMs (but in a different context), and browsed BTs a bit (and like you I have problems picturing its execution).

Recently I ran into GOAP (goal oriented action planners). In GOAP, you let go of rigid order in decision making, and instead have individual actions that you combine into achieving a goal.

I cannot tell you how this compares to FSM or BT, or even if it actually works, due to my 0 experience, but it sounds like this may be an alternative that could be of use, for some part of the decision making.

GOAP is really for high-level strategizing. It can be pretty smart, but it's really only good at selecting action states. For a game like Shadowrun it would be simpler, but in a top down squad shooter like OP describes it would likely be hybridized with an FSM or similar.

Sorry to possibly throw a wrench in your plans.

I have exactly 0 experience in making an AI, but have been reading about various tactics in implementing things, as they look interesting.

I know FSMs (but in a different context), and browsed BTs a bit (and like you I have problems picturing its execution).
Recently I ran into GOAP (goal oriented action planners). In GOAP, you let go of rigid order in decision making, and instead have individual actions that you combine into achieving a goal.

I cannot tell you how this compares to FSM or BT, or even if it actually works, due to my 0 experience, but it sounds like this may be an alternative that could be of use, for some part of the decision making.

GOAP is really for high-level strategizing. It can be pretty smart, but it's really only good at selecting action states. For a game like Shadowrun it would be simpler, but in a top down squad shooter like OP describes it would likely be hybridized with an FSM or similar.

You can make squad behaviour with a goap by having enemies exchange world state when they meet. E.g. Two enemies meet in a corridor, one has seen you before at x,y,z and the other hasn't. The first enemy pauses and they exchange information, now the second enemy knows you used to be at x,y,z and they can coordinate a combined plan of attack.

I'm planning on doing this myself to make enemies act like a team. In my case enemies only communicate with members of the same faction/race.

Thoughts?

You can do that with any architecture -- not just GOAP. You are confusing the decision reasoner with the knowledge representation layer.

Regarding an architectual mix between BTs and FSMs: Crytek uses exactly that approach. Basically the AI's decision making system is a classic BT with all the typical nodes (Sequence, Selector, Loop, Action, custom Decorators, etc.). Plus, it offers a StateMachine node that houses multiple State nodes. These are decorators that in turn house a single BT on their own. Transitions between States of the particular level a BT lives in can be triggered arbitrarily from within that BT by sending a signal. So, instead of having to propagate success or failure all the way up in the hierarchy, a signal can be sent, which gets handled by the StateMachinde by activating different State (i. e. BT).

Cheers!