Hello everyone, I am about to start looking into something that might be thought of an extension of boids AI with predatory logic. I'm not familiar with the "standards" if there are any for something like that so I thought I'd ask for some pointers as I start my research.

What my aim is, is to create an AI that's similar to Shadow of Mordor, Vermintide, Berserk or Left 4 Dead pack simulation that basically gets a random amount of enemies and places them around the character, surrounding him and potentially having "leaders" or at least a "guts" metric as to who leaves the pack and attacks you first whilst not falling on top of each other.

I guess the above is many different AI states as one but I guess my main concern is my starting point in making the encircling AI without falling on top of each other in an optimized way. Thanks!

How coordinated are your enemies? If you really have a set of N enemies with a group state (as opposed to memoryless boids-like individual behaviour) you can easily compute desired positions for each group member in which they collectively surround the target. Then you can update the destinations as the target moves and approach them with generic pathfinding among moving obstacles (most of which are cooperating group members).

For example, suppose L is a distance that is short enough that it's "impossible" to slip between two enemies L units apart: in clear terrain you can use the vertices of a N-sided regular polygon with the target in its center and sides of length L, while with significant fixed obstacles (e.g. trees or walls) you can assign enemies to each gap between obstacles, evenly spaced and enough to have a gap no greater than L between them.

@LorenzoGatti so you mean I just position them as I want, say on the circumference of a circle with the prey in center, keeping an L distance between each one of them but keeping them close and apart to each other. That's fine as a final positioning but as you ask, if I want to coordinate them better how do I do it? Do I tell them keep close to the point in the circumference or suggest ideas as to where he might escape therefore spread out in the circumference?

Problem 2 would be if someone wants to leave the circle but he's at the back, how does he go at the front without going through them. Also you say if I keep the distance L would be ok but how do they decide a preference to left/right/behind him by L.

My suggestion, as you noticed, was simplified because it was about a "final positioning" that can be updated and maintained.
As intermediate states and objectives, before the target is hopelessly surrounded, there are putting the target inside the convex hull of the enemies (i.e. formally surrounded but not in danger) and then closing the large gaps between enemies (with the complication of accounting for what gaps need to be closed urgently because the target is more likely to escape through them) until the gaps are comfortably small and the wolf pack attacks can begin.

Your questions suggest that you want to manage the enemies as excessively independent boids rather than as a members of an explicitly represented formation, leading to gratuitous difficulties like needing an algorithm for position "preferences" when each enemy should simply have an assigned destination (and ignore formation shape parameters like L).

I see, then yes boids is not what I'm looking for, maybe more like strategic placement AI.