The easiest approach is just to test a cone instead of a line. A cone test is just an angle check between two lines - you take one line from the player to the target, a second one from the player to the reticle, and you check the angle between those two lines (i.e. using the dot product) is less than the angular width of the entire reticle.

swiftcoder said:

The easiest approach is just to test a cone instead of a line. A cone test is just an angle check between two lines - you take one line from the player to the target, a second one from the player to the reticle, and you check the angle between those two lines (i.e. using the dot product) is less than the angular width of the entire reticle.

But how can I know to which enemy I should trace the line? It's may be 2 or more enemies on the screen at the same time.

When both combatants are stationary, then using a progressively widening circle should get good results. Perform one ray cast. If it hits, fire a bullet along that ray. If not, try again in a tight circle around the the initial ray. Continue expanding the circle until you hit a target, or until you give up and call it a miss.

When combatants are moving, things become a lot more complicated. Maybe I'm intentionally leading the target because the target is moving, and the aim assist will cause me to miss. Maybe I'm intentionally not leading the target even though it is moving because I expect the target to make a quick turn, and again the aim assist will cause me to miss. I don't think that there's a truly good solution to this problem.

G-Dot said:
But how can I know to which enemy I should trace the line? It's may be 2 or more enemies on the screen at the same time.

Unless we are talking about hundreds of enemies on screen, just check them all. A few ray casts aren't very expensive.

Ideally you'd have a cone-cast operation available to you, but it doesn't look like Unreal provides one. You can probably use a cone-shaped physics collision object to cut down the potential list of enemies (make it invisible and turn off physics response).

So I've tried to make the system, which I've mentioned in my first post. I was a little bit surprised, but it worked out. But with one exception. I thought that I should do that double cone trace every frame, but I realised that this is unnecessary. So I make this line traces each time player hit the fire button.

Also I've got some control over guns behavior. I've promoted trace distance, green and red circle radiuses to the variables and I can manually set up them in order to get more variety in guns. In addition I've promoted amount of line traces for each circle, because 8 lines for red and 4 for green was not very good values. Results can be seen in screenshots:

Not very hard gun to use, cause it have a pretty large green(100% hit) and red(50% hit) circles.

This gun is more difficult to use, because of the smaller circles.

This is how it works with enemies. Pink rectangle represents a hit point without gun's default spread.

It can appear that now aiming is very simple and in order to hit enemy player need just to look at him. But in reality my game is very fast and player got split of second to aim at enemy. Besides only green circle guaranty a 100% hit and it's not very large to land a headshot randomly aiming at enemy.
So summing up my feelings from playing with this system the game became better and I've get that I wanted. So I can say, that for me this system have worked out.