The game is a 4X space empire, the combat is autoressolved (players are not manually moving units around during battle - althrough there might or might not be some "orders" given to units before combat (like what targets they should shoot first)).

The scenario is simple, two fleets, made of several various space ships, meet and a battle starts.

The question is, how to make a battle mechanic that promotes combined arms (I mean how to make boring strategies like "make 100 uber battleships only and annihilate everyone" inferior to "make 50 battleships and 200 small escort ships")?

As you mentioned by "200 small escort ships" , a rock-paper-scissors mechanism might help by creating need for other ships, you can also integrate logistics into fleet (needing fuel/food etc supply from nearby planets/system which can also affect range and cost of going to remote systems) which will need logistic related ships (transport ships?) that will also need escort ships to convoy :)

What sorts of combat mechanics are you intending to simulate? I could imagine something like to-hit modifiers making smaller ships harder to target, so capital ships' guns are less effective for combating fighters or other small ships. You could simulate more detailed ship capabilities so that larger ships could have specific abilities crippled, which the player can make more likely by sending specialized ships along with others. Crew requirements? Large ships might need more crew to operate effectively, making them more expensive, or maybe require better trained crews, making them harder to field.

I don't know how the rest of your game is set up, but for an outside-of-battle idea it might work to have higher-tier ships be too expensive to just keep on hand and too slow to build to pump them out in time for a conflict. Mechanics like these could help create an incentive for balanced fleets regardless of in-battle performance.

The game is a 4X space empire, the combat is autoressolved (players are not manually moving units around during battle - althrough there might or might not be some "orders" given to units before combat (like what targets they should shoot first)).

The scenario is simple, two fleets, made of several various space ships, meet and a battle starts.

The question is, how to make a battle mechanic that promotes combined arms (I mean how to make boring strategies like "make 100 uber battleships only and annihilate everyone" inferior to "make 50 battleships and 200 small escort ships")?

Hehe, I'm in a similar situation, but my combat force is of much smaller scale. Nevertheless, different abilities could be very useful, therefor divide your ships into classes or if you have a open ship customization system installed, force ship specialization. This way your ships have different roles (yes, a lot like MMORPG tropes tank, damage dealer etc):

- tanks: take a lot of dmg before going down. Have an ability to taunt enemy to attack him (eg. this could be an active taunt, like a tractor beam or a passive taunt like a low damage area effect over time).

- damage dealer: lot of damage output, but long reload time and quite weak hull integrity

- shield: ships which can project a shield on other ships

- repair: ships which can repair other ships

- ammo/energy: ships which provides energy beams/ammo to quicken the reload times of other ships

- short/long range attacks: most effective at a certain range

- pets: ships which can spawn fighters

This way the player must think twice about what force is useful against what oppenent or in which situation.

- tanks: take a lot of dmg before going down. Have an ability to taunt enemy to attack him (eg. this could be an active taunt, like a tractor beam or a passive taunt like a low damage area effect over time).

I use the same approach. Currently my game has ships that act as "mean forwards" because they have lasting power, but their firepower is relatively low.

- damage dealer: lot of damage output, but long reload time and quite weak hull integrity

and my backliners act as glasscannons (easy to kill, but fire very powerful missiles from a distance).

- shield: ships which can project a shield on other ships

Since many of my ship designs have shields, this wouldn't apply, but I use the concept of projected shields which meshes well with the idea of "command ships".

You put one "command ship" that has a lot of shields to protect smaller vessels, and you give it an aura ability which strengthens fighters or escorts, and this could very well be more cost-efficient than just battleships. However, your command ships is more vulnerable (because he is alone for example).

- repair: ships which can repair other ships

That theoretically makes sense, but in practice, sacrificing a ships slot to repair another may be impractical and hard to balance, unless it is a) very cheap or b) has some firepower of its own, or sufficient defenses to warrant being used.

- ammo/energy: ships which provides energy beams/ammo to quicken the reload times of other ships

I buy into the ammo idea. Most ships should "fly" with bare minimum stocks to save on fuel, and as such, should have only limited cargo. In sustained combats that lasts a few "days", these ships could come in between salvos to restock your ships. Assuming your army is made of "tanks", you could survive battleships for a few encounters until they are without ammunition and at your mercy. In this specific case: tanks+ logistics would win over battleships.

In my game, I tend to keep cargo capabilities fairly low on small ships except on frigates. My ship classification insists that a frigate is generally lightly armed, but has decent armor and stellar cargo capacity. They are, essentially, armed freighters. This meshes well with the above idea.

More importantly, frigates don't need to be part of the fleet to be relevant. They may not even need to be part of your combined arms tactics to still be used. They can just ferry cargo from your planets to the location of the expected engagement (back and forth in loop). They are even hard to intercept by small fleets because they can still pack a punch.

- short/long range attacks: most effective at a certain range

Most definitely. In my game, most battleships triumph at medium to low ranges which leaves long range open for dedicated crafts.

More importantly:

I use maneuverability as an important metric: basically, how fast a ship can move, and how sharp it can turn allow me to determine how likely it is to be hit by the enemy. As a result, even at close ranges, the Battleships don't necessarily make short work of "anything". A dedicated squadron of light and maneuverable gunships, for example, could dodge most bullets with ease, taking several losses, but successfully sinking a battleship (making it an economic win overall).

In these situations, an opponent fielding just battleships is asking for trouble: it is actually a golden opportunity.

Battleship divisions should always be accompanied by smaller crafts that either focus on rapid-fire proximity shots (point-defense) or have sufficient speed and maneuver of their on to chase fast crafts and defeat them.

Another tactic I use is to have relatively the same amount of weapons on each ship (battleships might get 15-20 whereas a gunship can have 4-10). That way, it emphasizes the ability to outnumber the enemy. When all else fails (and you are stuck facing superior enemy ships such as battleships) you can just regroup all of your ships and defeat them through sheer advantage on weapon count. Since battleships will have fewer weapons that deal more damage on average, they will take down ships more easily, but a significant portion of the damage will be lost due to overkill (the last shot on every ship will have a higher % of the damage "lost"). Having more ships with lower damage results will allow to minimize the overkill ratio, and thus, maximize actual damage dealt.

So: Range, maneuver, number of weapons, role classification (tank vs glasscannon vs support ship) should give you a large assortment of things to play with. Everything else is down to execution.

What sorts of combat mechanics are you intending to simulate?

I don't really have preferences/restrictions here. I just want it fun and one that allows some interesting choices to be made (well, don't need to have all these, I'm not picky).

I was thinking of constraining it to combat mechanics only (no production cost or logistics, or at least not primarily). I just want combined arms fleet, while identically expensive, to be superior to homogoneous one.

About RPS (rock-paper-scissors) mechanic:

(assuming the "right" ship deals +100% damage to the "right target" ship)

Let's say one fleet is 30 ROCK ships. The second fleet is 10 ROCK, 10 PAPER, 10 SCISSORS ships.

Would the second fleet end up the winner here? (note: targetting rules (what ship targets which in what order) can be completely arbitrary if you wish)

I use maneuverability as an important metric: basically, how fast a ship can move, and how sharp it can turn allow me to determine how likely it is to be hit by the enemy. As a result, even at close ranges, the Battleships don't necessarily make short work of "anything". A dedicated squadron of light and maneuverable gunships, for example, could dodge most bullets with ease, taking several losses, but successfully sinking a battleship (making it an economic win overall).

In these situations, an opponent fielding just battleships is asking for trouble: it is actually a golden opportunity.

Battleship divisions should always be accompanied by smaller crafts that either focus on rapid-fire proximity shots (point-defense) or have sufficient speed and maneuver of their on to chase fast crafts and defeat them.

How exactly the combat mechanics of yours works? I assume it would require to track the "position" of fighters, also that they need to be "faster" than other ships (so they can attack the battleship before it incinerates everything else)? Also, you would need some "intercept" mechanic for fighters stopping fighters (also, how about you having less fighters fort protecting your battleships only while the opponent uses carriers, could these effectively protect battleships)?

BTW, I agree with using "maneuverability as an important metric", it's less confusing than "speed".

and my backliners act as glasscannons (easy to kill, but fire very powerful missiles from a distance).

How your targetting system works? Does the player can decide which ship will stay in the second line? How your ships attack (target the front line always first?)

Please avoid using MMO tropes, I do hate them, they always feel super game-y.

A simple Rock-paper-scissors approach might work

weapons could be split up into categories:

Good against fighters

Good against capital ships

That pretty much gives you four types of ships

fighters

bombers (fighters with anti capital ship weapons)

escorts (capital ships with anti fighter weapons)

destroyers (capital ships with anti-capital ship weapons)

With possibly another type of ship, depending on implementation

The Carrier (ship which carries/supports fighters)

In fact, you might move to drones instead of fighters, and then you can have it so that if the carrier is destroyed, the drones are destroyed or go inert.

You can get more detailed by having weapon ranges, fire rates, reload speed, some special weapons(tractor beams? stunning ion cannons?) But you can see how the systems might play out.

If the combat is auto resolve, you might look into how Dominions handles it's combat. It's very quick, but relies on some basic setup and rules for units. Unit formations, where they are placed on the battle line, and who their preferred target is. Some examples: having a forward force advancing in the center, with quick forces on the periphery, that are pegged to try to attack weak targets in the back line. Or a force that holds with some strongly armored stuff up front, and weak long ranged stuff in back.

EDIT: I missed Orymus3's post, but I'm on board with his views, and I'm a bit redundant =)

How exactly the combat mechanics of yours works? I assume it would require to track the "position" of fighters, also that they need to be "faster" than other ships (so they can attack the battleship before it incinerates everything else)? Also, you would need some "intercept" mechanic for fighters stopping fighters (also, how about you having less fighters fort protecting your battleships only while the opponent uses carriers, could these effectively protect battleships)?

Originally, I was going for an "accuracy" feel to it. Weapons have a certain accuracy (chance to hit expressed in %) and ships have a chance to dodge (also expressed in % in the form of a positive or negative modifier).

For example, a very heavy warship could have -10% maneuver, which means that a 70% accuracy weapon would actually hit it 80% of the time because it has a hard time to get out of the way. That specific bonus took into account both the expected ship's maneuverability and actual size.

But as it turned out, my combat system allowed me to replay combats in real-time and compute actual "courses" so that ships could move fast, and try to avoid projectiles, and maneuverability became much more organic. Having too many maneuverable ships, for example, won't necessarily reduce the amount of hits taken total, and you're still covering a large area of the battlemap, and if one ship dodges the bullet, another behind might just take the hit.

Kinda makes it less powerful, but I think it adds to the idea that a "swarm of Tie Fighters" would eventually get hit by quite a few missiles if only because of the sheer amount of space they occupy

But if you want to keep it simple, the % hit / dodge works just fine and reduces the need to keep track of additional information.

As it currently stands, smaller ships don't need to be necessarily faster, but it helps to close on larger foes under certain criterias. I do have a few notable exceptions however, such as a highly maneuverable saucer with no actual "helm direction" which can boost in any direction at any given time. It's not particularly fast, and it actually attacks from long range, so it can be a very big annoyance over time when massed in higher numbers.

Since maneuver minimizes the amount of hits taken, it also increases the amount of "shots lost" by the opponent, and these ships require very little armor (as they tend to effectively die in 1 shot from larger vessels).

Fighters don't need to have any specific fighter-interception mechanics. Currently, fighters are forced to move back and forth between mothership/carrier and target (to resupply and repair) which means the area in-between is a zone where fighter vs fighter encounters happen. They will both fire at one another until out of that range. Once closing on the battleship, they will evaluate the threat level. Some ships are fully dedicated at fighting other fighters (AF types) whereas others will focus on larger threats (Bomber types) and others are versatile (Hybrids). Depending on the craft and its intended purpose, they will engage accordingly. AF have higher maneuver and tend to fare better against other fighters, whereas bombers are really good at piercing shields, etc.

I'm not sure about what you meant with the carrier part, but it could be an interesting encounter. At midrange, a battleship would make short work of a carrier, whereas at longer ranges, carriers wouldn't take damage while their fighter swarm could be a real threat to the battleship. Assuming fighters survive the loss of their carrier ship (at least until the end of the fight) I would say that this would often result in draws (fighters surviving the loss of the carrier midway through the encounter, and dealing critical blows the battleship resulting in a no man's land). Economically speaking, the battleship would probably come on top given the sizeable amount of resources put towards getting fighters built, but I can see that adding a few frigates to slow down the battleship could drastically alter this (with only minimal fighter losses, all frigates, but a surviving carrier). Losing 2-3 frigates to a Battleship for the kill is a really good deal!

How your targetting system works? Does the player can decide which ship will stay in the second line? How your ships attack (target the front line always first?)

I use the same heuristics as my previous prototype. Basically, a ship is a collection of components of which weapons and helm control are not related.

Helm tends to have an objective, which is to get withing firing range of its primary target with what it considers its "offensive" arsenal only.

So in the case of a long-range ship (missile frigate let's say), the helm would consider its primary weapons to be its missile launchers only (ignoring anything else) and would move within range of a valid target to unleash missiles.

Other weapons (let's call them secondary in this case) would engage enemies as they enter a valid range, but would not have their say on the helm's control.

This makes it possible to keep ships at a distance, ready to fire with point-defense or anti-fighter weapons (gatling guns) instead of jumping in the fray.

Also I'm currently on the fence about forcing movement, that is, avoid ships being able to come to a full stop. Last time I implemented this, I was satisfied with how ships had to "arc" to keep optimal range with their target all the while staying clear from enemy shots. It felt dynamic, but in larger battles, it added some noise. I'm still hesitant about this AI feature, which is really just commenting one line of code out :P

Currently, I haven't delved very deep into targeting priorities. Originally, I wanted to have a system where components have intrinsic purpose and would tell the helm what their ideal target was and let helm decide from all available targets and ranges, but I felt it ended up breaking damage apart, with ships not focus-firing down specific enemies.

As much as I find it "boring", I tend to prefer how all ships try to focus-fire their closest enemies (often ending with 2-3 active targets on the front). Focus-fire sounds like a valid tactical approach under most circumstances.

I see that it could change based on some occurrences though. For example, high speed ships with the forced non-stop AI would end up moving "through" enemy lines and close in on linebackers organically. From there, they could inflict some damage to the artillery units.

As you can see, this specific part of the design is still up in the air (targeting AI that is) and I've playtested a number of approached, but will need to revise it as I move forward with development. It's very hard to say at the present what will bring the most fun and least frustration to the players. I do anticipate that my component approach will help me retain some flexibility in the long run to hit the right algorithm.

That theoretically makes sense, but in practice, sacrificing a ships slot to repair another may be impractical and hard to balance, unless it is a) very cheap or b) has some firepower of its own, or sufficient defenses to warrant being used.

One of the most annying and powerful enemy groups are the one which heals themself. It helps a lot that more powerful ships outlast longer.

Please avoid using MMO tropes, I do hate them, they always feel super game-y.
A simple Rock-paper-scissors approach might work

If tropes or not, these are two major design pattern used in a lot of games. Please don't measure everthing by the same yardstick. Which might be annoying in a PUG in a MMORPG could be really useful in a single player RTS game.

1. design pattern: Use different roles in a combat situation.

If you build your own army, then this is a really powerful design tool. Differnt roles have different strength and weaknesses.

2. design pattern: Add a fix relationship between roles.

This is more like a balancing guide, then actually a unit design guide. It helps the game designer to create balanced roles/strength/weaknesses at the cost of real variance. For the player it is easier to grasp the relationship of the roles/units.