Is there any place or book where I can find these combat systems on paper? For example rock/paper/scissors combat systems.

I understand the logic in it, but I don't know how to implement it. There must be many rules to make it fair. For example, the damage caused by a DPS to a Tank can't be higher per second than the dmg healed by a healer, otherwise the healer would be pointless and irrelevant. But even then, after a while, the healer will run out of mana and the tank will die, otherwise the fight could be infinite in time, becoming long a boring.

Another example, common in many games, is: if mage is able to attack first, rogue dies. If rogue attacks first, mage dies.

There are hundreds of thousands of games with this approach. To name the most famous one I'd say World of Warcraft. I don't want to make something like World of Warcraft, with 28342389498234 spells and 234283942394 talents.

It would be okay for me if it's only 2 abilities per class, as long as they are truly balanced. Where to start?

Some people consider rock-paper-scissors to be bad design - the lazy way out - preferring other methods like balanced asymmetry.

I understand the logic in it, but I don't know how to implement it. There must be many rules to make it fair. For example, the damage caused by a DPS to a Tank can't be higher per second than the dmg healed by a healer, otherwise the healer would be pointless and irrelevant.

Any decent amount of healing wouldn't be worthless. As long as it's just enough to give the tank the edge, it doesn't have to be equal to the DPS' damage.

For example, the damage caused by a DPS to a Tank can't be higher per second than the dmg healed by a healer, otherwise the healer would be pointless and irrelevant. But even then, after a while, the healer will run out of mana and the tank will die, otherwise the fight could be infinite in time, becoming long a boring.

I think you may be missing a couple of factors here. For example, based on your standard MOBA-style gameplay:

• The healer only needs enough healing to keep the tank alive long enough for the combined damage from the tank and the healer to kill the DPS (and DPS characters are usually quite squishy).
• The tank needs enough crowd-control to prevent a DPS from simply going around and attacking the healer directly (because healers are also squishy).
• Two DPS ought to be able to deal enough damage to force a tank + healer combo to retreat (otherwise tank + healer always wins).

Beyond that, everything should be up to positioning and individual skill, and the rest of balance can be tweaked on an ongoing basis.

I researched this same topic some time ago and the most influential advice that I received was actually the question, "How do you want your game to play?".

I think you may be missing a couple of factors here. For example, based on your standard MOBA-style gameplay:

• The healer only needs enough healing to keep the tank alive long enough for the combined damage from the tank and the healer to kill the DPS (and DPS characters are usually quite squishy).
• The tank needs enough crowd-control to prevent a DPS from simply going around and attacking the healer directly (because healers are also squishy).
• Two DPS ought to be able to deal enough damage to force a tank + healer combo to retreat (otherwise tank + healer always wins).

Also, the tank might have some damage reduction abilities, so the healer would have an easier time keeping the tank alive than another character. Just important to keep in mind that the healing ability probably shouldn't match the DPS of the high damage character.

It sounds like coming up with the specifics of how this should work would be one of the funner parts of the game design. Though I guess having knowledge gained from these types of games as a starting point would be helpful.

It's also worth setting up a simple simulation, so that you can plug arbitrary character matchups and values in, and see the result. Or build a prototype and plug arbitrary values in directly.

In multi-character setups, surprisingly small stat changes can thoroughly change gameplay.