How? I'm quite intertested in that. But personally I can't think how it could be done...
First, to avoid potential misunderstanding, I'd like to clarify what I mean by "diversification": I'm not referring to changing the "traditional" form of the genre, but rather to creating new offshoot forms. Similarly, the goal isn't to create new forms that would likely appeal to fans of the "traditional" form, because they already have something that appeals to them; the goal, instead, is to create forms that appeal to those who are put off bye some elements of the genre, but might otherwise enjoy the genre.
By analogy, consider the genre as an open-source project; diversification is then the creation of forks of that project by those who want to take it in a different direction. The original project is still there, but now there are new branches, making the project useful to a more diverse audience.
That said, there are a few extant branches of the roguelike tree: "roguelites", for example, are perhaps a family of the more extreme branches.
The TV Tropes article on roguelikes describes the genre as having four branches at the moment:
- "Hacklikes" (those that descend from NetHack, and perhaps the most direct descendants of Rogue itself)
- "*bands" (those influenced by Angband)
- "Coffeebreak Roguelikes" (the term that they use for roguelites)
- "Experimental Roguelikes" (which, to be honest, seems like a catch-all for those individual games that haven't spawned a full branch)
Thus the genre already has some diversity.
However, this doesn't really address the specific type of diversification that we're talking about here, since we're talking about combat mechanics in "traditional" roguelikes.
First of all, as this thread has pointed out, I'm not terribly experienced in roguelikes, so I'm perhaps not the best person to provide a suggestion. Nevertheless, I do have an idea, albeit an untested, slightly off-the-top-of-my-head one:
Since the choice of platform will likely affect our choices, but is otherwise not terribly important to our purposes here, I'll select the platform on which I've been playing Pixel Dungeon and Delver: an Android 'phone.
Since the goal is a game that is very much a traditional roguelike, but with combat that someone like me might enjoy, we'll start with a very traditional base: the player explores randomly-generated dungeons, collects loot, and encounters randomly-placed enemies. There's some degree of levelling, but that's not the focus of the discussion right now, so we'll leave that nebulous. The gameplay is turn-based.
So far, so traditional, I feel.
The difference is in the combat: instead of to-hit rolls and random damage, we'll use a skill-based minigame. Since enemies are fairly frequent, we'll choose something quick. The specific choice isn't important for the purposes of this thread, and may be something that calls for experimentation. For now I'm going to crib from a recent thread in this forum, and select an oscillating "accuracy"-style bar: simply put, this is a bar in which a "target" location of some size is located, its position randomised; an indicator drifts back and forth at some speed, and the player is tasked with tapping the screen when the indicator is within the target location.
Combat would then alternate between tapping to hit and tapping to defend, with success at the former resulting in damage dealt, and failure at the latter resulting in damage taken. Since we want this to be quick, an easy, weak enemy (a rat, for example) might have just one health-point and have broad, easy-to-hit target regions; on the other hand, a tough enemy (a dragon, for example) might have a fair few health-points, while an agile enemy (a bat, say) might have small, hard-to-hit target regions.
Further, weapons would affect the minigame: swords might be the "base" weapon, offering no changes; a bow might give you one free attack before you have to start defending; a mace might do additional damage, but reduce the size of the target area (being a slower weapon); a great cleaver might do double damage, but have a faster indicator (being a clumsy weapon); and so on. More powerful weapons, or perhaps character levels, might affect the size of the target regions, or the speed of the indicator.
Thus combat comes to rely on player skill, rather than die-rolls, while (hopefully) remaining challenging. It's a form that might not appeal to those who enjoy "traditional" roguelikes, but again, it's not targeted at that audience.
It's frustrating
This is not a feature that I often look for in a game. :P
Challenging, yes; frustrating, seldom.
But if you want to, you can finish an RPG even if you are a moron.
Perhaps adventure games might be a better comparison: barring those that have ridiculous puzzle-solutions, and ignoring the presence of walkthroughs, hints and so on, failure or success comes down primarily to the player's ability to solve the puzzles presented. Thus such a game (well-made) can be both challenging and fair.
I think what happens when success/failure rolls are not encapsulated inside any sort of interesting choice mechanic (say, Push Your Luck), is that it leads to situations in which death can't really be traced back to a choice by the player, and such situations are failure-without-learning, which is no fun. Learning-through-failure is a kind of fun, and it's a big part of the fun of roguelikes. "Oh, well now I know a little bit more about how to survive; I'll make better choices next time", and that's motivation to dive right back into it, to utilize what you've learned. But when you fail just because you rolled poorly a bunch of times, there's not that same motivation to start again.
Indeed, I think that this is my experience.
That said, it seems clear that there are those who do enjoy such gameplay, and, from some of the responses given in this thread, I think that I now better understand why they do (which was, as I recall, the main point of starting this thread). If I'm correct, then it's an experience akin to intelligent gambling: the odds are known and recognised, there are tactics that can be used to manipulate the odds somewhat, and there is a choice in which gambles are worth taking and which should be avoided. On top of that, such players find a thrill in knowing that they could lose at any moment.
(It might be worth noting here that I don't find much, if any, appeal in gambling either.)