A state diagram is a good place to start. Make sure you can make a game out of it before you invest the considerable resources required to represent it to the player.
I'm going to call this an impossible task, because of the proliferation of options. Even assuming that you got the system and interface perfectly modeled and intuitive to the player, asking someone to play such a game would require them to get good at the actual grappling contest, making your game (this subsection of it, even) as opaque and intimidating as a top level flight simulator. Maybe Royce Gracie could play and enjoy the game, but your average audience member is going to feel like a ten-year-old playing career mode in Madden: He'll be bad at the 10% he can make sense of, and just be pushing buttons at random for the other 90%.
Try to boil the contest down to some rudimentary elements. Whip up that state diagram and try to limit the actions in each state to a handful, so your players don't have to think about the exact positioning of all their limbs at all times. Maybe the dominant fighter has options to press his advantage with some direct attacks, to protect his advantage with some control moves and to disengage and transition to a striking fight. The dominated player can defend against attacks, attempt to take control by shifting to a state where he's dominant, or escape and transition to a striking fight. In terms of basic game dynamics, it would wind up looking like a blend of rock/paper/scissors with dice rolls and a tug-of-war, where you're competing for the resource of dominance and then spending that dominance to play out the larger contest of attrition.
So if my fighter has good reach and is long-winded, I'll prefer to keep my feet and stay mobile, while a stockier combatant might prefer to take the fight to the ground and keep it there as much as possible. I'd be willing to absorb a few hits or spend some stamina to disentangle myself and get back to where I can land punches, while he'd be willing to charge through my attacks and grapple with me. If he puts me on the ground, I might not even try to compete with him, and instead direct my efforts toward getting away, accepting that he'll get some good shots in while I worm my way out of the situation. If he sees me trying that, he'll devote some of his time and energy to delaying my escape, allowing him to do more damage before I resume kicking him in the teeth.
Will your fights have referees to keep it fair and clean? My only real hand-to-hand training was at a police academy, and our instructors advised us to always cheat and always win, so our escalation continuum went from verbal commands to lethal force, and our training at physical restraint included joint strikes, eye gouges, metal sticks and shotguns. Obviously MMA fighters are always making an effort to avoid maiming one another. Will your RPG have similar restrictions?
Balancing will, of course, be a nightmare, and you'll have to ensure that the choices a player makes are numerous and varied enough to allow for strategy, without becoming so myriad that it turns into a guessing game. Turn-based is a good idea, and I love your idea of having the turns themselves change length based on the situation, giving another incentive for a player to keep control of the bout. I think a lot can be done with that idea.