I think the question is whether you want a way to present dialogue-driven story during action sequences, or dialogue as a component of an action based game phase.

The positive/negative answer choice that´s been mentioned is an excellent option for the first.
The second aspect pretty much depends on what you do with dialoge, and on your combat system in general. In a rpg-like setting it´d be no problem incorporating dialogue options into a magic system (AD&D has been doing something relatively similar for ages), for a FPS-like environment special hotkeys or dialogue-as-weapon choices might be more appropriate.

One good example (imo) was the way Police Quest SWAT 3 handled this - if I remember correctly there was a two-layer dialogue menue activated with number keys. Not the most elegant solution, but once you got the hang of it it allowed to quickly choose the right sentence. Most important thing though is to have your dialogue have an effect on whatever you´re doing.

"This is the Police, drop your weapon NOW!"

I would like to combine both aspects of this into one system based around the central idea of conversation, rather than splitting them up; though, as mentioned, this could cause some organization problems.

It might be necessary to simply break it down into "to enemy" and "to ally" categories, and having different systems for each.

One thing I forgot to mention is the Persona series of games - they have a sort of 'alternative' battle mode where you can try to talk the monsters into submission, and if you succeed you get special resource items you can't get by defeating them in combat.

What if the options available were more situation/action specific? If I point a gun at somebody, maybe I have four options, based on the D-pad, for example: "Drop your weapon!" "Hands on the wall!" "Freeze or I'll shoot" etc...

But if I don't have a weapon pointed at anyone, I have other options instead. Let's say I'm chasing someone down an alley instead. The same buttons will then correspond with phrases more appropriate. "Stop, you're under arrest!"

Maybe each button could be catagorized somehow so that no matter what the situation is, you know that by pressing up you're going to be more friendly/diplomatic, pressing down it's going to be more hostile/aggressive, etc. What is said exactly would just vary depending on what you're doing, what you have in your hand, and who you're pointing it at.

In a fantasy RPG scenario rather than having your team mates blindly following your commands they could disagree with your tatics and you need to convince them or compromise.

The player may not be leading the attack and must take orders. If the leaders tatics are flawed then you must convince them otherwise.

In fact, conversation on the battle field could gain or loose you tactical advantage. Shouting across a battle field may give your battle plan away. Listening to the enemies conversation could help you identify the leader, killing the leader would then cause the others to flee.

Just to mention something I have seen recently which I found was rather cool.
While fighting one of the final bosses in KotOR2 (well, actually, the last two), the fighting is interrupted by dialogue scenes in which you are given various choices: you can simply tell them off and get back to the fighting, or try to convince them that it's not too late to come back to the Light side or be redeemed, or simply try to manipulate them to lower their morale.
I don't know if the various choices had different results, but the opponent was definitely affected (that is, I dunno if it's my choices that affected him, or if it was completely prescripted and no matter what I chose he would have become weaker anyway).
The only very stupid thing was that all my buffs were lost in the time it took to go through the dialogue, which was a bit frustrating. Thankfull the AI was so stupid that I could happily hide behind one of the conveniently placed pillars in the room and pump myself up again for the next round.

My point is, I found it absolutely cool, very similar to something I would have expected in a japanese RPG. My only gripe was that it was rather obstrusive in the flow of the fight. I would rather prefer something like a Taunt attack amongst my special attacks, especially in the case of KotOR where you can pause in the middle of a fight, it would be very easy to select the type of Taunt/Dialogue you want to "attack" your opponent with...

On PC, there's enough "dead" keys that you should be able to, use shortcut keys for a relatively broad but shallow conversation tree - the tricky part would be finding the on-screen real estate to list the options without either obscuring the action, or being overlooked by the player.

On a console, having a "talk" button that brings up a wheel of options selected by the secondary stick may be the best option - though you probably want the options displayed before they take over the input.

In either case, I don't see briefly sacrificing elements of the input to select a comment as a problem - if you're in a situation where you can't afford to stop controlling your crosshairs for half a second while you press a key or tilt a joystick, your avatar probably can't spare the attention to make conversation either...

If you want people to actually be able to converse freely in-game, rather than selecting from a standard (context sensitive) dialogue tree, then you're probably best off encouraging them to get headsets, and supporting voice transmission - if you're planning to parse natural language input, speech recognition shouldn't scare you off, and if you're not planning to have AI reaction to player chat, then you can relay sound about as easily as text (bandwidth issues aside)

Everyone knows about the stupid color/personality test. There is gold, blue, orange and maybe red, or something like that bottom line there is four colors that match a particular personality.

Now suppose you had some kind of good/evil system, ranging from -5 to +5 (arbituary values). In the heat of battle they can use one of four buttons, which will correspond to the one of the color traits. Now Depending on what color personality the enemy has, he will react in a different fashion. So now we 40 different responses. To further increase that you might want to have a different set of responses for a battle where the enemy is ranked really lower then you are, around the same height as you and prehaps another one for enemies that are stronger then you. It could again be deversified by having a unique character class, the terrain type, etc...

I like the color idea; but I'm concerned that each of the "properties" might be hard to convey to the player. They might not understand what color signifies what property.

What about altering the action so that it lends itself naturally to ebbs and flows?

I've noticed that in movies (especially buddy-cop movies) a large amount of the talking between allies and enemies happens behind cover. They'll discuss plans, relate plot, even get into philosophical debates! In science fiction movies, conversation happens when enemies are getting into position or arcing away from each other, not so much when the lasers are blazing.

For this to work, the game's systems, especially AI and any time-sensitive effects / spells / etc. would have to be structured so that there are detectable dialog states. In movies, all the action occurs instantly, so ticking time-bombs or overloading reactors don't still keep counting down while you're talking.

I supposed you could pair this with context sensitivity, similar to how wall flattening is implemented in many games. Or maybe its a icon that you can respond to which switches the game mode.