Advertisement

AI Server for MMO system?

Started by July 19, 2002 03:32 PM
70 comments, last by BouncePup 22 years, 6 months ago
It might just be a farfetched idea, but couldn''t a cluster be a pretty neat solutions?
-Shadowwolf
We''re assuming these and the game servers are already in clusters.
Stephen ManchesterSenior Technical LeadVirtual Media Vision, Inc.stephen@virtualmediavision.com(310) 930-7349
Advertisement
Unfortunately there has been a lot of digression from the main topic, but I''ll try to go through and see if I can pull out some gems/lumps of coal and discuss them.

quote:
A machine with a given amount of processing power is almost always much cheaper than two machines each with half the processing power.


Only up to dual processing. Once you go above dual, you''re adding cost at an exponential rate and if you understood the x86 architecture you''d know that dual processing truly is the ''sweet spot'' for performance.

quote:
Having AI on a seperate server (glad to see this idea I tossed out over a year ago caught on) allows you to update the AI continually without shutting down the server to do it.


I''ve been preaching this for a while too. Not for the same reasons, since there are ways to get around rebooting. You can make the reinitialization transparent to the end user, unless you have some AI change that absolutely MUST happen across the entire cluster at the same point in time. Few changes are THAT necessary, unless you really screwed up your AI. Game logic changes are even something that wouldn''t need a reboot if you know how to implement the change gradually and your architecture supports the feature.

quote:
Handling network traffic and state updates is nothing compared to the actual AI routines.


This is completely true. Hardware expenses are relatively non-issues nowadays. Your personnel will cost more than all other expenses combined (literally). Even using something like Dolphin, Myrinet or another of the ultra-high speed networking options out there now won''t increase your expenses to the point of outweighing your personnel expenses. Considering the fact that those options mentioned allow you to sent across the network at speeds approaching that of the RAM speed on the machines that fingh mentioned for EQ (dual 500s, RAM = 100Mhz SDRAM; 800MB/sec potential, real world speed of 500MB/sec, only 250MB/sec/processor Myrinet hits ~300MB/sec), truly fast enough to transfer data to and from the AI machines.

quote:
I think the response time of the AI will be noticably slower, which is probably the opposite of what you want.


I''m not sure that it won''t still be fast enough to where the client won''t really notice the difference. Unless you''re doing ALL of the AI on the AI server... then you''re doing something wrong on the game logic server with the code or you''re making the areas too large.

Of course, there are ways around all of that and ways to ensure that no matter how large an area is, it will never be too large. I won''t get into that though.

quote:
What about coordinating a large army in combat, where some ''soldiers'' are out of visual range of the lead unit/NPC?


I don''t think that you have to go this far initially. You should start a little less complexity initially, then move forward as you have the potential. I prefer the parts of AI that players will actually notice first, then go into more difficult problems. As for the problem that you''re pointing out, you end up in the same situation in a traditional AI configuration. AI is no different either way, just a different method of bookkeeping.

quote:
What about a MP server. The Cost less then buying to identical servers. Just init the World Serv and have it run on CPU_A and have the AI serv Run on CPU_B.


Um... we''re already assuming that anyone making an MMOG is going to be running dual machines. Also, if you''re writing your apps correctly, there are going to be more than one thread for AI and world logic, probably closer to 4-6 threads just for the world logic. Otherwise you can run into some serious problems.

quote:
You could distribute the AI across the clients machines


NEVER trust the client. I can''t put it a better way.

quote:
The reason I chose to offload the AI from the main server was because the main server doesn''t need AI. It''s a huge waste of resources that should be dedicated to game rules and messaging.


You sound like you''re talking about a situation where you''re running one machine for an entire world (and another for all the AI). When you get over 4 square kilometers or so, you need more machines. That model is limited much more than other models like UO, AC or even EQ. It''s also not a good thing to consider AI to be a waste of clock cycles, no matter where they''re occuring. It sounds like you''re trying to put too much world into a single machine.

quote:
The requirements of Everquest are ancient and no longer applicable to MMORPG projects yet people still seem to latch onto them.


Unfortunately there are telltales in your posts that say that you are working under a lot of the same game logic problems that EQ has. You seem stuck in a particular architecture. This is the reason for most of the arguments that follow... well that and an overall lack of maturity of those involved, it should have never went on for 2 pages.

quote:
The other main reason for seperate AI processes is you don''t have to take down the main servers to update AI routines or add NPCs.


There are other ways around this. The first stage is to make your game more dynamic, then you''re halfway there. We simply have built a game server that never has to go down. Individual servers go down, but the clients never see it because no server is locked to a particular area. Even if an active server crashes the clients will never see more than a 30 second delay before other server(s) pick up the load and start handling the affected area(s). Sure, this would mean that there will be a half hour to an hour where the AI logic wouldn''t be consistent across all servers, but realistically the client wouldn''t notice it because AI rarely actually affects the client to an extent where changes would be noticed within a small amount of time. Humans aren''t all that observant, contrary to popular belief.

quote:
I am the worst typer in the world.


That''s typist.

quote:
The backend has undergone many changes in the last year or so.


Unfortunately almost all of the same problems that it had at release are still there because the architecture has never changed. I agree that it''s historical importance shouldn''t be minimalized, but it''s importance to current or future projects shouldn''t be overemphasized.

quote:
In the case of EQ, those people running it are throwing a lot of money away with the model they used.


One-time expenses are irrelevant at this scale. They don''t even end up calculating into the overall expense report as being anything other than a tertiary expense. I''m betting that bandwidth expenses over the past 3 years have overtaken hardware expenses on computers. Personnel costs are probably higher than everything else combined. I''m not really sure why you state that they''ve wasted money though. They started their design a good 5-6 years ago, before Multiplayer gaming was in anything other than it''s infancy. I would say that their design was pretty damn good considering what they had to work with.

I would say that their real waste in money was their design, which involved a zone depending upon a particular server. That and the fact that a server had to be up, even though it was serving no customer load (no characters in the zones it ran).

quote:
Having the AI as a separate process allows you to do both without taking down the main servers. It doesn''t really matter if the process is on another machine or not for patching purposes.


I keep wondering why it really mattered where the process was for restarting it. Restarting a process on one machine and restarting it on another is just a matter of location. Unless your AI is actually part of the game logic, there is no reason for it to require a reboot on one machine and not on another. It''s just a matter of how you code it, where you run it is irrelevant. Happily someone else thought of this too.

I''ll continue my response in the next post, this one is long enough already.
quote:
This is where the "moeny they don''t know what to do with" thing comes in. A startup won''t have the money to buy big quads or eightways, not that I think anyone should, so they have to work with single or dual processor machines.


Unfortunately, when you''re talking x86, quads and octals are a big waste of money. Since you''ll need more than one anyway, simply buy more duals and run a good, fast network. Buying a half dozen Myrinet cards and dual machines to go with them is still way less expensive than a single octal machine.

If you''re talking Power or Sun, then you''re GTG with quads, Alpha is dying, so you''re better off avoiding it for any game for the future, Itanium is too new and too subject to change (and too damn expensive, you can get a quad x86 machine for the cost of a dual Itanium) and nobody knows how it''s going to shake out yet.

quote:
It also costs money in down time when they want to upgrade the AI.


I''m just curious how you can do this without rebooting the server and they can''t. What is so special about running a process in one location as opposed to another? You make several statements that really has me fascinated that you''re a programmer, since they go against basic computer science. A process is a process is a process, it doesn''t matter where you run it, if you have to restart it on one machine, you''ll have to restart it on another.

quote:
Actually you can and it''s a good start when first trying out the method. It''s just not pretty.


Now you''re just arguing to argue.

quote:
KalvinB, there comes a time when you have to stop being a troll.


I feel that I should point out that he wasn''t the first to fling the mud. From Mr. Manchester:

quote:
KalvinB: Good ideas are often thought of as excentric to the under-developed mind. :-p


Putting a smiley face doesn''t lessen the blow when you call someone ''a fucking child'' in prettier terms. This post was nothing but insulting and rude in an extreme. I would have apologized had I said this, since he didn''t deserve it at that point and I would say that this was the cause for the resulting rudeness by KalvinB.

quote:
What is your background in MMOG''s?


I''m not entirely sure that this is a benefit. Baldur''s gate was one of the most successful RPGs and it was built by a team of game programming virgins. Just because you don''t have a particular experience doesn''t mean that you''re any less qualified to talk about it. If anything, with a new genre, you should listen more to the newbies because they might have ideas that the people that have worked on the current games of the genre hadn''t thought of. It''s not like it''s an established genre with over a decade worth of development into it. New ideas are usually the most valuable.

quote:
A fun trick I have in Tombstone is that when someone else logs in under your username and password, they instantly take control of your character.


It''s also a neat trick that EQ had until a while back (don''t know exactly when). I let a friend of mine play on my acct. I would occasionally log in only to find myself in the middle of a fight. Nothing new...

quote:
I retested my NPC client using the 100Mbit LAN. 750Mhz Duron running the NPCS, 1000Mhz Duron running the server. Logging in at a rate of 20 NPCs per second isn''t a problem. It could probably go faster.


I didn''t think that we were discussing running AI off-server to prove how many NPCs that it could run, we were talking about how much more complex it could be. Even with NO graphics, a 750Mhz Duron isn''t going to handle extremely complex NPCs.

quote:
Another reason to use seperate AI/NPC servers is to allow a variety of types of NPC servers. One I intend to do is a CinemaBot where you can actually script NPCs to act out scenes giving the game world a great deal of life. And also an AttackBot that would handle the NPCs that are out to get people.


The problem with your different bots method is that no NPC should be one dimensional. One NPC might be a Duke that, after you kill his sister, he comes after you with a small army. After you are dead, he returns to his normal duties. A merchant might be a merchant during the daytime, but at night he is just another city dweller, with a home and family. You can''t portray that with the NPC styles that you''re talking about.

quote:
We''re assuming these and the game servers are already in clusters.


This is a foregone conclusion with newer games, but we should point out that EQ isn''t a cluster, per se. It''s a group of stand alone servers. Each zone is effectively a completely independent server. Only the communications (chat) server is shard global.
Tombstone is running the game world on a single server currently since that was the most efficient way at the time to get the ball rolling on the project. "Gang Wars" is the same way. Since I''m not expecting thousands of players to be logged in at once, keeping the world small would force more social interaction. With what I''ve learned since starting the project, it would be possible to expand that without much difficulty.

"I didn''t think that we were discussing running AI off-server to prove how many NPCs that it could run"

It could handle more than 750 but the point was that if I have enough processing power to handle 750 basic NPCs (missions, bartering, chat, ect) then I could cut down the number a CPU should handle down to say 50 to allow for the NPCs to have more quality instead of quantity. I can have more powerful machines handling the more intelligent NPCs while using less powerful machines to handle more of the basic NPCs.

"A process is a process is a process, it doesn''t matter where you run it, if you have to restart it on one machine, you''ll have to restart it on another."

Depends on how you integrate it. If you integrate the AI with the game server too tightly like Single Player games, it''s more difficult to just restart the AI portion and it''s impossible to change it without having to shut down the game server as well. If the game server is also distributed then it doesn''t really matter since the other game servers will kick in to take it''s place until it''s up again.

The whole point is to distribute the workload based on what it''s doing and not mix everything together and distribute that. Like an assembly line. One person paints, one welds, one cuts, ect. If you need more painters you add a painter. If you mix those functions together you have to add in someone who can paint, weld and cut to get more work done. Someone who does all three is more expensive to hire than someone who specializes in one area. And one who specializes is generally more effective.

If all you need is better AI it makes sense to have it serperate from everything else so the system can focus on AI and not be limited because it has to do 3 other things as well.

Ben


IcarusIndie.com [ The Rabbit Hole | The Labyrinth | DevZone | Gang Wars | The Wall | Hosting | Dot Com | Rate My Screenshot ]
quote:

quote:
--------------------------------------------------------------------------------
I think the response time of the AI will be noticably slower, which is probably the opposite of what you want.
--------------------------------------------------------------------------------

I''m not sure that it won''t still be fast enough to where the client won''t really notice the difference. Unless you''re doing ALL of the AI on the AI server... then you''re doing something wrong on the game logic server with the code or you''re making the areas too large.




Two issues here.

First of all, certain people involved in the original discussion were adamant that all AI be self-contained in a seperate server process. Like you, I don''t think that''s really the way to do it.

Second, my argument has always been that running a seperate AI server is possible, but would probably not offer a huge return on the investment. The idea is to reap some benefit from doing this, at least enough of a gain to justify the extra time and expense. When you say "the client won''t really notice the difference", I personally would have a hard time justifying it.

Advertisement
quote:
...adamant that all AI be self-contained in a seperate server process.


Yeah, this is about as smart as shooting yourself in the foot Too many limitations with this model.

quote:
Like you, I don''t think that''s really the way to do it.


Most definitely. I think that the simple tasks (pathfinding, attacking, spellcasting, etc...) should be handled by the machine that the area that the NPC is located in is running on. More complex AI should be handled by a separate AI server. If those tasks are too much for your ''zone'' servers (using that term VERY loosely), then you''re obviously making your pathfinding and combat algorithms WAY too complex and need to scale it back to a more reasonable level. Having a separate server handle all of the basic AI is simply a waste of network bandwidth and it most definitely WILL overload your network, whether you use Myrinet (3+Gb/sec) or whatever. You''ll have to run too many redundant data sets with that model. Just not practical for a system that needs to scale indefinitely.

quote:
Depends on how you integrate it. If you integrate the AI with the game server too tightly like Single Player games, it''s more difficult to just restart the AI portion and it''s impossible to change it without having to shut down the game server as well.


Now you''re talking about different programs. This discussion is about running the AI in a different location. I''m sure that most everybody can agree that having it running as a separate process (whether on or off the zone server) is a foregone conclusion as being better than having it run as part of the same process as the game server applet itself, whether as a separate thread or not. I don''t think that anyone here was arguing that you should run the AI in the same process, if they were, then yes, they are wrong. But I don''t think that they were, so please get off that hangup and assume that they at least aren''t idiots. I prefer to give everyone the benefit of that doubt.

When you assume that they are running the AI in a separate process, it''s irrelevant where it runs, except that for basic AI you don''t have to make redundant copies of basic information (character location, race, faction information, etc...) or alternatively have to send that information across the network repeatedly as the NPC AI needs that information when you run the process locally on the ''zone'' server.

I think that there are most definitely parts of AI that are easier to run on the local machine, but there are also parts that either have higher processing needs or requires global data that is easier kept on a separate db server specifically for the AI.

(back to fingh)

quote:
When you say "the client won''t really notice the difference", I personally would have a hard time justifying it.


It really matters on what you''re running local to the server and what you''re running on an AI machine. Also, the time for a local LAN transmission to/from an AI dedicated machine is almost non-existant (< 20ms). Most definitely not enough for a player to notice the difference as opposed to running the complex AI locally on the server. I think that the benefits to having scalable, complex AI that can run independent of the zone servers is most definitely beneficial. There is a more global view that can be obtained easily without having to make requests of several other zone servers.

With one AI running with a local view (pathfinding, combat, etc..) and another with a global view (quests, strategy, NPC chat, etc...), you get two different types of information and two different types of reactions. Each is appropriate to their actions. While a King may not pathfind to an enemy lord (local level information), he might discuss that enemy lord or create quests that involve that enemy lord (global view information).

Two different types of AI, two different views necessary, two different types of action.
(missed a comment that I meant to respond to)

quote:
Since I''m not expecting thousands of players to be logged in at once, keeping the world small would force more social interaction. With what I''ve learned since starting the project, it would be possible to expand that without much difficulty.


But the point is that this model (Tombstone) won''t scale. We''re talking about 2 completely different situations. One is a model where it MUST scale because of the market, yours is a small server that won''t matter whether it scales or not because it''s small enough to not matter. Creating a server model that will scale is completely different from creating one that can''t. Network bandwidth is definitely a finite resource, if you''re eating up 10Mb/sec with your current model under test simulations and will need to be bumping up to 150-250 servers for content and player load, the model you''re working under is going to become laggy and deficient. I''m not saying it''s a bad model, it works great for what you''re doing and very well may be the ideal model for your games, but it won''t scale to multiple servers without a LOT of rework. Unless you want to show your customers "LOADING, PLEASE WAIT..."
Actually the only thing stopping Tombstone right now is DirectPlay which isn''t too difficult to replace with Winsock and the way accounts are stored which also isn''t a problem since I now know mySQL which would allow me a central database server.

To expand Tombstone all I have to do is plug in another server and load it up. For more NPC, I plug in another server and load it up or just run another instance of the NPC client on an existing server.

With Winsock, telling a client to disconnect and reconnect on a different port (to get a new server for the region they''re going to) is completely transparent. I could even just have a "lobby" who''s sole job is to forward packets to the right server so the client never has to disconnect.

The game world is already divided into 16 regions. The time it takes to switch servers would be no more than the time it takes to get information for a new region which isn''t long since regions are small but large enough you won''t be running in and out of them very often.

Tombstone is completely modular making it incredibly easy to expand to account for the number of people wanting to play.

Ben


IcarusIndie.com [ The Rabbit Hole | The Labyrinth | DevZone | Gang Wars | The Wall | Hosting | Dot Com | Rate My Screenshot ]
quote:
Solinear said:
I''m sure that most everybody can agree that having it running as a separate process (whether on or off the zone server) is a foregone conclusion as being better than having it run as part of the same process as the game server applet itself, whether as a separate thread or not. I don''t think that anyone here was arguing that you should run the AI in the same process, if they were, then yes, they are wrong.




Two more issues... (or "The root cause of arguments here")

One, there is absolutely nothing wrong with running AI in the main process (threaded or otherwise). Now, I would agree that running a seperate process, using shared memory segments MIGHT be better for certain overall designs. Even this "better" model assumes that locality is preserved (same hardware, clustered or otherwise). It''s being done today (in-process AI) effectively. Could AI get better? Of course. Do you have to make the architecture extremely complex to do so? Not necessarily.

Second, the main reason I see for this thread going awry in the first place is absolute statements like the last sentence of the previous quote:
"I don''t think that anyone here was arguing that you should run the AI in the same process, if they were, then yes, they are wrong." I thought smanches and I had already covered the idea that there isn''t necessarily a right and wrong answer here. The AI in pretty much every game I''ve ever played or worked on is in the main process. Suddenly that is wrong? No. CAN it be done in a seperate process on server hardware far removed physically? Sure. Is it the best way to do it? I sure wouldn''t bet my job on it...

Without giving away proprietary information, let me just say that I''ve talked (however briefly) with Solinear about his companies server architecture, including the use of seperate AI structures. At that time, I didn''t raise objections, and I still wouldn''t. For him and his team, maybe that''s the right thing to do. Is it right for every MMOG? Probably not.

This topic is closed to new replies.

Advertisement