I'm working on a server side game engine for mmo games, and have been thinking through what physics features would be most wanted. The engine is open source but not quite to a first release, this is one of the last pieces to get in place. I'd appreciate any feedback!

Here is the list of features I have so far, mostly all implemented.

- Fast 3d spatial partitioning, for efficient 'players around me' queries.

This sacrifices some accuracy for performance. It guarantees you will get every object in range, but you might get a few just outside of range also. I use 2d grids within grids, and then do some additional filtering on the third axis which is not 100% accurate, but it's fast.

- 3d location tracking for all movable objects.

Nothing to special here, just the ability to do basic queries on anything in the game world to get distances between objects,etc..

- Collision detection & general physics

This is one I'm still unsure of how far to go on. Right now my thought is to have physics for immovable objects. Ensure players can't walk through walls, things like that. The approach I'm playing with is to use A* pathfinding on the server and verify that against the client. If they get into a position that the server finds impossible, they are probably cheating. But right now I just have 2d pathfinding, still playing around with the best solution for 3d.

The physics is just to prevent cheating, not replace any physics on the client.

Are there any other must have physics related features that you would need if making an mmo?

Chris

Why not a full rigid-body simulation? More sophisticated physics capabilities are going to be in demand in the future, not fewer. If you want to stay relevant you'd have to compete with existing physics packages at the very least.

Do you have a particular game you want to support, or is this "a general purpose server"?

Solving the MMO problem "generally" is very hard, because with MMOs, the "constant factor" matters, and different games have different requirements on rules. Some games really need server-side physics simulation. Other games just need a "phone switch" that sends commands between players, with the ability to arbitrate rules-based die rolls every once in a while. Trying to shoe-horn all different kinds of games into a single server structure is unlikely to meet most games' needs all that well.

When it comes to fast spatial queries, we used a multi-resolution grid for There.com, but if I were to do it again, I'd probably just go with something like a loose quad- or oct-tree. Also, it's not always that CPU is scarcer than network bandwidth, so whether "getting a few more entities" is OK or not depends on the game specifics. Once you have an entity in cache, though, doing an additional distance-squared test is close to free, if you need the precision.

My advice would be to design a particular game, and make sure your server can meet the needs of that particular game, mainly related to player/player communications (how about flagging abusers?) transactions (player trade, monster drops, etc) and rule events (did I hit or not? For how much damage?)

Do you have a particular game you want to support, or is this "a general purpose server"?

It's specific to massive multliplayer. The physics area is where I'm most lacking in knowledge and experience.

The main focus of the engine is to have a solid architecture with the right abstractions that make concurrent programming as easy as possible. It should be easy to write game logic that can run in a highly concurrent environment, but that's not the situation in most game engines out there, and there are 0 open source engines specific to games that even try to do this.

Currently the solved problems are ones I have personal experience with and I consider core to any large multiplayer game. It uses the actor model which is based on Akka, with Udp/Udt networking via Netty. There is a built in entity-component system, and a persistence system designed for high write volumes. 90% of the code is ruby. Leveraging java for the heavy lifting but being able to stay in ruby most of the time makes it a highly productive environment without sacrificing performance where it matters.

So anyways, Physics in general,and server side physics specifically, is one of the few areas that I haven't had experience with at any scale. I want to put in place functionality that will be common to 3d multiplayer games in general.

Based on the feedback I'm getting already from this forum and other places, is that I should probably put in a full blown physics engine, but make it a bit modular so you can pick what to use,and tunable so you can run it at varying update frequencies.

Also just to give some context, I spent the last several years working on around a dozen successful social games, some multiplayer some not. Some of these were at very large scale, and the problems that come into play at scale were the first ones I tackled in this engine. I was growing increasingly frustrated with working at what had become a large game company, and decided to take what I had learned and go create something I really wanted to create on my own terms. Things like using jruby instead of pure java or C/C++ came from practical experience that showed language performance actually has very little to do with system performance on the server side. I also wanted an engine that provides fully integrated unit/functional testing, which is another gripe I had about the game industry in general. Being able to test game logic using tools like rspec and cucumber is kind of a game changer compared to what most game dev's currently do. There is just flat out nothing comparable on the java/C side. And being able to stub java methods from ruby is seriously cool:)

Many things you say are true, such as system design is more important than language choice when it comes to scalability. As long as you're I/O bound, your memory throughput and CPU utilization doesn't matter, for example.

I have two additional comments:

1) There exists a large, open source, game server engine that attempts massive scalability. This was Project Darkstar from Sun, since then moved off of Sun/Oracle into the RedDwarf Server project. By trying to be general, it unfortunately lost out on the scalability part.

2) Without a specific game with specific performance requirements and specific rule sets, I fear that your system will not actually solve anyone's particular problems.

Many things you say are true, such as system design is more important than language choice when it comes to scalability. As long as you're I/O bound, your memory throughput and CPU utilization doesn't matter, for example.

I have two additional comments:
1) There exists a large, open source, game server engine that attempts massive scalability. This was Project Darkstar from Sun, since then moved off of Sun/Oracle into the RedDwarf Server project. By trying to be general, it unfortunately lost out on the scalability part.
2) Without a specific game with specific performance requirements and specific rule sets, I fear that your system will not actually solve anyone's particular problems.

Yes I noticed darkstar several years back. There were several other attempts also that I can't remember off the top of my head. And yes I think almost all of them tried to solve too many problems.

I'm actually talking to some local indie game companies in an effort to continue the development within the context of a real game, I think it's important to stay within something real. But I disagree that an engine has to be specific to just one game. There are problems common to all massive multiplayer games. If you can solve those with the right abstractions, then it's reusable across different games.

The problem with genuinely massive games is that they don't generalize.

Scalability and massive-scale performance require making certain concessions. For one game those trade-offs might make perfect sense; for the sequel you may have to go revisit all of your assumptions. (For reference, the architecture between Guild Wars and GW2 changed substantially, just as one example.)

There are real limits to what can be done at massive scale, and how you encounter those limits depends to a painful extent on your game's design. Even going from auto-targeting to FPS-style shooter mechanics can require a complete redesign of the systems involved; they simply have too divergent needs.

Engines that try to be general tend to make games that all feel similar. At massive scale this turns into "your engine is only capable of making one game with different skins." Maybe if you are extremely careful and clever you can get two or three basic designs working in a general way, but you're not going to create, say, the Unreal engine of MMOs.


Of course, there's also my bitter and cynical side which wants to speak up and say that until you've personally shipped an MMO (or equivalent technology) you're not qualified to tackle these problems, but that might not be true, I don't really know :-)

The problem with genuinely massive games is that they don't generalize.

Scalability and massive-scale performance require making certain concessions. For one game those trade-offs might make perfect sense; for the sequel you may have to go revisit all of your assumptions. (For reference, the architecture between Guild Wars and GW2 changed substantially, just as one example.)

There are real limits to what can be done at massive scale, and how you encounter those limits depends to a painful extent on your game's design. Even going from auto-targeting to FPS-style shooter mechanics can require a complete redesign of the systems involved; they simply have too divergent needs.

Engines that try to be general tend to make games that all feel similar. At massive scale this turns into "your engine is only capable of making one game with different skins." Maybe if you are extremely careful and clever you can get two or three basic designs working in a general way, but you're not going to create, say, the Unreal engine of MMOs.


Of course, there's also my bitter and cynical side which wants to speak up and say that until you've personally shipped an MMO (or equivalent technology) you're not qualified to tackle these problems, but that might not be true, I don't really know :-)

I think we are talking apples and oranges here. While I want to put in a good physics system that's usable by basic games out of the gate, that level of features is not what I consider 'common to all games'.

By common to all games I mean things like a good abstraction for concurrency, handling high write volumes to a database, things like that. And those are common to pretty much any game at scale,not just mmo's. They are also things that most games still don't do well.

One of the biggest problems in server side game engines is having good abstractions for concurrency, and that's the #1 issue I'm addressing in my engine. I think it's unfortunate that some of the top game companies just completely punt on good concurrency abstractions.

There is no reason why a core game engine can't be used for several games without major refactors on every game. This really isn't even really in debate IMO when it comes to core architecture. Maybe every 5-6 years you have a major refactor as new paradigm's come along, but they just don't come along every year. It's far more likely that most developers are simply solving already solved problems, and because they don't know it's already solved,doing it poorly.

At a higher level I agree, components of a game engine will change from game to game. But even there you see a lot of developers going through learning curves and solving already solved problems over and over again. To some extent it might be unavoidable, but it certainly can be much better then it currently is.

I also strongly disagree about needing mmo specific experience. In fact I think the game industry needs more new blood with a different perspective. I came into the game industry after having spent 15 years working on finance and web stuff at scale. I found the game industry way behind where we were in terms of scaling stuff, and I found a general inward looking attitude. They genuinely believed their challenges were specific to them, and so they never looked outside of their industry as a matter of habit. Except for one specific area that had nothing to do with scaling, every engineering problem they had was an already solved problem in other industries.

One of the biggest problems in server side game engines is having good abstractions for concurrency, and that's the #1 issue I'm addressing in my engine.

I think that depends on which game engine you're looking at, but I'd be interested in seeing your particular spin on this.
I've seen and/or built engines based on ZeroMQ, UDP broadcast, and Erlang, all of which had good concurrency scalability properties for the particular kind of game they supported.

I think it's unfortunate that some of the top game companies just completely punt on good concurrency abstractions.

In my opinion, that's what you get when your permeating workforce relation attitude is "you should be happy you get to make games 60 hours a week; we don't need to pay market rates for senior skill." This is something that any large game company seems to acquire through osmosis, and I have no idea why that is or how to fix it.

I don't think "massive multiplayer" is specific enough to determine spatial/movement/physics service needs. RPGs, FPSs, and space sims all exist in the MMO space and each have different needs. I've seen attempts to get something in the RTS vein into the MMO space as well.

My feeling is that your framework needs to support the server doing at least the same simulation that the client does. How the client and server simulations will differ and how to reconcile the two will be specific to the game.

It seems like the best you might be able to do is to implement reference systems for one or more specific genres as a starting point or example for plugging simulation into the framework.