So I've been designing what I call a Distributed MQTT Router system that will at least augment my existing MMO netcode(maybe even replace it)..  In that process I've been discovering that it's not an altogether unique idea.  Apparently some folks are using similar systems already.

Here's a pretty good paper on it that I just found.

https://pdfs.semanticscholar.org/89f3/1fa3876d44f983197df593ad69834483d0a1.pdf

So, really my question is, how popular is this idea?  I see some folks using a service called PubNub to make some games(mostly peer-peer stuff I think), I read some of their docs and it also feels very similar to what I'm constructing(though their scope isn't really focused on games).  Additionally, every major player out there has a pub/sub service of some cloud scale..  So, it's usefulness in other arenas is quite obvious.

Have you considered the technology(pub/sub) for use as a complete game server netcode solution(real-time/turn-based/authoritative/peer-peer/etc), and if so, why/why aren't you using it? 

(I'm not trying to discuss the merits/drawbacks of UDP vs TCP, I understand the differences and the overhead concerns for low latency networks.  And I'm aware that most pub/sub systems use TCP, which makes it immediately unpalatable to most game server designers..  This is something that I think might change over time though.  so, if that's your only reason for NOT using pub/sub, I get it.)

I could see developing my solution into an open-source project and possibly even a commercial service if there is some desire for it.  A pub/sub MQTT server that can route/re-route write/rewrite messages and compile and run C# code on the fly?  I think that might be useful to somebody.  What do you think?

Thanks!

If you are not familiar with MQTT, it's a lightweight pub/sub messaging system that is designed for reliability, speed and minimal code footprint, mostly used in the IoT realm currently.

see: http://mqtt.org/

At the very highest level: As long as you meet your goals, it doesn't matter how you do so.

At a slightly lower level: TCP based systems versus UDP based systems have different scalability and performance concerns. Most of the "web services" people don't care about UDP because it's not a good match for what they need, which means there's less easily accessible open source solutions for UDP than TCP messaging cross-bars, even though UDP may be better for some particular game (depends on the game.)

At a slightly lower level than that, implementation concerns like "are clients easily available for the platforms I care about" and "is there good support for expressing and efficiently encoding the kinds of data I care about" may point you towards one solution versus another, assuming that they both support the scalability goals that you need.

And, at the bottom, you end up with "does this system have enough hooks for me to do content-based routing and optimization, and support optimizing the mesh for the needs of my specific game as I grow." Most games never get to have that problem, though, because they die before they get that far, so ti's usually not super high on the checklist of necessary properties.

There are many cases where a messaging crossbar is useful, but they are all slightly different. A "chat room" has a well defined set of people who all need to see each other, which means that a topic-based pubsub system is a great match. A "game world movement" system has less of that, because in a world where player A only sees player B, player B sees A on one side and C on the other side, and player C only sees player B (because of distance-based cut-offs,) there is no single "topic" for getting data you want; instead, what each player sees is custom to that player. You can build this as a set of topics and subscribe players to multiple topics -- A subscribes to B, B subscribes to A and C, and C subscribes to B, in the above case. Then you need to create/tear down subscriptions when people move around, which ends up creating a lot of churn that you have to optimize the system for. You also need to make sure the system doesn't allow some client to just subscribe to all possible topics, and "cheat" by seeing the entire world.

An alternative then is to build the cross-bar in RAM in a game server, where queries on "who can see what" are efficient and can be done very frequently, and just send the stream of "data this client needs to see" without worrying about establishing/tearing down different topic subscriptions. This ends up generally being lighter weight long term, and thus the architecture of many MMO games that focus on world position/movement as a game mechanic. (As opposed to, say, Hearthstone-style card games or whatever.)

If you have more details requirements for all of the parameters that matter (encoding types, lookaside capability, subscription churn rates, and so forth,) then you can make a better call on whether to use a traditional topic-based broker architecture, or something more custom.

3 hours ago, hplus0603 said:

At the very highest level: As long as you meet your goals, it doesn't matter how you do so.

This is my general philosophy. haha

Quote

At a slightly lower level: TCP based systems versus UDP based systems have different scalability and performance concerns. Most of the "web services" people don't care about UDP because it's not a good match for what they need, which means there's less easily accessible open source solutions for UDP than TCP messaging cross-bars, even though UDP may be better for some particular game (depends on the game.)

So very true, I've been scouring the net for something "like" mqtt that uses UDP, not much mention of it anywhere..  I think ZeroMQ has some mostly undocumented UDP abilities, but like I said, not much mentioned about it anywhere.

Quote

At a slightly lower level than that, implementation concerns like "are clients easily available for the platforms I care about" and "is there good support for expressing and efficiently encoding the kinds of data I care about" may point you towards one solution versus another, assuming that they both support the scalability goals that you need.

That's not really in my worry-house right now.   Implementation concerns come with Options, not many of them out there like this that tailor to game developers.

Quote

And, at the bottom, you end up with "does this system have enough hooks for me to do content-based routing and optimization, and support optimizing the mesh for the needs of my specific game as I grow." Most games never get to have that problem, though, because they die before they get that far, so ti's usually not super high on the checklist of necessary properties.

Right, and I'm not really thinking about AAA titles moving their infrastructure over to a system like mine either.   Chances are they are already using the best tech they can get their hands on anyhow.  I'm talking about something that might speed a game to market by a significant amount by abstracting the server build and deployment process, allowing the developer to focus more on the code and less on the system that has to support it.  

Quote

There are many cases where a messaging crossbar is useful, but they are all slightly different. A "chat room" has a well defined set of people who all need to see each other, which means that a topic-based pubsub system is a great match. A "game world movement" system has less of that, because in a world where player A only sees player B, player B sees A on one side and C on the other side, and player C only sees player B (because of distance-based cut-offs,) there is no single "topic" for getting data you want; instead, what each player sees is custom to that player. You can build this as a set of topics and subscribe players to multiple topics -- A subscribes to B, B subscribes to A and C, and C subscribes to B, in the above case. Then you need to create/tear down subscriptions when people move around, which ends up creating a lot of churn that you have to optimize the system for. You also need to make sure the system doesn't allow some client to just subscribe to all possible topics, and "cheat" by seeing the entire world.

A lot of those specific scenarios are addressed in the .pdf I linked above.  A good read so far, but I'm not done yet haha..

Quote

An alternative then is to build the cross-bar in RAM in a game server, where queries on "who can see what" are efficient and can be done very frequently, and just send the stream of "data this client needs to see" without worrying about establishing/tearing down different topic subscriptions. This ends up generally being lighter weight long term, and thus the architecture of many MMO games that focus on world position/movement as a game mechanic. (As opposed to, say, Hearthstone-style card games or whatever.)

Essentially this is how my system works for the game I'm building, using Redis as the RAM based cross-bar, the mqtt routers are the public facing pub/sub netcode, as well as arbitrary server code execution points for distributed operations/localized entity interactions.

Quote

If you have more details requirements for all of the parameters that matter (encoding types, lookaside capability, subscription churn rates, and so forth,) then you can make a better call on whether to use a traditional topic-based broker architecture, or something more custom.

I'm more interested in the feasibility of this thing I'm building finding some use in somebody else's game, not so much if it fits my use case..  Because it does fit my use case, perfectly, or I rewrite it.

The grand idea is to leave that part open to the game developer.  The architecture of the messages/topics isn't going to be predefined by the system(aside from the routing mechanics which relies on specific topic ids).  Leaving the full(basic) capabilities of the mqtt pub/sub system available to the developer.  Obviously there would have to be some concerns for that on a shared hosting system, but for an open-source server, I see no reason to push a format.   It will all be end-user code that runs on the router(server), aside from basic routing functionality which is obviously built in. 

So, I'm mostly picturing this as a fairly simple, spin up the docker/virtual server in the cloud, publish your server logic in C# form to a specific topic with authentication and encryption, and now you have multiplayer support in your game, while still controlling nearly all of the code that runs on the server and not caring one bit about actually Engineering a server/netcode, solution.  

Not a AAA solution out of the box, but I think it might be a AAA time saver for somebody, especially for prototyping.

using Redis as the RAM based cross-bar

Redis is not RAM, Redis is network-attached-RAM, which is an order of magnitude less efficient than co-locating the RAM with the routing/signaling server.

The architecture of the messages/topics isn't going to be predefined by the system(aside from the routing mechanics which relies on specific topic ids).

This is the layered, web-service-like view of the world, and I have never seen that work out for a significant game.

The reason is that games have significantly lower latency bounds than web services, and also have significant gains from small efficiencies that end up letting you push more active players onto the same server ("RAM crossbar.") Hence, games that are dependent on their networking implementation always end up implementing their own system that lets you blend content, encoding, visibility, and transport into a smart cross-layer algorithm.

The two kinds of game networking libraries/frameworks/systems I've seen are

1. Very low-level libraries (Enet, RakNet, etc) that deal with packet addressing, sequencing, NAT punch-through, and perhaps serialization.

2. Full-stack libraries, where the defintion and assumptions of the library have to match your game mechanics. The built-in networking in Unreal Engine, for example, falls in this bucket.

When a game developer builds a networking system, it always ends up being option 2, and it may very well be built on top of some library from option 1, but there really isn't a layering or separation that actually makes sense and is efficient enough to try to find some point between the two to provide as a library. These are the attractor states. If you want to play in scene 2, you have to pick a particular networking "genre" and impose a lot of opinion on the user of your system.

8 minutes ago, hplus0603 said:

Redis is not RAM, Redis is network-attached-RAM, which is an order of magnitude less efficient than co-locating the RAM with the routing/signaling server.

Indeed, I wasn't implying that it IS ram either.  I do understand the difference.

8 minutes ago, hplus0603 said:

This is the layered, web-service-like view of the world, and I have never seen that work out for a significant game.

Good, keep watching.

8 minutes ago, hplus0603 said:

The reason is that games have significantly lower latency bounds than web services, and also have significant gains from small efficiencies that end up letting you push more active players onto the same server ("RAM crossbar.") Hence, games that are dependent on their networking implementation always end up implementing their own system that lets you blend content, encoding, visibility, and transport into a smart cross-layer algorithm.

I think you're still thinking I'm making a game/tailoring this system to fit the standard "fully shared simulation" methodology.  I'm not.  And I don't believe that that methodology would work very well in my setup either, not for large games anyhow.

8 minutes ago, hplus0603 said:

The two kinds of game networking libraries/frameworks/systems I've seen are

1. Very low-level libraries (Enet, RakNet, etc) that deal with packet addressing, sequencing, NAT punch-through, and perhaps serialization.

2. Full-stack libraries, where the defintion and assumptions of the library have to match your game mechanics. The built-in networking in Unreal Engine, for example, falls in this bucket.

When a game developer builds a networking system, it always ends up being option 2, and it may very well be built on top of some library from option 1, but there really isn't a layering or separation that actually makes sense and is efficient enough to try to find some point between the two to provide as a library. These are the attractor states.

There's a lot more than 2 out there, they're just not labeled or originally written to have "Game" in the title.  You seem to have a pretty strict view of what's possible in this arena.  I'm assuming it comes from YEARS of experience and an obvious high level of intelligence.  I'm wondering though how often you look for answers out-side of the box?   Because that's where I live.  And that's where a lot of clever little small game programmers live too.

8 minutes ago, hplus0603 said:

If you want to play in scene 2, you have to pick a particular networking "genre" and impose a lot of opinion on the user of your system.

It's only opinion if it isn't backed up by proven performance.   I intend to build the system into something that has proven functionality before trying to encourage others to adopt it.

I do really appreciate the feedback and thoroughly explained info, as always it's invaluable to me personally.

But the purpose of this thread was centered on the questions I posed in the original post.

i.e.:

Is there some popularity to the use of pub/sub for core network mechanics in games(any kind of game really)?

Are you using pub/sub in your game?  Why/why not?

Would you consider using something like my system, or the ones described in the .pdf I Iinked(very similar to mine), to build a network game?

I guess I should have been more explicit on the question(s) I was asking.  And maybe posted up in a less technical area of the forum...

It was intended to be a general survey, not a technical discussion on the feasibility of my systems components when considered outside of my very specific context.   Oh well, I guess I'll try it again in another thread, on another day when I can better articulate my intentions.

I've personally tried other things ? The closest I got to something working was a TCP based session protocol/crossbar for messaging, which ended up at < 100 ms latencies for hundreds of thousands of users (real, not simulated.) In the end, we ended up using it more for web service data invalidation, than actual game data. (This was an Erlang cluster, connected to through a variety of websocket, plain SSL socket, and HTTP long-polling) It used the topic-names subscription model, but had the benefit of being largely chat-group-like in ordering, no physical proximity there. Access control was by far the most annoying thing, using the most resources.

So, when I say "I've seen only two systems actually work out," it's because, among all the systems I've seen, those are the two that have longevity. I've kept up with a variety of engines, libraries, post-mortems, and conference sessions. Note I'm not saying "UDP or bust" -- obviously, the "starcraft" lockstep model over TCP is fine, although again, there the players are in discrete little clumps, much like chat rooms.

Quote

 I don't believe that that methodology would work very well in my setup either, not for large games anyhow.

If you're building Hearthstone-like games, or Clash of Clans-like games, or Starcraft-like games, then what you're suggesting sounds fine. But maybe now is the right time to actually tell us in a little more detail what particular kinds of games you're targeting? If you're targeting Unreal style games, or Planetside style games, or GTA-V-Online style games, or Destiny-style games/matchmakers, then I don't see how your approach will be good enough to displace the task specific libraries, for example. Hence: If you go with option 2, pick a genre!

Are you using pub/sub in your game?  Why/why not?

In the last two large games I helped build the engine for in some way:

1) Social online 3D world: PubSub on top of a custom Erlang cluster (started way before Redis was even a thing.) It worked out OK, mainly because there was very little physics/game rules -- the use case was very similar to "chat rooms" and "stock quotes" where obviously this model works well.

2) Large multi-game world where player groups "surf" game instances, and each game instance is fully scriptable/customizable by users: PubSub doesn't work for game mechanics; networking built on top of RakNet with heavy customization. (Additional gnarl: some games allowed client authoritative state)

31 minutes ago, hplus0603 said:

I've personally tried other things ? The closest I got to something working was a TCP based session protocol/crossbar for messaging, which ended up at < 100 ms latencies for hundreds of thousands of users (real, not simulated.)

That sounds excellent to me.  When was this?

Also, I really gotta ask, crossbar?  I can't find a definition that gives me any kind of solid grasp on how you keep using that word..

31 minutes ago, hplus0603 said:

In the end, we ended up using it more for web service data invalidation, than actual game data. (This was an Erlang cluster, connected to through a variety of websocket, plain SSL socket, and HTTP long-polling) It used the topic-names subscription model, but had the benefit of being largely chat-group-like in ordering, no physical proximity there. Access control was by far the most annoying thing, using the most resources.

All I see here is that you once made it work with older technology, but you didn't like the implementation, so you moved on.

31 minutes ago, hplus0603 said:

So, when I say "I've seen only two systems actually work out," it's because, among all the systems I've seen, those are the two that have longevity. I've kept up with a variety of engines, libraries, post-mortems, and conference sessions. Note I'm not saying "UDP or bust" -- obviously, the "starcraft" lockstep model over TCP is fine, although again, there the players are in discrete little clumps, much like chat rooms.

I'm not questioning your ability to analyze what works or what you've seen work.  Or your obviously massive experience in the field.  In fact this isn't even the discussion I was trying to have.

31 minutes ago, hplus0603 said:

If you're building Hearthstone-like games, or Clash of Clans-like games, or Starcraft-like games, then what you're suggesting sounds fine. But maybe now is the right time to actually tell us in a little more detail what particular kinds of games you're targeting? If you're targeting Unreal style games, or Planetside style games, or GTA-V-Online style games, or Destiny-style games/matchmakers, then I don't see how your approach will be good enough to displace the task specific libraries, for example. Hence: If you go with option 2, pick a genre!

There's no targeting going on.  It's called putting out feelers..  Kinda like fishing, but without all the water and scales.  It was an attempt at a general query into what some individual developers might have to say about the rough ideas I was proposing.  As far as the game I'm building, only the future will tell what genre it ends up in.

I have a feeling that you and I have very different personalities, and work styles. haha

31 minutes ago, hplus0603 said:

In the last two large games I helped build the engine for in some way:

1) Social online 3D world: PubSub on top of a custom Erlang cluster (started way before Redis was even a thing.) It worked out OK, mainly because there was very little physics/game rules -- the use case was very similar to "chat rooms" and "stock quotes" where obviously this model works well.

2) Large multi-game world where player groups "surf" game instances, and each game instance is fully scriptable/customizable by users: PubSub doesn't work for game mechanics; networking built on top of RakNet with heavy customization. (Additional gnarl: some games allowed client authoritative state)

 

Thanks!

The Erlang solution got started in 2010 or so, and shipped for real in ... 2012? It's still going strong and doing its thing, so I don't think it's "older technology." It's seen at least one major upgrade, to add redundancy in places it didn't have it. (Btw, I think the WhatsApp team is almost entirely on Erlang)

It really is an example of "horses for courses."

On 11/16/2018 at 5:38 AM, Septopus said:

I could see developing my solution into an open-source project and possibly even a commercial service if there is some desire for it.  A pub/sub MQTT server that can route/re-route write/rewrite messages and compile and run C# code on the fly?  I think that might be useful to somebody.  What do you think?

You can definitely run that kind of thing as a service. Look into other middleware + service companies like Photon, PlayFab and GameSparks. It's common for these kinds of "backend-as-a-service" companies to mostly act as a message router for your game clients, but also allow running small snippets of code on their servers in response to events. Plenty of people do pay for that kind of thing.

However, just to keep you grounded for your own game, have a read of "Scalability! But at what COST?". These kinds of architectures (even slim, lightweight, fast ones) often add so much overhead that the phone in your pocket running a single-threaded solution could outperform a very expensive cluster of servers...
Also, you can fit a tremendous number of processing power + RAM into a single server box these days. It's also fairly affordable to give that box 1Gbps of upload bandwidth to the internet. Running an actual MMO off of a single server box isn't ridiculous. IMHO the main reason to have multiple physical boxes is so that globally located clients can connect to a local server -- we can't change the speed of light so the only solution to latency is distributing servers globally.