For a mmorpg server, which design should I use?

Assume 300 players spawning in same place, they seeing each other. For this we have to send 300 * ( 6(header) + 70(spawn packetpayload) ) bytes of packet (22,2 KB) each user. In total 6,5 MB.

1.

struct sendbuffer

{

char buffer[4096]
int offset
int size

};

(Logic is not written completely or maybe erroneous but you will understand)
void writepacket(void *buf, int size)

{

//check against overflow

memcpy(sendbuffer.buffer + offset, buf, size);

sendbuffer.size += size;

}

When sending,

call writepacket(&packet1, 10)

call writepacket(&packet2, 55)

call WSASend(sendbuffer.buffer, size)

on wsasend io operation completed,

offset += transferredBytes (parameter that GetQueuedCompletionStatus gave us)

call WSASend(sendbuffer.buffer + offset, size - offset)

2.

picking up a fixed-size pool allocator

when building packets, just allocate from pool and write in it. There is no copy but it will cost a lot of memory.

And queue these allocated packets for user, then send it one by one when WSASend calls completed send next one.

3.

Having multiple outstanding calls of WSASend for each packet?

or any better implementation?

Wait, so you have 300 simultaneous players at the same location? Yeah, you are going to stress a simple connection.

None of those three options you presented will solve the fact that you need to send out a ton of data.

300 simultaneous visible avatars is a load on any system. For example, WoW limits raid groups to 40, and EVE Online uses an in-game concept they call "time dilation" where they slow everyone down so network messages can transfer and server processing can keep up.

After your description there is an immediate and enormous design problem: Why do they all need to be at the same place? Can you move them around, put them far enough away that they don't need to be visible?

Note that you've got much more than just dropping 300 individual messages to start. You also have the N2 problem of notifying every nearby player about every other player's nearby actions. Almost immediately every one of those 300 players are going to start running in their own preferred direction. That's around a hundred thousand messages you will need to be throwing around every time everyone communicates an update.

So if you assume five updates per second for nearby objects, you're going to be sending a half million messages in a second. Even if your server in a data center can handle that rate, many of your customers won't have the bandwidth.

In addition to moving them so they are in different spatial zones (and therefore don't need to know about each other), consider staggering events over time so it doesn't stress your server. On most games I've worked on, adding and removing items from the game world is a relatively expensive operation. Moving them around once they exist usually isn't that bad, but adding and removing causes all kinds of registration work that propagates through most other systems. If you can stagger the work to different times that can reduce it from a flood of simultaneous messages into a steady flow of work over time.

Wait, so you have 300 simultaneous players at the same location? Yeah, you are going to stress a simple connection.

None of those three options you presented will solve the fact that you need to send out a ton of data.

300 simultaneous visible avatars is a load on any system. For example, WoW limits raid groups to 40, and EVE Online uses an in-game concept they call "time dilation" where they slow everyone down so network messages can transfer and server processing can keep up.

After your description there is an immediate and enormous design problem: Why do they all need to be at the same place? Can you move them around, put them far enough away that they don't need to be visible?

Note that you've got much more than just dropping 300 individual messages to start. You also have the N2 problem of notifying every nearby player about every other player's nearby actions. Almost immediately every one of those 300 players are going to start running in their own preferred direction. That's around a hundred thousand messages you will need to be throwing around every time everyone communicates an update.

So if you assume five updates per second for nearby objects, you're going to be sending a half million messages in a second. Even if your server in a data center can handle that rate, many of your customers won't have the bandwidth.

In addition to moving them so they are in different spatial zones (and therefore don't need to know about each other), consider staggering events over time so it doesn't stress your server. On most games I've worked on, adding and removing items from the game world is a relatively expensive operation. Moving them around once they exist usually isn't that bad, but adding and removing causes all kinds of registration work that propagates through most other systems. If you can stagger the work to different times that can reduce it from a flood of simultaneous messages into a steady flow of work over time.

<<After your description there is an immediate and enormous design problem: Why do they all need to be at the same place? Can you move them around, put them far enough away that they don't need to be visible?

>>I know a game that handles averagely 250 players in same place (they seeing each other), they do attacks, walks, use potions with low-latency.

<<Even if your server in a data center can handle that rate, many of your customers won't have the bandwidth.

>>Why? Walk packet would be 67 bytes (including ipv4 + tcp header 40 bytes), If everyone walking per user have to download (67 * 300) = 19 kb packet only. If they do another actions too, I'm sure it won't go more than ~45 kb. However, this size will be reduced since tcp nagling is activate.

But which one is best method to use? I can have agent server(s) if It would be solution, simply gameserver will tell to agentserver what to send to players. (Game <> Agent server communication will be fast, since they are in same LAN) Game which I talked above uses that system, there are 3 or 5 agent servers.

A typical server uses multiple queues interally. Let's assume you're using TCP:

First, there is a queue of "entities I need to update the player about." This is something like a list or set or whatever of pointers-to-entities, or, perhaps, ids-of-entities. These are dynamically allocated, but each entity has a fixed small size, so allocation is efficient.

Second, there is a queue of "data to actually write on the socket." This is something like one fixed-size array of bytes per player, often used like a ring buffer. This is pre-allocated once the player connects, so it's contiguous and a known quantity. Typical size of a record: 16 bytes (8 byte "next" pointer, 8 byte "entity" pointer.)

The "update" loop for a player then looks at how much available space there is in the outgoing buffer, and if there's enough, takes some of the queued entities, and generates updates for them into the outgoing ring buffer, and de-queues those entities from the update queue. (or puts them last, or whatever mechanism you use) Typical size: 1024-8192 bytes (too much may cause a lot of latency building up in the queue for slow connections)

Finally, the "send" loop for a player will, each time the socket comes back as writable, bytes between the current head of the ring buffer, and end of the ring buffer, and passed to send(). send() will dequeue as much as it can fit in local kernel buffers, but no more, and tell you how much that was. You remove that much from the ring buffer.

Note that, when you send data, you send many commands/updates in a single packet. You want to pack as much as possible in a single call to send().

If you're using UDP, then you remove the ring buffer. Instead, when it comes time to send a packet, you generate enough updates to fill whatever packet space you have. Let's say you make the maximum UDP datagram size 1200 bytes; this means you attempt to fill 1200 bytes of updates from the head of the queue (minus whatever protocol overhead you're using for sequence numbers or whatnot.) You may also want to put the entity update links into an "outstanding updates" queue if you receive acks from the other end. This "generate data and call send()" happens in one go, and thus you only need a single static buffer that is the size of the maximal packet size, to share between all possible senders.

When there are 300 players in one area, you don't generate data for 300 players at once. Instead, you enqueue 300 players' worth of entity updates into the list of entities, and generate 1200 bytes at a time, and repeat this until there are no entities to update. (Although you will always have entities to update unless the player is alone in the world :-)

Also, when a socket is sendable, it means that a call to send() will not block; it will dequeue your data and put it in some kernel/network buffer. Thus, you don't ever need multiple outstanding send requests. If you use overlapped I/O, then you will typically keep one overlapped struct and one buffer per player, pre-allocated, just like the ring buffer in the TCP case (and, for overlapped I/O with TCP, that's exactly what you will be using.)

A typical server uses multiple queues interally. Let's assume you're using TCP:

First, there is a queue of "entities I need to update the player about." This is something like a list or set or whatever of pointers-to-entities, or, perhaps, ids-of-entities. These are dynamically allocated, but each entity has a fixed small size, so allocation is efficient.

Second, there is a queue of "data to actually write on the socket." This is something like one fixed-size array of bytes per player, often used like a ring buffer. This is pre-allocated once the player connects, so it's contiguous and a known quantity. Typical size of a record: 16 bytes (8 byte "next" pointer, 8 byte "entity" pointer.)

The "update" loop for a player then looks at how much available space there is in the outgoing buffer, and if there's enough, takes some of the queued entities, and generates updates for them into the outgoing ring buffer, and de-queues those entities from the update queue. (or puts them last, or whatever mechanism you use) Typical size: 1024-8192 bytes (too much may cause a lot of latency building up in the queue for slow connections)

Finally, the "send" loop for a player will, each time the socket comes back as writable, bytes between the current head of the ring buffer, and end of the ring buffer, and passed to send(). send() will dequeue as much as it can fit in local kernel buffers, but no more, and tell you how much that was. You remove that much from the ring buffer.

Note that, when you send data, you send many commands/updates in a single packet. You want to pack as much as possible in a single call to send().

If you're using UDP, then you remove the ring buffer. Instead, when it comes time to send a packet, you generate enough updates to fill whatever packet space you have. Let's say you make the maximum UDP datagram size 1200 bytes; this means you attempt to fill 1200 bytes of updates from the head of the queue (minus whatever protocol overhead you're using for sequence numbers or whatnot.) You may also want to put the entity update links into an "outstanding updates" queue if you receive acks from the other end. This "generate data and call send()" happens in one go, and thus you only need a single static buffer that is the size of the maximal packet size, to share between all possible senders.

When there are 300 players in one area, you don't generate data for 300 players at once. Instead, you enqueue 300 players' worth of entity updates into the list of entities, and generate 1200 bytes at a time, and repeat this until there are no entities to update. (Although you will always have entities to update unless the player is alone in the world :-)

Also, when a socket is sendable, it means that a call to send() will not block; it will dequeue your data and put it in some kernel/network buffer. Thus, you don't ever need multiple outstanding send requests. If you use overlapped I/O, then you will typically keep one overlapped struct and one buffer per player, pre-allocated, just like the ring buffer in the TCP case (and, for overlapped I/O with TCP, that's exactly what you will be using.)

I'm using TCP.

I didn't understand your message completely because of my english weakness but can I result that?

Define RingBuffer<n> in user class.

void DoSend()

{

WSABUF buf[2];

DWORD bufNum = 0;

if (m_ringBuffer.GetFirstContiguousBytes() > 0)

{

buf[0].buf = m_ringBuffer.GetFirstContiguousPointer();

buf[0].len = m_ringBuffer.GetFirstContigousBytes();

bufNum++;

}

if (m_ringBuffer.GetSecondContiguousBytes() > 0)

{

buf[1].buf = m_ringBuffer.GetSecondContiguousPointer();

buf[1].len = m_ringBuffer.GetSecondContigousBytes();

bufNum++;

}

if (bufNum > 0)

call WSASend(wb, bufNum, &overlapped);

}

Build packet, then copy it to ring buffer => m_ringBuffer.write(&packet, size)

if WSASend operation is not pending, call DoSend() (If its pending already, It will send it automatically when current is sent)

On WSASend completed, m_ringBuffer.Skip(transferredBytes) then call DoSend()

Having multiple wsabufs is good for this?

Yes, multiple WSABUFs (two of them) helps if you want to send the contents of the end-and-beginning at the same time.
You don't need to "first build the packet, and then copy into the ring buffer" -- just build the packet straight into the ring buffer.

Yes, multiple WSABUFs (two of them) helps if you want to send the contents of the end-and-beginning at the same time.
You don't need to "first build the packet, and then copy into the ring buffer" -- just build the packet straight into the ring buffer.

<<Yes, multiple WSABUFs (two of them) helps if you want to send the contents of the end-and-beginning at the same time.

According to your comment,

Note that, when you send data, you send many commands/updates in a single packet. You want to pack as much as possible in a single call to send().

So, I wanted to send data 4096 byte by 4096. Should I keep that method or any better?

What should I do if WSASend returns with these errors => WSAENOBUFS, WSAEWOULDBLOCK

And one more question, can WSASend operation complete with transferred bytes zero in any situation? (like WSARecv does for graceful close)

So, I wanted to send data 4096 byte by 4096. Should I keep that method or any better?

No, that's fine. Also, you're likely to see less than the full buffer available to send each time you tick the network generation, unless the user is in a very crowded area.

What should I do if WSASend returns with these errors

Wait until the next network tick and try again.

can WSASend operation complete with transferred bytes zero in any situation?

I suppose if you called on it to transfer zero bytes, it could complete successfully with zero bytes.
I would expect that if the connection was closed on the other end, you'd get an error, rather than a completion with zero.
If you don't call it with an argument of zero at any time, it's probably OK to treat this as an error condition.

So, I wanted to send data 4096 byte by 4096. Should I keep that method or any better?

No, that's fine. Also, you're likely to see less than the full buffer available to send each time you tick the network generation, unless the user is in a very crowded area.

What should I do if WSASend returns with these errors

Wait until the next network tick and try again.

can WSASend operation complete with transferred bytes zero in any situation?

I suppose if you called on it to transfer zero bytes, it could complete successfully with zero bytes.
I would expect that if the connection was closed on the other end, you'd get an error, rather than a completion with zero.
If you don't call it with an argument of zero at any time, it's probably OK to treat this as an error condition.

Thank you for your attentive helps!