When I first build my browser based game I use JSON as the data-interchange protocol. But then I realize the data in JSON object is too big for transmission. Although JSON serializaiton and deserialization may be the fastest because they are optimized for V8 engine, but NO JSON.

Then I have switched to using a 3rd party protocol or specifically MessagePack protocol as the implementation. I guess using protobuf or other 3rd party protocols give similar result. This binary data-interchange protocol reduce much of the traffic but I realize I am still building an Object before encoding into a binary ArrayBuffer. It seems I can just convert the data directly into TypedArray and send them as binary data without using an object, and that will also eliminate the property names in the objects, saving a few bytes for each message. If the message contains a list of objects it can save more space. This would look like my own custom binary data-interchange protocol which doesn't even need to involve encoding and decoding via MessagePack protocol.

I hope my custom binary data-interchange protocol is framework/language independent so I can implement it in different backend and frontend, not limited to Javascript. But as I have little experience with this before, I hope to get some feedback here. Using a 3rd party library/protocol may give me confidence as it is being used by many projects and the failure rate is low. But I can also get rid of the 3rd party protocol and use my own, which looks much cleaner and I know all the details of the protocol.

At the end of the day my protocol may look something like this:

(Using '|' to separate each type of data but the '|' doesn't put into data transmission as I know how it is organized)

TIMESTAMP|MESSAGE_TYPE|PLAYER_ID|PLAYER_X|PLAYER_Y|PLAYER_ROTATION ...

Some of the 3rd party binary protocols include the field names in the transmitted message, some do not. And sometimes it doesn't matter because they implement useful compression methods that make up for other inefficiencies. Also, some of these systems allow you to add 'extension' types which can be laid out however you like, including the sequential binary example you gave. Besides, there's nothing forcing you to encode everything as an object, which may become a name/value map - you could just send a list of values instead.

Most games have traditionally used their own low-level binary serialisation - i.e. your second example - but most games were written before MessagePack et al existed, and most games aren't written in Javascript or designed to run over Websockets. It's probably best to use 3rd party code where practical.

In this previous GameDev article, I used text messages, delivered with some optimization mechanisms, to transport the positions.

https://www.gamedev.net/resources/_/technical/multiplayer-and-network-programming/optimizing-multiplayer-3d-game-synchronization-over-the-web-r3446

This does not directly answer your question, but I hope it can be useful.

Some of the 3rd party binary protocols include the field names in the transmitted message, some do not. And sometimes it doesn't matter because they implement useful compression methods that make up for other inefficiencies. Also, some of these systems allow you to add 'extension' types which can be laid out however you like, including the sequential binary example you gave. Besides, there's nothing forcing you to encode everything as an object, which may become a name/value map - you could just send a list of values instead.

Most games have traditionally used their own low-level binary serialisation - i.e. your second example - but most games were written before MessagePack et al existed, and most games aren't written in Javascript or designed to run over Websockets. It's probably best to use 3rd party code where practical.

Thank you. With regards to the layout of the data I can send everything as a list of values and encode/decode the data using MessagePack. However as I mentioned before the encoding and decoding methods in those 3rd party serialization protocols may not be as efficient as using JSON as it is native in Node.js/javascript and is optimized in Google's V8 engine. If I organize my data as shown in the second example and don't convert it into smaller size types before encoding with 3rd party tools, I don't think the 3rd party tools will convert it for me automatically. But if I have already converted the data into smaller size data types, encoding it into another format using the 3rd party serialization method seems unnecessary. For example, I may need to send one part of the message using uint8 instead of the default double precision float type which is 64bit, but the 3rd party tool won't do that for me automatically. Unless I want to convert the entire message, which is the value list into the same data type.

"the encoding and decoding methods in those 3rd party serialization protocols may not be as efficient as using JSON as it is native in Node.js/javascript and is optimized in Google's V8 engine" - try it and see. Or just read up on it; MessagePack includes benchmarks on its site, for example. What's your actual problem? You already accepted that JSON was too large so you can't have it both ways.

If the question is, "can I send 6 simple data values with this tool and have it automatically serialised into small and efficient values" then the answer is yes.

"the encoding and decoding methods in those 3rd party serialization protocols may not be as efficient as using JSON as it is native in Node.js/javascript and is optimized in Google's V8 engine" - try it and see. Or just read up on it; MessagePack includes benchmarks on its site, for example. What's your actual problem? You already accepted that JSON was too large so you can't have it both ways.

If the question is, "can I send 6 simple data values with this tool and have it automatically serialised into small and efficient values" then the answer is yes.

My question is a bit complex:

Say I have a message data like this

text-text-uint8-float64-float64-float64-float64-float64,

I will call this message.

Now I want to encode this message. I want to send it as:

binarytext-binarytext-uint8-uint16-uint16-uint8-uint16-uint8.

The tool won't do it automatically for me, or will it? I am not sure. But I am sure it doesn't do that for rotation value if I want my rotation to have binary scaling. If I convert the data types by myself before sending the sequence out I don't really need the tool to encode for me. The tool is there to help me convert things from text/float/integer to binary types but if I want a particular binary data type for a particular field and some sort of binary scaling it won't handle that for me automatically.

In this previous GameDev article, I used text messages, delivered with some optimization mechanisms, to transport the positions.

https://www.gamedev.net/resources/_/technical/multiplayer-and-network-programming/optimizing-multiplayer-3d-game-synchronization-over-the-web-r3446

This does not directly answer your question, but I hope it can be useful.

Thanks that protocol looks very nice. I prefer to use a value list given that I know the order of the fields.

Okay, there are several separate issues here:

• There isn't really a transformation that changes 'text' to 'binary text'. Text is typically sent as-is, either zero-terminated or length-prefixed. All these off-the-shelf binary protocol tools will handle this. If the text is long, sometimes it's worth compressing it first, but I don't think that's relevant here.
• Converting a float64 to a uint16 is indeed something you'd have to do separately. But that's not 'encoding as binary', that's just an arbitrary data transformation. You'd perform the transformation before passing the data to the tool, and you'd perform the inverse translation on the other end. The presence or absence of the tool is of no interest (except in one way - you may not even need to perform the transformation, because these tools often have their own efficient way of representing values. MessagePack and ProtocolBuffers both compress integers pretty well, for example. (Floats, they do nothing with, I think.))
• If you're working in Javascript then you usually don't deal with 'uint16' or any other low-level type; normally you have strings and numbers and that's about it. You can go via TypedArray, but that is not worthwhile for a set of heterogeneous types, as in your example (e.g. string, integer, several floats), especially when you have a tool available that will happily serialise them all to a binary format for you.

Okay, there are several separate issues here:

• There isn't really a transformation that changes 'text' to 'binary text'. Text is typically sent as-is, either zero-terminated or length-prefixed. All these off-the-shelf binary protocol tools will handle this. If the text is long, sometimes it's worth compressing it first, but I don't think that's relevant here.
• Converting a float64 to a uint16 is indeed something you'd have to do separately. But that's not 'encoding as binary', that's just an arbitrary data transformation. You'd perform the transformation before passing the data to the tool, and you'd perform the inverse translation on the other end. The presence or absence of the tool is of no interest (except in one way - you may not even need to perform the transformation, because these tools often have their own efficient way of representing values. MessagePack and ProtocolBuffers both compress integers pretty well, for example. (Floats, they do nothing with, I think.))
• If you're working in Javascript then you usually don't deal with 'uint16' or any other low-level type; normally you have strings and numbers and that's about it. You can go via TypedArray, but that is not worthwhile for a set of heterogeneous types, as in your example (e.g. string, integer, several floats), especially when you have a tool available that will happily serialise them all to a binary format for you.

Thanks for pointing out the issues. I am trying to use TypedArray now and I can see the benefit of using it in serialization. If all the messages add up for all the players I can probably save several KB per second by simply changing one value in the sequence. Otherwise every number will be stored in float64 and sent as the packet over the network. I have also observed games of similar type and I can see the value in converting float64 into smaller size data types. But if you deem MessagePack and ProtocolBuffers both compress integers pretty well I think maybe I should keep using these tools as they can compress the integer values after I transform the data types.

the data in JSON object is too big for transmission

Is that actually a fact? Have you measured it? Have you made an estimate of how much you can save? Are those savings worthwhile?
It may be that paying attention to encodings (making key names short, not sending unnecessary fields, etc) will make it as small as it needs to be, and you can then spend your limited time on something else.

If you need to do binary encoding/decoding, then you should be using an ArrayBuffer with a Int8Array view.