Hi.

I'm at the stage where I now need to send the games map file's to clients that dont have

correct map version or no map at all.

This all get send on top of a boost::asio rapper I created. the max packet size will be 1200 bytes and the send and recive lockfree queue size is packet * 128(not sure how big to make this) any thing over that amount of messages will be lost.

Map's consist of 8 files.

1.BlendMap .dds or .jpg

2.CostMap, holds cell info for terrain A* path finding

3.HeightMap

4.Map Defined Menus, Menu's for selected objects and that get defined in mapedd

5.Map Defined Objects. Objects also get defined for each map

6.Map.map, defined map data like trees and triggers

7.Object Defined Upgrades, upgrades defined when creating the map

8.Mini Map, .dds

all the data files are stored as google protocol buffers that can be serialized to a string

or stream.

But the image files the .dds and .jpg. What do I do when one is created on win 32 and sent to a win 64 bit system and visa versa.

I can have a google protocol buffer with a byte array(this is infact a string anyway).

Q1.Do I only copy the image data and then create the resource on the other side.

Q2.Or do I memcopy the whole file to the byte buffer.

Second Point.

I think I may need a whole system for file sending and file recieving.

because the files will need to be sent one at a time and also split up to fit into a packet and reassembled at the other end before writting it back to file.

A file will be split into filepackets

.

class filepacket
{
in32_t fileID;//need to be able to send multiple files at same time
int32_t chunkID;//0 for first file chunk
in32_t Totalfilesize;//valid file size for chankID = 0 only the first file packet
bool End;//last file packet = true
bytes data;//all the bits
};

.

Now a class to hold all the file packets for a file.

.

class FileStream
{
std::map<chunkID,filepacket> FileStream;

};

.

Then the big Dirty Manager class

.

class FileDownLoadManager
{
std::map<fileID, FileStream> SendFiles;
std::map<fileID, FileStream> RecievedFiles;

//this class could blow out in code size

void threadProcessSendFiles(void)
{
if(SendFiles.WholeFileReady_And_Not_AllreadySending)
Startfilesend();

//do othere stuff like build the send file stream from file ready for sending
}


void threadProcessRecieveFiles()
{
if(networkpacket_ISFILE)
{
AddRecievedFile(packet);
}

//when a recieved file is completed it saves to file and removes itself
RecievedFile_WriteToFileIfReady();
};

.

Q3.Should it be one class does send and recieve. Only of file chunks

Q4.Or one class to manage sending files, And one for recieving files only.

Q1/2: Sending the file as-is should be fine since thats how files are transferred around anyways (might need some endianness check?). Its also compressed so thats good too. Maybe if the files are tiny (some 16x16 icons or smaller? :/) and you needed the decompressed raw bytes at the receiving end, you could transmit them raw to avoid decompression overhead or just for simplicity.

But overall I see no reason not to just send the file, especially if you need the actual file data (to save it as file in receiver) and the compression is great to have.

Q3,4: You need the file data to be ordered (the file is pretty useless if its incomplete...) and you need all of the data, therefore you should just use TCP to send them and not worry about the details. This is exactly the kind of thing it was made for.

As for classes, do whatever you do for other data when it comes to sending and receiving/interpreting it.

Don't re-invent the wheel.

You mention lock-free queues and MTU rates. You question a choice of memcpy of data. You write about breaking the files up yourself manually, transferring and correcting data chunks. DON'T DO THAT.

Use a standard compression and send it using one of the many bog-standard file transfer protocols across your connection. On the other side check the checksum, unpack it, validate the contents, and be done.

For example, you might use the 7z libraries to compress it (see the tutorials, you basically just pass some filenames and it does the work) then use TransmitFile on Windows to send it over your existing socket. Or, if you are using a library that partitions your sockets into data channels to do fancy things with sharing multiple data feeds over a single socket or a reliable-over-unreliable channel, they almost certainly have file transfer support. If they don't, open a second socket and transfer over that.

One thing that isn't clear is if it's a central server sending the maps or if it's peer to peer between two players.

If it's server based use something like http and a Web server and use the built in gzip compression offered by your Web server. for peer to peer use transmitfile or similar.

Hi.

It is a central server clients up load a map from the map editor to server then players can down load and play server maps.

So I should not use my current game message system but allow windows file system to use the socket. I'll look into how boost does it then. there is some streaming I saw it when looking at asio docs.

Hi.
It is a central server clients up load a map from the map editor to server then players can down load and play server maps.

So I should not use my current game message system but allow windows file system to use the socket. I'll look into how boost does it then. there is some streaming I saw it when looking at asio docs.

Use a webserver for it. Really.

You'll gain free Compression, ssl, caching, load balancing, authentication, proxy friendliness and tons of pre written libraries to handle it.

Don't reinvent the wheel when sending files from a central server is a solved problem.

As recently as last week I implemented this exact feature in our game using libcurl and straightforward GET/PUT requests to a web server. There are a plethora of examples on how to use it, most of them only a few dozen lines. zlib is similarly great for handling the compression locally before upload. As mentioned above, no point re-inventing the wheel when all these solutions already exist, and are fairly easy to use.

Oh. My server is just a c++ app on my laptop linked to no ip host.
It only has my network layer doing all the communications, based on a packet message system. I'm not sure if I can share the socket as it's bound to the packet message system.
See all network messages get posted to the send queue.
I may have to open a new socket. Boost asio has a example but I can't see how it's working yet looking into it.

1/2: Either of the two options can work, if you implement them right.
3/4: Either of the two options can work, if you implement them right.

To distribute files, how about spinning up a HTTP server that serves the files. You can use If-None-Match and Etag headers to only actually transfer the map if the client has the wrong version. The benefit of this is that HTTP services / hosts / proxies / CDNs are very easy and cheap to set up.

The second best existing file transfer protocol is Bittorrent. You can look for a BT library or open source client you could drop into your code.

If you want a truly "drop in, no configuration needed" solution for any player to host a new map, then putting the download in your own protocol might be worth it. If you can't use TCP, though, you're going to have to bascially re-implement the TCP protocol semantics on top of UDP, which is doable, but really annoying and unlikely to be worth it. If you don't learn from TCP, then you will either end up with slow downloads, or you will end up with over-congesting links, which in turn leads to network saturation and slow downloads.

Hi.

I found this Here(Urdl) its built on top of boost by some of the boost crew.

I have it working to connect to the http://www.boost.org/LICENSE_1_0.txt as in this Sample.

But how do I connect to my lap top what do I need to run to connect.

I was hoping I could just replace the URL http://www.boost.org/LICENSE_1_0.txt with http://127.0.0.1:86/C:/RTSServer/test.txt the location on laptops HD.

Any Ideas.