Watch out for directory sizes.

We have systems in my day job that regularly iterate directories containing 160,000 or more files. The I/O load on the system is immense.

The fast and efficient solution to that is to create a filename prefix tree.
For each file, create one or two levels of directories:

foobarfile.dat -> fo/foob/foobarfile.dat

Iteration then means recursively iterating the directory in question. How many characters per level, and how many levels, depends on the spcifics of your data.
This works extra well when the file names are a hash of the contents, because each level of the prefix will have an even spread of data.

The "real" solution is to realize that a file system isn't the right solution for that use case, and start using a database :-)

The fast and efficient solution to that is to create a filename prefix tree.For each file, create one or two levels of directories:

foobarfile.dat -> fo/foob/foobarfile.dat

Iteration then means recursively iterating the directory in question. How many characters per level, and how many levels, depends on the spcifics of your data.This works extra well when the file names are a hash of the contents, because each level of the prefix will have an even spread of data.The "real" solution is to realize that a file system isn't the right solution for that use case, and start using a database :-)

I absolutely agree. Don't do what they did where I work :)

I'm actually trying to migrate it to storing by date right now, e.g. 2015/06/27/foobarfile.dat.

It's not that simple because it's a huge Frankenstein system with tons of hard coded references to said directories ;)

It's not that simple because it's a huge Frankenstein system with tons of hard coded references to said directories ;)

But you do have a full integration test setup, so you can just change it, run the tests in sandbox, and see what breaks, right?
I mean, nobody would build a system they couldn't test, right?
:-)

A data base is just a file system anyway is it not.

A database is a collection of information that is organized so that it can easily be accessed, managed, and updated. In one view, databases can be classified according to types of content: bibliographic, full-text, numeric, and images.

For some reason I cant get out of this quote box lol(oh and now I am out of it WTH).

I sort of have a mini data base for the server the server holds a foldername that all the maps get uploaded to. then all files names from that folder get placed in a container, Ready for clients to retrieve a game map to play.

I mean, nobody would build a system they couldn't test, right?

Thats not always true. Works check out program not do a good job. again yesterday. the scanner activated a mouse click that printed a label funny as.

oh wait more on clients accounts if you type there whole name for the account it puts a ladder foot item in the sales area what.

and you want me to use some other persons software.. at lest with allready made software people already know there holes. less work for the hacker here.

Thats not always true.

I thought it was pretty clear that I was being ironic :-)

Software development is, generally, done to make a business money. If the business isn't really in the software business, but just automates tasks to save on manual labor, then as soon as it "works" it's "done."(Automated) testing is for people who want to be able to change things, which generally means software-development businesses.

It's not that simple because it's a huge Frankenstein system with tons of hard coded references to said directories ;)

But you do have a full integration test setup, so you can just change it, run the tests in sandbox, and see what breaks, right?I mean, nobody would build a system they couldn't test, right?:-)

Of course, there is a test system, and three development systems (one for each of us developers) the complexity comes from making sure the tests cover all edge cases... But that's a problem with developing anything :)

Well I'm up to the stage where the client can hit the refresh button and the server will send a batch of mapnames, But on the last key strike I had a thought that the client can spam the refresh which will in the end if multiple clients do it. It will slow the server in the end.

I thought of limiting the refresh button with a timer but that seems bad when you are looking for maps for real.

So now I need a way for the server to send small amounts of data to all joined players until they have a complete list of map names, But that could be lots of map names to lots of people.

How do you manage this data getting sent.not storage on hard drive but sending to clients chuncks of filenames.

would you just have a process loop that sends a batch to a client then leaves the loop do other server stuff next time around process next client and keep doing it one send per client. when all clients have that batch get the next batch and so on.

the clients sides UI can handle streamed in mapnames the client may not see any thing until a send come in but thats ok they have a fake refresh button to press hehehe.

Any other ideas.

I had a thought that the client can spam the refresh

This is known as a "denial of service attack" and is something you have to work around server-side.
Putting a timer on the refresh button on the client isn't particularly great, because a determined attacker can just make the same kind of request without using your client (and thus not being limited by the button.)
A better way to do it is to remember the last time a refresh request came in, and if the refresh request is too early, either ignore it, or pend it until a reasonable time has expired. If multiple refresh requests come in at the same time, pend them all, and resolve them with the same data once the final timer has expired.

If the goal is to show active server instances (which is perhaps different from "all maps" -- because many servers can use the same map, and some maps may not have active servers) then perhaps a better idea is to put state in the protocol. When the client first connects, it gets sent a list of all servers automatically. Then, as new servers connect, or servers disconnect, the clients are sent an update with the deltas, every so often.
You'll also want to use timeouts and keep-alive messages to make sure that idle connections don't take up resources for too long, but that's best handled at the connection layer, so it applies to anything your game does.

Hey all.

It was proving to be difficult to limmit the user sending manny request in the current set up. So I sat on the idea for a week or 2, and came up with a 2 prong solution.

The first was to send the data as the user requests by refresh button, user can click this to there hearts content.

Second I created a network message monitoring system, That sits ontop of the network message system on the server side. Each message we are monitoring goes through

this class and if the message is sent to frequently it gets logged and dismissed if request frequency is to high.

Working ok so far I have a 2060 millisecond elapsed time for the server map list message, and can be spammed 3 times with in 2.6 seconds where it will dismissed by the server 3 time but cant get it to kick yet(not my pressing the refresh button that is).

How do you know what would be a good time value I just pulled 2060 out me ass, first it was 60 that was to short.

I managed to lay the code out to do key look ups on std::maps. Other then that the network system is up and running. All that is left is to weave it into the game.

Heres the gist of the network message monitor system.

.

//-------------------------------------------------------------------------
//this will hold info on the frequence of the messages in milliseconds
//timgettime value
//-------------------------------------------------------------------------
class cMsgMonitorAttributes
{
	
public:
	DWORD LastTime;
	int32_t Frequency;//how many time we did it before the allowed time
	int32_t CurrentNumberOfTries;//when we get to 3 we ckick the client

	cMsgMonitorAttributes()
	{
		Frequency = 0;
		CurrentNumberOfTries = 0;
		LastTime = 0;
	}

	~cMsgMonitorAttributes(){}


};//end class cMsgMonitorAttributes
/////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////








//-------------------------------------------------------------------------
//we need one of these for each network message we want to monitor
//-------------------------------------------------------------------------
class cMsgMonitorType
{
	DWORD MinimumMessageTime;//any thing under this time and the message is rejected
	
	int32_t FrequencyBeforeKick;//the max frequency before we kick a client on the third try
	//client ID and the record for the client
	std::map<int32_t, cMsgMonitorAttributes> MonitoredTypes;
	std::map<int32_t, cMsgMonitorAttributes>::iterator it_type;

public:

	cMsgMonitorType()
	{
		MinimumMessageTime = 0;
		FrequencyBeforeKick = 0;
		
	}

	~cMsgMonitorType(){}

	//-------------------------------------------------------------------
	//must remove dead clients
	//--------------------------------------------------------------------
	void RemoveClient(int32_t id)
	{
		//find and then remove the client
		it_type  = MonitoredTypes.find(id);
		if(it_type != MonitoredTypes.end())
		{
			//we can remove it
			MonitoredTypes.erase(it_type);
		}


	}//end RemoveClient
	////////////////////////////////////////////////////////////////////////


	//------------------------------------------------------------------------------------
	//this is where we add clients based on the passed in message type and the client id
	//if the function returns 0 we can process this message its valid
	//returns 0 for ok to proccess or 1 for kick client to many tries and could be spammer
	//returns 2 for no processing
	//-------------------------------------------------------------------------------------
	int MessageAccepted(int32_t id, DWORD time)
	{
		//see if we are in the list first
		it_type  = MonitoredTypes.find(id);
		if(it_type != MonitoredTypes.end())
		{
			//ok where in the list check the time
			//if the last time and time is under the allowed time we return true
			DWORD ctime = time - MonitoredTypes[id].LastTime;
			if(ctime > MinimumMessageTime)
			{
				//we can process
				MonitoredTypes[id].LastTime = time;
				MonitoredTypes[id].Frequency = 0;
				MonitoredTypes[id].CurrentNumberOfTries = 0;

				//ok we can process this message
				return 0;
			}
			else
			{
				//up data times
				MonitoredTypes[id].Frequency++;
				if(MonitoredTypes[id].Frequency > FrequencyBeforeKick)
				{
					MonitoredTypes[id].Frequency = 0;//reset

					MonitoredTypes[id].CurrentNumberOfTries++;
					if(MonitoredTypes[id].CurrentNumberOfTries > 3)
					{
						//we need to kick client
						return 1;
					}

				}//end frequency
					
				//we dont want to procces this message
				return 2;

			}//end to many recieves for this client

		}//end all ready added to the list
		else
		{
			//ok where not in the list add us and return 0 we good to proccess this message
			MonitoredTypes[id].LastTime = time;
			MonitoredTypes[id].Frequency = 0;
			MonitoredTypes[id].CurrentNumberOfTries = 0;
			return 0;//ok to proccess
		}//end and new client to this message type

		
		//some thing wrong
		return 2;
	}//end MessageAccepted
	///////////////////////////////////////////////////////////////////////////////////////////////////



	void SetMinimumMessageTime(DWORD time){MinimumMessageTime = time;}
	DWORD GetMinimumMessageTime(){return MinimumMessageTime;}

	void SetFrequencyBeforeKick(int32_t frq){ FrequencyBeforeKick = frq;}


};//end class cMsgMonitorType
/////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////





//----------------------------------------------------------------
//so we can initialize the cMessageMonitor with the types of
//network messages to work on
//----------------------------------------------------------------
class cMessageMonitorInitData
{
public:
	DWORD MinimumMessageTime;//any thing under this time and the message is rejected
	int32_t MessageType;//what newtwork type this is
	int32_t FrequencyBeforeKick;//the max the frequency before we kick a client on the third try

	cMessageMonitorInitData()
	{
		MinimumMessageTime = 0;
		MessageType = 0;
		FrequencyBeforeKick = 0;
	}
	~cMessageMonitorInitData(){}

};//end cMessageMonitorInitData
//////////////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////////






//---------------------------------------------------------------------------
//this class is assigned a set of messages and some params for each
//so we can monitor over use of messages and block them
//it will lock with a boos mutex
//----------------------------------------------------------------------------
class cMessageMonitor
{
	//we need a map of messages and a map for each message for client IDs
	//the key is the network message type
	std::map<int32_t, cMsgMonitorType> MonitoredMessages;
	std::map<int32_t, cMsgMonitorType>::iterator it_type;
	//to remove a client we need to go through all the message types and remove the id

	//we lock the above list
	boost::mutex io_mutex;//

public:

	cMessageMonitor()
	{
	}

	~cMessageMonitor()
	{
	}


	//-------------------------------------------------------------------
	//must remove dead clients
	//gets locked down mutex
	//--------------------------------------------------------------------
	void RemoveClient(int32_t id);



	//-----------------------------------------------------------------------------------
	//we call this to initialize the messages we want to monitor
	//each item in the list is a message type and what time we want to monitor
	//we pass a vector of cMessageMonitorInitData for each message me wont to monitor
	//this only sets the message type later we add clients based on the message
	//------------------------------------------------------------------------------------
	bool MonitorNetWorkMessages(std::vector<cMessageMonitorInitData> &initdata);




	//------------------------------------------------------------------------------------
	//this is where we add clients based on the passed in message type and the client id
	//if the function returns we can process this message its valid
	//we lock this with a mutex
	//returns 0 for ok to proccess or 1 for kick client to many tries and could be spammer
	//returns 2 for no processing
	//-------------------------------------------------------------------------------------
	int MessageAccepted(int32_t id, int32_t msgtype);




};//end class cMessageMonitor
//////////////////////////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////////////////////