I had a bit of stress at my dayjob this spring with a lot of overtime, so I kinda lost the motivation for working on my game. This last week I've been getting into the game again. Since I've kinda forgotten a lot about working in Unity I wanted to start on something that I could do purely serverside (C#) - refactoring how storage is being handled. I was already performing database transactions asynchronously, but it was a bit cumbersome and not even implemented safely. DB tasks were (sometimes) being enqueued in a thread-safe way, but they were referencing live game entities themselves, thereby sharing them between the game logic and database layer, without any locks, mutexes or other syncronization constructs - OUCH!
Other times blocking DB queries were being issued directly by the code that handled network messages from the client (login, create player etc.). Obviously this would starve the networking thread pool for no reason, and also result in DB queries waiting on locks - also ouch... 

Basically it was all a bit of a mess. I wanted to move more towards a share-nothing architecture, and a single writing DB thread. I have to give credits to It-Hare (Sergey Ignatchenko) for his inspirational articles. He's writing a multi-volume book on MMO development and has a lot of early versions of the articles for his book freely available on his website.
A couple of articles with relevance to what I'm doing atm:
http://ithare.com/gradual-oltp-db-development-from-zero-to-10-billion-transactions-per-year-and-beyond/
http://ithare.com/eight-ways-to-handle-non-blocking-returns-in-message-passing-programs-with-script/

I also wanted to be able to batch up logical transactions in a single database transaction, enabling me to:

1. Issue batch updates and inserts

2. Cut down on the amount of transaction logfile thrashing

This would provide me with a higher potential throughput at the cost of some latency which doesn't bother me as long as the player is able to log in within a few seconds. The game itself never waits for database transactions anyway, so these "few seconds" are an acceptable upper limit for latency of DB transactions.

In addition to that I wanted an architecture that could support moving the DB server to another computer.

This increased latency would of course make any blocking calls to the DB an even bigger No-No than it already was, so reading the articles (among others) mentioned before inspired me to have a single thread for persistence, with logical database transactions being put into a queue (coupled with continuation lambdas) to be batched up and executed.

So I made a persistence server with an app-level cache where transactions can be queued and a single DB update thread that just pulls transactions from the queue, performs them, commits the batch and returns the results of the transactions to the caller.
To be honest this has not (yet) been implemented as a server but just as an in-process object with an Enqueue() method and an OnResult() event. I don't know which communication protocol I would use - Rest, homebrew TCP, something totally different? But I don't want to bother with it now anyway since it is not important at this stage and whatever I choose would probably be changed anyway. I also decided not to decide on a storage technology. My old DAL layer used SQLite but I want to get rid of that third part dependency for now so I'll just be saving JSON files to avoid having to update schemas with frequent data changes. I'm actually not even doing that yet - I'm just using the app-level cache itself which means nothing is saved on program exit
I then made a PersistenceController that will run on the gameserver.
It has methods like LoginAccount, UpdatePlayerCharacter etc. In addition to the normal parameters they also take Action/ Action<T> parameters so you can supply a lambda expression to act as a continuation on completion of a transaction.
These methods just enqueue the transactions, letting another thread handle the sending of the transactions to the persistence server. This transaction sending is also being done in batches from the controller to cut down on ping-pong - these batches do not nece
Mutable reference-type parameters (basically anything heavier than a String or an int) are mapped to DTOs before being enqueued, so that no live game data is shared between threads.

Time for a little code, showing how the handling of a request for availability of an account name resulted in a blocking call to the database. Fine for serving an individual client with low latency but blocking a thread on the iocp pool for several milliseconds? A horrible idea in a multi-player game. 

Notice SendMessage doesn't in fact send anything - it just queues the message for sending. 

public override void Execute(NetworkMessage message, ServerConnection con)
{
	EndianBinaryReader reader = message.GetReader();
	string name = reader.ReadString();
	bool accountNameAvailable = !PlayerDataMapper.AccountExists(name);
	con.SendMessage(ServerMessageFactory.CreateAccountNameAvailableResponse(name, accountNameAvailable));
}

Here is the same code using the persistence controller, with the continuation as a lambda expression. 

        public override void Execute(NetworkMessage message, ServerConnection con)
        {
            EndianBinaryReader reader = message.GetReader();
            string name = reader.ReadString();
            PersistenceController.AccountExists(name, exists =>
            {
                bool accountNameAvailable = !exists;
                con.SendMessage(ServerMessageFactory.CreateAccountNameAvailableResponse(name, accountNameAvailable));
            });
        }