Hello! I'm working on an online turn based multiplayer game as a hobby project. My game is implemented using Node.js and Javascript.

Currently I'm designing how implement the persistent storage so that games can recovered in case of server crash.
My idea is to use a in-process memory store for data access in the game server logic. When a store modification is being done, the in-process memory data store is always updated directly and the persistent storage changes are queued. I was thinking of something along the lines of this:

function Store(memoryStore, persistentStore) {
    // Different queues for operations of different importance.
    // If the different operations need to happen in order and be in consistent state, they must be placed in same queue
    this.queue = new Queue(persistentStore);
    // Could be something like game chat. Saving them at different time doesn't affect the atomicity
    this.lessImportantQueue = new Queue(persistentStore);
    this.actionHandler = new ActionHandler();

    this.doOperation = function(action) {
        // Generate applicable change using the memory store as source of truth
        var actionResult = actionHandler.handleAction(memoryStore, action);
        memoryStore.doOperation(actionResult);
        this.queue.push('doOperation', [actionResult])
    }

    this.doLessImportantOperation = function() {
        memoryStore.doLessImportantOperation();
        this.lessImportantQueue.push('doLessImportantOperation', arguments)
    }
    
    this.getSomeData = function() {
      	return memoryStore.getSomeData();
    }
    
}

function ActionHandler {
    // Generates operation using the memory storage as source of truth
    this.handleAction = function(memoryStore, action) {
    	return action.complete(memoryStore);
    }
}

// Stores game state altering operations
function Queue(store) {
    this.items = [];

    this.push = function(operation, args) {
        this.items.push([operation, args]);
    }

    // Can be called directly, after certain queue size or at an interval
    this.synchronize = function() {
        var p = Promise.resolve();
        this.items.forEach(function(op){
                  p = p.then(function(){ return store[op[0]].apply(persistentStore, op[1]); });
          });
        this.items = [];
    }
}

// Special type of store that has synchronous interface
function MemoryStore() {
    this.getState = function() {
        ...
    }
        
    this.getSomeData = function() {
     	return ... 
    }

    this.doOperation = function(actionResult) {
        ...
    }
}

// Uses external persistent storage
function PersistentStore() {
    // Get state as an object that can be used for creating a memory store
    this.getState = function() {
        ...
    }
        
    this.getSomeData = function() {
      	...
     	return ...
    }

    this.doOperation = function(actionResult) {
        ...
        return Promise.resolve();
    }
}

The Store class is used as top level layer for fetching and storing data.

Contrary to the example, I also considered being able to use a remote data stores for primary data access. The good side with that would be that the server wouldn't need to hold the whole game state in memory, consistent game state could be read from different processes and the server would be quite much stateless. The two latter benefits are highly questionable. The downsides I found were
1) either all server code involving data fetching would need to be asynchronous or the whole state would need to be fetched in advance.
2) data fetching would be a lot slower from remote database, even one like Redis

Example:

this.doOperation = function(action) {
  // Generate applicable change using a store as source of truth
  actionHandler.handleAction(mainStore, action).then(function(actionResult) {
    // Main store can either memory store with asynchronous wrapper or a persistent store
    return mainStore.doOperation(actionResult);
  });
}

As the game server is normally a single process, store like Redis should not be needed as primary data store. Deciding between use of the memory store (synchronous) or possible use of both memory store (asynchronous wrapper) and persistent storages (asynchronous) has big implications for the codebase. With choice to use asynchronous stores, essentially lots of code would be asynchronous in places where one wouldn't expect asynchronous code.

Every persistent store function call would be an atomic operation. Thefore the game state would stay consistent if the game server would shut down, whether some of the operations are still being completed or not. This structure would allow changing the frequency of storage which would mainly effect performance. Buffering the operations would allow for less round trips to the database. The performance would not likely be problem with any of the solutions I introduced, but would allow more games to be run in one server.

I'm currently thinking about following questions:
1) Does the choice of the primary and secondary database make sense here?
2) Does it make sense to exclusively use synchronous in-process memory database for game data processing instead of database such as Redis or another type of database like RDBMS or noSQL database?
3) Is the plan for the storing data and queueing reasonable? I haven't seen literature nor good design patterns for this yet. Having to repeat the queue push for every operation seems kind of unnecessary. I'm also thinking if I'm overengineering.

TL;DR: Should a turn based game store its state in memory and use asynchronous periodic backup or should game data be stored remotely directly at cost of lots of codebase being filled with callbacks/promises?

Feel free to also note if my post doesn't provide enough information for allowing discussion.

The latency of requesting data from a remote store will totally kill any attempt to keep game state remote. Compared to all the display information (applied CSS rules and DOM elements) a browser keeps for text on your screen, the state needed even for a highly detailed game with lots of units is pretty small in RAM. I doubt you'll run out of memory.

You also don't want to store each element of the game in its own little chunk or database row or whatever. This generates way too much overhead. A single "JSON.stringify()" on your entire map data structure would be quite sufficient. Also, keeping behavior and actions separate from the raw data is, in general, a good design principle, so not using objects with methods for the raw data would probably help. Instead, use separate functions or objects that are "mutators" on the state. Those mutators may, in turn, end up being your command objects.

There are a few additional patterns that work well for updating game states and checkpointing them.

One is to store the initial conditions, and then each command being executed, as well as what's needed to reproduce your random number generator, and stream it all out. If you need to recover, simply replay all commands from the beginning. If your model is "queue commands, resolve turn" then clearly you have to store all commands per turn, to play them back at the right turns. If you use timing, then your timers and commands need to be appropriately timestamped.

Another is to write "save game" as a feature, and simply call it once after each turn, saving the game state for later restore.

A third is to implement simulation as two copies of state. Commands look at the old state, and transform the old state into a new state. If you can enforce that old state is never changed by the commands, you can build a full chain of all states the game has gone through, and "timeline wind" forward and backward. Although for a very late game with lots of turns, you might eventually run out of space! Keeping last turn and next turn, though, would be reasonable. However, when resolving commands, if you want order to matter (Unit A goes first and kills unit B, so the command for unit B to attack A will not take affect) you have to take that into account in your command resolution -- because, in the constant "old state," unit B still exists! On the other hand, this may be what you want, as it means that turn order don't matter, everybody does their commands, and THEN units that died are removed.

Currently the game stores state for each player separately as the players have their own point of view of the game. Only current state and the actions are stored. One major reason for the separate states is because each player has limited visibility over the game map.
The alternative was to only keep one version of the state and then process it each time to hide the information that the player doesn't have access to. Due to the limited visibility and its effects, this would have required replaying the whole game from begin state when an action is executed. I was thinking that this would not perform too well if the command history size gets to hundreds or thousands. One example showing why this is needed is that in a player's point of view a map node could look different than what it is in reality in the global state
The game state basically consists of the current game world state and all of the actions that have been executed in the game. The slowness would not be visible in a player versus player game, but would probably become visible with AI players.

For example, if there are two players, then using the memory store the server keeps a global state where all game the information is available and two other states for each player that only has the information available that player has access to.
Players get informed of each action that can be seen by them.

In the code, basically when an user completes an action, the following is done
1) Process the action, generating the result. Random numbers are calculated here. This processed version is used for persistence
2) Apply action to the global state
3) Generate processed versions of the action for each player. If the processed action results in any visible action for the player, apply it to player state
4) Send the processed actions (if visible) for each player separately

8 hours ago, hplus0603 said:

The latency of requesting data from a remote store will totally kill any attempt to keep game state remote. Compared to all the display information (applied CSS rules and DOM elements) a browser keeps for text on your screen, the state needed even for a highly detailed game with lots of units is pretty small in RAM. I doubt you'll run out of memory.

 

The client side has a separate memory store of the state, which should have the same content as the server keeps for the player's point own of view. This allows the player to wind backwards and forwards freely on the client side. The remote store would only possibly have been used on server side. Even when only used on server side, the delays would become noticeable at least with AI players. With this, using remote data store in the server processing probably wouldn't still make sense?

8 hours ago, hplus0603 said:

Also, keeping behavior and actions separate from the raw data is, in general, a good design principle, so not using objects with methods for the raw data would probably help. Instead, use separate functions or objects that are "mutators" on the state. Those mutators may, in turn, end up being your command objects.

The game objects do have some behaviour but all the needed state is stored in game object's property collection, which allows easy way of serializing the game objects. It also contains the information needed to construct the correct type of objects back in case of restoration.
The game actions generate operations as result that are used to modify state. This later allows higher level of game customization as the game will have an event system to generate actions. Map creators can basically create their own commands by using the primitive level operations to modify the state.

8 hours ago, hplus0603 said:

You also don't want to store each element of the game in its own little chunk or database row or whatever. This generates way too much overhead. A single "JSON.stringify()" on your entire map data structure would be quite sufficient.

This would definitely simplify the remote storage implementation a lot. The store basically would need to only have a method to store serialized state and retrieve it. I had not considered this approach yet. I was originally thinking of implementing this using a schemaless database but I was still going to implement data manipulation operations in that database also. With overhead, are you mainly referring to this implementation work overhead? I was thinking that the performance wise manipulating the remote database could perform better than serializing and storing the whole state. Implementation overhead is much more significant here and the serialization would definitely be better if its feasible.

This takes me to think between backing up the state by either
1) Storing the global state and visible state for each player
2) Storing the initial state and the processed actions so that the exact same game can be reproduced

Both of these methods have multiple tradeoffs. In the first method the periodic save operations are heavier than only sending the commands. The game restorations could possibly be a lot quicker as the remote store state only needs to be deserialized. The second method saves storage space and performs better during the game as only the action need to be stored remotely. They could be incrementally stored as often as needed without any performance implications. Game restoration could be a heavy operation. This could potentially prevent possibility of effectively inspecting a game state when it's not ongoing. This could be needed as the games can be paused and resumed later through the game's website.

To come to a decision I will likely measure how long persisting a game state takes and how long it takes on average to rebuild the game state from actions using an average length game. In case the periodical save game operations are too heavy I will investigate the second option. In the second option it is inspected if the delay is too great for showing the game information on the game website or if it takes too long to resume a game.

One major reason for the separate states is because each player has limited visibility over the game map.

Are you talking about the client or the server here? The server will typically always have the complete view, but only give each client a separate view based on what they can see.

If you only store client views, then nothing can exist if no client sees it. Perhaps for some games, this might be acceptable, but for most turn-based games, there's "world state" that no player can currently observe, yet needs to be persistent.

1 hour ago, hplus0603 said:

Are you talking about the client or the server here? The server will typically always have the complete view, but only give each client a separate view based on what they can see.

If you only store client views, then nothing can exist if no client sees it. Perhaps for some games, this might be acceptable, but for most turn-based games, there's "world state" that no player can currently observe, yet needs to be persistent.

Ah, this was about the server side. In server side, there is one state stored which contains complete view and additionally a state for each player which has only the information about the world that is available for them.

In the action processing, both the state containing the complete view and the player's state are used to get the final result of their action. As an example in move unit action, if an unit moves towards area that they cannot see yet, but there is a enemy unit in that path, then it is discovered using the complete view. The movement is stopped as enemy units cannot be passed. Then using the player states, each player only gets to see the parts of the movement where the moving unit was in the area visible for them. The action's data will contain all the map nodes that were revealed after that movement. Newly discovered units are then added to the map and the nodes that are no longer visible are deleted from the player's state.
In action processing, the player state can be either be retrieved by having it stored in addition to the complete view. Alternative to having separate states is to create the player point of view at the time of processing the action. However, simply processing the current state wouldn't generate the exact same result as the real player point of view. For example, if one the map terrain tiles is destroyed outside the player's vision, the state generated from the complete view would not then have that node. In the player's point of view that node should still be there as the the player never saw the tile being destroyed. Then the server would mistakenly allow the player to try move through the destroyed object even if the player didn't have the information that it was destroyed. The action processing could become quite heavy if the player point of view would need to be constructed each time from the initial state. Hopefully I didn't miss anything here, I tried to think of a way of how the multiple states would not be needed.

Also, to provide the players their point of view when they join the game, the server needs to either
1) start from the initial state and apply the player specific actions to the state to provide player the current state
2) keep separate states for each player
This also shows some parts of the tradeoff. Separate states allow retrieving the game state quicker. This could useful in case the state needs to be retrieved outside a gameplay session which I described in previous post. For the game itself, creating the player point of view could be fast enough if all actions don't need to applied again each time. Separate states requires more RAM/storage. I chose to use one complete view and separate states for each player.

On client side, when the players join, they request the state from the server, where the server replies with the state for the player in question. When an action occurs, server sends the visible action to client and the client applies the change to the memory store which is located on client side.

Thank you for the replies so far. This problem in first post consists of multiple broad topics/problems. Hopefully this discussion will at least provide insight to implementing such kind of multiplayer game.

Some of these points are important to go through before implementation as a change can require modifying a big part of the codebase. With this I'm mainly referring to the choice of allowing use of remote stores in the game processing. Some simpler games and apparently MMOs can use remote stores in their game logic processing. I'm trying to see where that becomes a feasible option. I could probably create a separate topic for this.