Good to know. Too often that's a place where people make some serious mistakes, investing extended time and effort to plug in an architecture designed for millions of distributed transactions per second when a very simple data store is more than adequate.

Very often this type of storage is best pronounced:
std::map< std::string, std::string >

And if you must back it up with a database, run an upsert on it every few minutes.

For a simple server that's good for many gigabytes of information, assuming you're on a 64-bit system like your server probably is.

2 minutes ago, frob said:

std::map< std::string, std::string >

Unless you live in c# land like I do.   haha, then that just looks vaguely like some c I used to know...  How's that song go.. "Like somebody I used to know.." 

lol

"had your friends pick up your records and changed your number..."  bwahahaha

ear worm  GO!!

On a more serious note, if these databases ever exceed 1gig in size, there's certainly a problem.  I think I'm mostly just having trouble explaining how I intend to use these things I ask about.  People see my rep number and think I'm running full speed into a brick wall.   In effect it's going to be more of a messaging system, with ram level persistence.  Because I don't like the idea of messaging directly between machines as that requires WAY more code and way more traffic than centralizing the data ever could, plus that wouldn't be scalable unless I plugged in or wrote something like ZeroMQ...  Bleh..

It isn't about the rep number, it's about how frequently it is brought up.

People come in asking how they can integrate a solution that handles terabytes per second for their networking solution that would take a large team multiple work-years to implement, when they're unlikely to ever produce a finished project.

They come in asking about "what is the best one", looking at online reviews. Perhaps best for a large business system distributed across cities, but not for small games.

On 10/31/2018 at 12:57 AM, Septopus said:

I'm building a massive persistent online world in a "single shard" environment.

 

Hmmmm.... You sound like you're working on the exact same thing I am.  I think I'm just approaching it from a different angle.

52 minutes ago, Gnollrunner said:

Hmmmm.... You sound like you're working on the exact same thing I am.  I think I'm just approaching it from a different angle.

Indeed.  Except, you are performing magic, and I'm playing with legos.  Maybe slightly magical legos, but still.  Lol

@frob Yeah, I can see that.  That's pretty much what most of my posts look like too.  Haha.  To be honest I expected far more criticism and far less actual "community" than I have found here.  Guess that's why I keep posting.

6 hours ago, Septopus said:

Indeed.  Except, you are performing magic, and I'm playing with legos.  Maybe slightly magical legos, but still.  Lol

I wouldn't necessarily say that. In some ways you are farther along than me. Also you seem reasonably competent an you have a lot of enthusiasm. That should take you a long way if you stick with it.

Riak has a lot higher overhead per transaction than Redis. It's a different kind of beast -- more of a document database (like Amazon Dynamo, or Cassandra/Scylla,) than a network-attached-RAM server. Redis is more like memcached, although with persistence. But still not durability! Redis will checkpoint, and you will lose anything changed after a checkpoint when you crash. (It also has other modes, but they too have a variety of problems.)

If you need actual persistency, just use MySQL or perhaps Postgres. Or one of the high-performance reliable key/value databases like rocksDB or foundationDB if you don't need advanced schemas/indexing. (just promise me you won't use MongoDB -- it promises many things, and ends up failing anyone who tries to scale a business and evolve an application with it.)

Regarding shards that move with players: Two players may want to go different directions, so the only way you can avoid having to move players between shards would be to have one shard per player. There are network topologies that work exactly like that -- the original military DIS standard, as well as the over-engineered HLA replacement, are structured like that. They end up bottlenecking on the network, typically at much fewer simulated entities than you could squish into a single process if you used static geographic sharding.

Btw, I can see how the word "shard" is overloaded. In systems architecture, "vertical sharding" is when you separate different functional areas to different systems, that you can scale individually. "horizontal sharding" is when you can distribute the load within a single system onto many different machines (in games, typically geographically.) What players of MMORPGs call a "shard" is generally called an "instance" in systems architecture -- you spin up multiple full copies of whatever your system (game) is, and let players choose one instance to interact with. Instancing is kind-of like another level of horizontal sharding, except not really, because the workload across instances is not interchangeable; a datum (player character) only makes sense within the instance.

Awesome info again @hplus0603 Thanks!  More or less the same conclusion I came to regarding Riak.  More than I need/want.

Redis so far has done nothing but make everything I've touched with it a little(or a lot) better.

I've disabled all of it's checkpoint and persistence capabilities, it's basically just a highly supercharged memcached(preliminary tests of memcached in the same environment agree).  With pub/sub that works amazingly, well enough that I'm completely trashing all the inter-server netcode I WAS writing.  It's really quite brilliant, and everything I was hoping it could be.  I'm kicking myself hard for avoiding centralizing the data sooner.  I'm currently using it entirely asynchronously to store short-term game-state data and for its phenomenal pub/sub capabilities to keep my servers in sync.

By shards that move with the players, I mean something more along the lines of (brace yourself) semi-authoritative clients.  Everybody builds game servers like mainframes, and I know why that is.  I'm not arguing the merits of a fully authoritative server architecture(I'm still building one, sort of), but I can't help but see most of it's faults too and think that there are ways to use the network/system resources more efficiently and more importantly, improve the quality and interactivity of the simulation by making all simulation dependent decisions as close to where the player makes theirs as possible. 

I see it as a trust-but-verify type of scheme with a somewhat decentralized authority, and not one that cares exactly what's being rendered @xyz in every players sim at every single moment.  More what is Feasible to have been(within the rules) or Should Have Been(because a server sent an NPC to get them) rendered near the position they said it was at the time they said it was... I really have no idea how it's going to work out, so far so good is all I can say.   

Ultimately, I envision a game where a lot of the content is AI and/or otherwise dynamic and interactive,yet remains fully operational even if all the other real players have been lag-dropped from your game session.  You can go about exploring the world and/or making changes as you like, even if your lag is so terrible you can't see any other real players.  A system that can filter the game down to its most basic operational state for almost anybody, playing almost anywhere.  Possibly replacing lag-inaccessible player characters with non-networked NPC analogs that derive behaviors from the actual players themselves.  I got ideas...

But that's why I need all the super-fast yet temporary storage that is accessible from all of my various servers.  It will act like a round-robin-database that only keeps a certain window of history ready to re-game cheats(verify/approve client decisions), and for other general behavior analysis and misc. uses by the games AI. 

At all costs I'm trying to avoid writing code that plays with time.  I'm not going to add time to time stamps so they look like they didn't happen in the past, so that my server simulation can allow a loot drop to happen in the future.. or some other bull.. haha   I'm not even sending time stamps with my data.   I'll simply check the immutable historical records to see if what the client said happened(within the tested client latency window of the claim's arrival) vibes well enough with the record, and if so, approve the loot drop that should still be waiting on other processes client side anyhow(like a more realistically elongated death/destruction sequence).

yeah, most of this I'm just putting into words right now, so if it seems a bit unpolished... haha