Yes, QUIC is a UDP protocol, so it can work around many of the TCP/TLS problems. It's more comparable to DTLS with an additional layer of framing on top. It's probably possible to build an entire game on top of QUIC, although the infrastructure in the rest of the world hasn't quite caught up yet. Also, QUIC does share many of the DTLS overhead concerns, which may or may not matter based on what your particular game needs are. I have not built anything with QUIC, so I don't know exactly how raw it is if you step outside the “I am Chrome” sweet spot :-)

QUIC also gives you individual channels, similar to what most “lightweight UDP” protocols (like Enet, Raknet, etc) will give you. You may want to check those out and see if it's a good enough fundament for your game. The benefit is that there's some chance it will actually become a widely supported standard, which means that your game could have wider compatibility long term (including the possibility of UDP-games-in-a-browser.)

Jonor said:
Thanks for the information, the biggest hurtle I’ve found so far is finding a example of TLS being used with c++ an TCP. Seems finding documentation is nonexistent or I am just looking in the wrong places.

It is nearly always implemented in a library rather than writing yourself. Just like you could write your own video codecs or your own zip decoder, it isn't normally done. A skilled team could implement it, certainly, but it is far easier to use an existing library that is already certified and validated to work correctly. Both Unity and Unreal support encrypted network connections if you enable it, both added it about two years ago but it requires some effort to configure and to provide a cypher suite. Steam's GameNetworkingSockets are encrypted using a suite based on QUIC and hardware accelerated AES, and Amazon's Lumberyard has optional support for TLS/DTLS.

As for performance, it depends on the cypher suite being used. An older AES256 algorithm is compute heavy but has hardware support on many chips. There are software-only cyphers that are much faster, and some cypher suites that use significantly smaller cyphers sacrificing security difficulty, but for game content where data is stale within seconds that often isn't a problem. (You wouldn't want it for voice chat, though.) No matter what you choose there will be a cost for the cypher system.

I had to think about that topic also several times in the past and will have to in the future as well. My thoughts on the topic are that first, you don't need to use AES256 while AES128 is almost as secure. The increased key size in AES Block Cipher Encryption doesn't really matter to the basic algorithm because it increases only the amount of cycles used to encrypt a block, not the security perse. From experience, even a software AES algorithm is already fast and small enougth to be implemented by your own at a good level of proof.

But the real problem is still the handshake. It turned out in the past from mistakes of several big companies like Sony for example, that reusing the same key over and over is not the best practise and putting a fixed key into the software is also the worst idea you can have. So the most power should be put into thinking about how to generate and provide a proper encryption key. My favorite solution was provided from Bitcoin. Some slight interest into financial stuff and the first Bitcoin hype brought me to study the technology and have a brief insight into crypto security related to it as well. Please note, I'm far away from being a crypto expert but had some success on implementing the algorithms on my own with proof to well established libraries.

Something I really like to use is AES in combination with ECDSA. ECC (elliptic curves) and the data signing based on them, are somehow fun to use and secure as well, to verify identities. So a solution we used in a commercial project in the past was to provide an ECDSA based handshake from totally random keys and establish an AES encrypted secure communication channel with diffie hellman key exchange over ECC. This means that both sides use some crypto math to compute the same public shared key from the ECDSA public key of the other end, which was used to sign the handshake message. You can then use that key to compute another, secret because not shared key-pair, which is different on each side, and feed that key-pair back to the ECDSA algorithm. So instead of the public key used to sign the handshake message, following messages have to be verified with the new not shared but computed key. Because it isn't shared, it is considered to be secure. We then used the public key of the end we want to send a message to AES128 encrypt the message because the sender computed the public key of the target and the target computed the private key to sign their messages with it, which leads to the same public key by design of ECC. We used smaller (128 bit) and so faster key-pairs for generic ECC messages because they are random anyways and compute in less time, while more relevant but less often messages, like login with username and password, are secured by 384 bit key-pairs.

As from what I read, this is somehow similar to how TLS works. However, the only SSL/TLS implementation that I found from googling was that from OpenSSL and to be honest, the library is a mess!

I didn't know about QUIC before but will definitely have a closer look into it as well. In the end it depends how much secure data you transmit between client/server or client/client in a p2p network. Updating position data for some players is for sure less significant if you perform proper sanity checks in your game as chat messages, account or login data. The most important thing to keep attention to and what makes the Bitcoin principle so successfull is to no trust anyone, especially not the croud ?

I've found a few light weight libraries I'm researching to see if any of the options work for me, OpenSSL is such a mess that it's hard to read an get a good grip on it. I am even thinking of just writing all my network code, getting it into bytes then bring someone in to do the security work for me since it's such a new subject for me.

Handshaking only needs to occur once, and the initial connection cost is already covered by a ton of other tasks in games. Key exchange is rather complex and a common source of security bugs for young libraries.

As for the level of security, it helps to understand your threat models and know when your data becomes stale. Most game data is stale within seconds, if someone decrypts it and can use it twenty seconds later won't hurt the game, the big advantage is to prevent a range of immediate communication attacks, so even an easily-broken cypher suite like DES can work, although it doesn't have hardware support these days. Attackers may employ complex tools, but they're not spending millions of dollars to continuously decypher game data, instead those attackers would simply compromise the box. Using a proper encryption system can help solve problems like eavesdroppers and session splitting, but doesn't need a cypher that can stand up for eons.

Voice chat is a little trickier both for threat models and stale data. Some people discuss more private details on game chat, and probably deserves a higher grade of encryption and people would expect more than a few seconds of security. It can also be subject to wiretap laws, which vary around the world.

But the real problem is still the handshake. It turned out in the past from mistakes of several big companies like Sony for example, that reusing the same key over and over is not the best practise

This is why the handshake should take place over an already-secure channel, if possible! Use TLS for login, and exchanging keys.

Separately: Re-using a key is TOTALLY FINE as long as you use a new nonce for each packet. If you can't use a stream cipher, you can look into variable-nonce approaches like CTR block mode. Or just brute force it by generating a new nonce for each packet, and include the nonce in the packet. (The nonce, by itself, is not secret.)

These concerns are of course common – if you can find a well supported library that does what you need, it's probably to go that way. For TLS, OpenSSL API is pretty bad. You probably wouldn't go too wrong by checking out Botan, which does both TLS (over TCP) and DTLS (over UDP.) If you want to try out QUIC, you could try msquic, or lsquic.

@undefined thank you for all this valuable information, just out of curiosity have you checked out MBed before? Its fairly lightweight mBed.org I’ve also looked at QUIC

Jonor said:
thank you for all this valuable information, just out of curiosity have you checked out MBed before? Its fairly lightweight mBed.org I’ve also looked at QUIC

You have an option to wrap by CLI code some C# System class from library, that gives encrypted streams. I do not know how much up to date they are though.

@jonor I only know mbed as an ARM microcontroller development framework, which has … very little to do with secure network protocols. mbed is like “Arduino” or “Win32” or “linux” – a programming environment. It may have some built-in functions for a variety of functions, which would presumably work well on their target microcontrollers, but chances are high that if they support QUIC or TLS, those libraries are actually wrappers on top of some other, pre-existing implementation, and if you're building for a regular computer or server, you're better off going straight to those libraries.

Couple of suggestions:
If using TLS is too much of a hassle, consider using plain RSA.
RSA is quick and easy to implement especially by hand, speaking from experience.
If the CPU cost becomes too much (1024bit RSA costs about an average of 1ms of CPU time per encryption / decryption separately), you can use RSA → XTEA.
XTEA is super cheap (properly implemented about 5µs for 50+ bytes of data) and can be securely used by encrypting the initial XTEA key exchange with RSA, getting the best of both worlds.

Note that while plain RSA is pretty secure, it is vulnerable to some sophisticated attacks / information leakage without proper padding, so make sure to look up OAEP to fully secure it at some point.