Snapshot of Capture Session/UDP Packets: 

https://ibb.co/j1EmmS

https://ibb.co/knqsRS

https://ibb.co/imXHt7

Text I would like to translate/read:

 https://pastebin.com/59NF0jiB

 https://pastebin.com/BQGfdQuc

I also have the .cap file attached but Wireshark won't load it which is why I provided links to the text and snapshots. I captured the data using Microsoft Message Analyzer. 

Log-in-Authentication(LUNI).cap

Reverse engineering protocols using hex dumps is generally quite challenging if you don't have any documentation or source code to compare to. You will obviously need to understand low-level concepts like byte order, various encodings of basic data types (ints/floats/varints/strings/etc) and it would be quite helpful to understand how data is generally serialized to streams (files or network packets aren't that different, though, so if you have written file format parsers, that will help.)

You may in fact be better off taking a disassembler (such as "IDA") to the game client (or server?) and look at what buffer it uses to send the data, and then look at what code writes into that buffer, to try to figure out what goes where, and why. This of course means that you need to understand disassembled code, but that's the kind of low-level information you should be familiar with to successfully reverse engineer a full game, anyway.

Authentication protocols often involve exchanging public keys (which look like random arrays of bytes) and negotiating shared symmetric keys (which also look like random bytes).

4 hours ago, hplus0603 said:

Reverse engineering protocols using hex dumps is generally quite challenging if you don't have any documentation or source code to compare to. You will obviously need to understand low-level concepts like byte order, various encodings of basic data types (ints/floats/varints/strings/etc) and it would be quite helpful to understand how data is generally serialized to streams (files or network packets aren't that different, though, so if you have written file format parsers, that will help.)

You may in fact be better off taking a disassembler (such as "IDA") to the game client (or server?) and look at what buffer it uses to send the data, and then look at what code writes into that buffer, to try to figure out what goes where, and why. This of course means that you need to understand disassembled code, but that's the kind of low-level information you should be familiar with to successfully reverse engineer a full game, anyway.

17 minutes ago, Nik02 said:

Authentication protocols often involve exchanging public keys (which look like random arrays of bytes) and negotiating shared symmetric keys (which also look like random bytes).

I do have the server files which are NOT encrypted. They are fully readable in c++. I know which files are responsible for making/sending the packets however being I am only knowledgeable in java I don't quite understand what's going on with it (also being I'm just learning networking). 

My goal is to be able to read what the client is sending to the sever, not necessarily what the server is sending to the client. 

Again I don't know much about it but Since I already have the code for the server to send and 'encrypt' the packets to the client, couldn't I use that to read the client packets, being the server has to read the packets and send a response back? 

If the server has only the public key of the client (which is a good and common practice), the server can't decrypt the data that it itself encrypted for the said client. Only the client's private key can decrypt.

Note that commonly only the secure handshaking is asymmetric, and after secure channel is established by exchanging the public keys for each participant, the actual communication is done by negotiating a shared symmetric key because they are easier to process and manage. The shared keys can be rotated or renegotiated many times on a secure transit session.

15 hours ago, hplus0603 said:

Reverse engineering protocols using hex dumps is generally quite challenging if you don't have any documentation or source code to compare to. You will obviously need to understand low-level concepts like byte order, various encodings of basic data types (ints/floats/varints/strings/etc) and it would be quite helpful to understand how data is generally serialized to streams (files or network packets aren't that different, though, so if you have written file format parsers, that will help.)

You may in fact be better off taking a disassembler (such as "IDA") to the game client (or server?) and look at what buffer it uses to send the data, and then look at what code writes into that buffer, to try to figure out what goes where, and why. This of course means that you need to understand disassembled code, but that's the kind of low-level information you should be familiar with to successfully reverse engineer a full game, anyway.

I have the program dissembled however out of the 95,000 lines of sub .text files i'm not sure how to narrow it down to finding the one that deals with the encryption of the packets. I can send you snapshots of what I have if you want. 

The library calls to send and receive data for UDP are typically sendto() and recvfrom(). You should be able to find those in the disassembly. If it's on Windows, it may be something like WSARecvFrom() and WSASendTo() instead, which exist in parallel with recvfrom() and sendto(). You should be able to see whichi functions call those functions, and figure out which functions in turn call those functions, and so forth, to figure out how packets are generated and sent, and received and decoded. If you have working executables for both sides, that should be totally doable, although you do need to learn assembly and probably C level programming to do it.

Also, can you run the server binary that you have? If so, you can attach with a debugger, and simply put a breakpoint where packets are sent/received, and trace your way backwards. At that point, it's not that much different from debugging a program you wrote yourself, assuming you wrote the program in assembly and forgot all about it afterwards :-)

While it is true that some protocols will negotiate keys, and some will even use a key exchange protocol and perhaps sign the keys with public/private crypto, my experience has been that most games don't bother. Games aren't particularly attractive targets for man-in-the-middle attacks; instead, the main problem for them is cheating, and cheaters have full control of the client machine and thus can attack the network protocol outside any attempt to encrypt data. Thus, might as well save the effort and debugging pain and not encrypt it.

14 hours ago, hplus0603 said:

The library calls to send and receive data for UDP are typically sendto() and recvfrom(). You should be able to find those in the disassembly. If it's on Windows, it may be something like WSARecvFrom() and WSASendTo() instead, which exist in parallel with recvfrom() and sendto(). You should be able to see whichi functions call those functions, and figure out which functions in turn call those functions, and so forth, to figure out how packets are generated and sent, and received and decoded. If you have working executables for both sides, that should be totally doable, although you do need to learn assembly and probably C level programming to do it.

Also, can you run the server binary that you have? If so, you can attach with a debugger, and simply put a breakpoint where packets are sent/received, and trace your way backwards. At that point, it's not that much different from debugging a program you wrote yourself, assuming you wrote the program in assembly and forgot all about it afterwards :-)

While it is true that some protocols will negotiate keys, and some will even use a key exchange protocol and perhaps sign the keys with public/private crypto, my experience has been that most games don't bother. Games aren't particularly attractive targets for man-in-the-middle attacks; instead, the main problem for them is cheating, and cheaters have full control of the client machine and thus can attack the network protocol outside any attempt to encrypt data. Thus, might as well save the effort and debugging pain and not encrypt it.

Sweet, found the calls (sendto() and recvfrom()) going to dig into them a bit more. If I hit a roadblock I'll let you know, Thanks for the help!

On 2/14/2018 at 10:39 PM, hplus0603 said:

The library calls to send and receive data for UDP are typically sendto() and recvfrom(). You should be able to find those in the disassembly. If it's on Windows, it may be something like WSARecvFrom() and WSASendTo() instead, which exist in parallel with recvfrom() and sendto(). You should be able to see whichi functions call those functions, and figure out which functions in turn call those functions, and so forth, to figure out how packets are generated and sent, and received and decoded. If you have working executables for both sides, that should be totally doable, although you do need to learn assembly and probably C level programming to do it.

Also, can you run the server binary that you have? If so, you can attach with a debugger, and simply put a breakpoint where packets are sent/received, and trace your way backwards. At that point, it's not that much different from debugging a program you wrote yourself, assuming you wrote the program in assembly and forgot all about it afterwards :-)

While it is true that some protocols will negotiate keys, and some will even use a key exchange protocol and perhaps sign the keys with public/private crypto, my experience has been that most games don't bother. Games aren't particularly attractive targets for man-in-the-middle attacks; instead, the main problem for them is cheating, and cheaters have full control of the client machine and thus can attack the network protocol outside any attempt to encrypt data. Thus, might as well save the effort and debugging pain and not encrypt it.

/// Call sendto (UDP obviously)
    /// \param[in] s the socket
    /// \param[in] data The byte buffer to send 
    /// \param[in] length The length of the \a data in bytes
    /// \param[in] ip The address of the remote host in dotted notation.
    /// \param[in] port The port number to send to.
    /// \return 0 on success, nonzero on failure.
    int SendTo( SOCKET s, const char *data, int length, const char ip[ 16 ], unsigned short port );

This is the function the server library uses to send the UDP data. Since i'm reverse engineering this, couldn't I save the "const char *data" to a file and see exactly what the server is sending to the client, before it "encrypts" it? Also I looked at the assembly code for the server and client as far as the sendto() and recvfrom() functions go but i couldn't figure out how exactly I could decode the UDP packet data when being captured live across the server and client. I've looked at where the data values are stored such as "exi' and "esi" etc., how how their moved around/calculated, but I can't see how that would tell me how it encrypts the data. There was a "__security_cookie" function that was called, however just like the other functions, I could't find out how it encrypts the data other then maybe generating a number (Checksum possibly??) that is used to encrypt the data every time it's sent. I've looked at several videos explaining the packets and assembly language and while i am learning how they function it doesn't help me find out how to decode the information i want. I guess, in a reworded version of my overall question, is it possible to decode the series of packets I captured above (in the links) to 'human readable'?

Here's a link to the website with documentation of the information i'm trying to collect (even though it's already documented I'd like to learn how to do it myself). http://lu-docs.readthedocs.io/en/latest/packets.html

Here's a sample format of what I'd like to find. (It's the exact data being sent in the captured packets I have [although mine are still encrypted] however i'm not sure where they found the "53-01-00-00", and what that means as far as the packet itself, and how they categorized the other data being sent into the sections they have [such as "L:512" and the "u32" or "u16")

Thanks for the help. Understand most likely it's stuff I still have to dive into myself but if you know anything more based the information I have that'd be great. Either way I got a point in the right direction 

Since i'm reverse engineering this, couldn't I save the "const char *data" to a file and see exactly what the server is sending to the client, before it "encrypts" it? 

No, the data passed to the sendto() function is the same as you will see in the packet on the wire.

Instead, you need to look back up the call stack and see who calls this function, and then figure out how that data is generated. Sometimes, setting write breakpoints on buffers being constructed will help. (This is obviously easier if the buffers are in a predictable address in memory.)