You can only prevent man-in-the-middle attacks if the key is known to both client and server, but not to the man in the middle. If you encode the key in your client, and someone can get a copy of the client, then the client can be disassembled to extract the key.

UDP already does CRC checking of the packet contents; you don't need to also do that. It doesn't add anything.

I would add a signature to each packet instead; this will both authenticate the sender (protect against spoofed packets) and provide the same packet validation as the CRC would give you.

Last, if you want to avoid man-in-the-middle attacks, look into Diffie-Hellman key exhcange.