I think of you have something that allows modding on pc generally that is a plus point. And if you try and use this as an anti-cheat measure in an mp game people are going to break it anyway (where is it going to put that crypto key? And if you use any well known libraries it is likely to be easy to locate in the compiled code).
Now if you really want to protect a file (although generally not find the owner of the computer for the reasons mentioned. More useful against loss or some remote hacking.), basically you are just looking at regular encryption algorithms like AES 256, maybe with digital verification using say SHA256 HMAC and maybe also something like RSA (you encrypt the AES key itself with RSA so you only need to be able to decrypt then have the AES block cipher for the content). Crypto libraries will have all these things included.
The own file type is just obfuscation, people reverse engineer binary file types all the time. The reason to go with a custom file type should be performance, as you can create something without expensive parsing/transforms, and often more compact than a deflate/gzip/etc. XML.