I'm currently porting a game to MacOSX, and most of it is working, but when I'm press ESC to quit the game, it quits, and the a dialog pops up, saying "<nameofgame> quit unexpectedly, open again or ignore?"

Now, I want to know if anyone knows what is the cause of this behavior.
When the ESC key is pressed, I release all resources in memory, destroy the application window, and call exit(0), what could be the problem?

Are you sure that 0 is being returned to the OS? Run your program from the command line and examine the value of $? after it finished, e.g. use the echo command.