I am wondering if anybody has the experience of writing a compiler/virtual machine/profiler system that supports self-modifying code. This is of great interest to me, since in some cases it can be applied to produce cool results. For example, the hotspot technology in Java allows the vm/profiler to identify the hot chunks of code, and optimize them. This is quite cool since it can combine runtime information to do the optimization without users' interaction! Does anybody think of any other application of self-modifying code or how to do this? Any bit of info on it would be very welcome!

Just a note:
As long as you target byte code only, self-modifying code definately is an option. Self-modifying machine code, however, is considered harmful and might even prevent your app from running on platforms that support execution protection (e.g. Windows XP 64™.
Since I don't have WIndows XP and also don't have an Athlon 64 processor, I cannot tell whether this protections prevents self-modifying code in all cases. It's just something you should check.

Regards,
Pat.

The protection in Windows XP is simply to stop execution in memory that's not marked for it (i.e. the stack). Specifying PAGE_EXECUTE_READWRITE when allocating the memory with VirtualAlloc, or with VirtualProtect for already allocated or static memory will allow you to execute code from there.