Hey folks!

I'm planning to write game engine in the future with C++. For now, I'm just studying this area. But I came across problem with live coding in editor. Reopening the editor and loading all game assets at every time code changed is a big problem. There are some solution for this problem. For example:

1. Using scripting language
2. Compiling to .DLL and dynamically loading

I wanted to choose first approach in the beginning. Even I successfully embedded 3 languages (lua, c#, js) into hello world c++ program. Each one have it's pros and cons. For example:

• Lua is very small (under 200kb) and very fast (with LuaJIT), but has less features.
• C# has lot's of features (generics, classes, lambas etc), but it's size is very large for mobile games. (~11MB with Mono-C# on android build)
• JS is simple and powerful language. But it's slow without JIT. For speed, you must use JIT based engines like V8. But iOS don't support JIT. I tried Duktape, but it's slow.

But I wanted to give a chance to second approach. Because it's very fast and small. But C++ is a dangerous language. You can crash whole editor with small bug. And memory must be carefully managed. TLDR; My questions are:

1. How can I hook into malloc of dynamically loaded library? With this way, I can free all memory allocated by dll before unloading it.
2. How can I handle exceptions? Especially CPU signals, e.g.. division by zero, access violation (segfault). I tried Windows' Structured Exception Handlers but I failed.
3. Is it possible to run dll inside virtual address space? With this way, programmer can't mistakenly alter editor's memory. Or don't give access to dll to write outside of it's space. This is something like checking range of pointer.

Thanks.

Your should think about the second approach first because you wont couple your changed game code directly with the editor rather than define an interface from your editor code to the gameplay code you changed.This means you might need some kind of communication between your editor and your library in a way that both understand each other.

One thing I came across this topic was when seeking for a better implementation of C++ RTTI (e.g. Reflection). I read a post from a developer (I didn't find the blog now) that explained their use of RTTI for hot reloading of their game code in editor. I implemented a kind of C# Expando Object in C++ using my own RTTI (Reflection/ Meta) System so that C++ code can make assumptions about objects it gets and call methods without the need to know the object exactly rather than keep a list of property/ function pointers and an instance pointer to the object instance.

18 hours ago, elgun said:

How can I hook into malloc of dynamically loaded library? With this way, I can free all memory allocated by dll before unloading it.

You can't (that easy) but force your team mates to keep certain kind of code quality and use allocators you provide in the core code for your game/ editor. Another way that is really unsafe and not platform independent is to replace malloc/ free assembly code and hook yourself into it. I have seen this example when doing some toying arround VirtualAlloc on Windows and determined possibilities for Anti-Cheat and game hacking protection approaches when something in your address space is called, it will be routed to a custom implemented function before calling the "real" WINAPI function pointer.

19 hours ago, elgun said:

How can I handle exceptions?

This is a complicated topic. You could try/catch anything inside your editor code that will call your game code but inside the game code there are barely chances to prevent editor from crashing. But this is also a Unity issue when writing code that forces editor functions to crash so I won't keep to much thoughts on that.

19 hours ago, elgun said:

Is it possible to run dll inside virtual address space?

They run always in your programs address space

7 hours ago, Shaarigan said:

They run always in your programs address space

There are VirtualProtect() in Windows, and mprotect() on linux for marking memory pages readonly. But I don't know is it possible to safely mark whole program readonly except dll's memory

On 9/11/2018 at 5:55 AM, elgun said:

• How can I handle exceptions? Especially CPU signals, e.g.. division by zero, access violation (segfault). I tried Windows' Structured Exception Handlers but I failed.

Throwing exceptions across DLL boundaries is not generally a safe thing. I seem to recall that it at least requires that the DLL and the executable using it were built with the same compiler (and the same compiler version!) and runtime libraries and that the exception isn't allocated on the heap by the DLL and then deallocated by the executable. Throw C++ exceptions across module boundaries puts you in for a world of pain. I suggest re-investigating SEH or not using exceptions as an error handling mechanism.

Have you seen how the Handmade Hero project handles this?  Here is the first episode on the topic, believe 22 & 23 cover it also, it does use C but that is not an issue honestly:

Is there any simple language that can be both compiled and interpreted? Especially intended to scripting.

15 hours ago, elgun said:

But I don't know is it possible to safely mark whole program readonly except dll's memory

You missunderstood the use of VirtualProtect. It dosent protect your memory from being manipulated or even protects a DLL, VirtualProtect just sets kernel mode flags to system pages/ memory while those memory is accessed by the kernel.

When the kernel wants to read data from certain memory because user-mode code tells it to do, a protected memory Write and or Execute will cause an exception.

When the kernel wants to write data to certain memory because user-mode code tells it to do, a protected memory Read and or Execute will cause an exception.

When the kernel wants to execute assembly code from certain memory because user-mode code tells it to do, a protected memory Read and or Write will cause an exception.

What you usually do with VirtualProtect is to change certain parts of your memory for certain purposes. Like in the SO question I linked above, you need to change the memory location of your function pointer to Read/Write first. This way you could change the address of the first instruction to one of your functions, this function can NOT be called in this mode because kernel will complain about execution flag. Then you change the flag back to Read and/or Execute so the function could be again called properly and will on execute JMP into your hook function, for example to prevent cheat software from working.

13 hours ago, elgun said:

Is there any simple language that can be both compiled and interpreted?

Any language could potentially be compiled and also interpreted but if you want natural support, you need something like Java or C# because of their runtime character.

But I think your intention is wrong. What are benefits of having a compiled language that also will be interpreted as script when you need external bindings for example to the rendering API? How do you plan to include such a language into your engine code when shipping the "editor scripts"?

If you want the Unity approach, then you have to embedd a runtime into your engine and treat anything "as script", even if it will be still compiled e.g. translated and so may result in the same unsafe behavior that you try to prevent here.

Again, Unity dosen't care about editor crashes or the even worser editor deadlocks above C# exception level because they can't catch exceptions that aren't handled by the runtime