Does anyone have any insight or experience with DirectX or Directsound in terms of hotloading C/C++ code? I have a lot of code running inside of a DLL that gets hotloaded at run-time frequently. I've noticed that if I attempt to initialize and setup either DirectX or Directsound within the DLL I seem to hit undefined behavior.

My best guess so far is that these APIs are using some static memory somewhere to implement COM interfaces (like pointers sitting in static memory, but initialized at run-time). For example if I reload my DLL DirectX seems to mostly work, at least locally, until Present is called, which results in a strange NULL pointer access violation within d3d9.dll (from one of my GPU vendor's driver threads).

So far the only workaround I've been able to get working is to initialize DirectX/Directsound from within my main executable and pass along associated pointers to my reloadable DLL. This sort of sucks though, since personally I was hoping to keep these technologies localized to the DLL and avoid extra header inclusions for the executable.

Anyone tried experimenting with this sort of thing before?

Sadly I don't have any experience with DirectX or Directsound dependent modules hot-loading. In my work we are using some hot-loaded modules which are DLLs (whole project is in C - which makes some things a lot more simple, and also we have certain rules what these modules should do).

You generally (when using hot loading) need to make sure that there are no direct references to code in memory (address will change with recompile). Everything that would use function pointer needs some kind of "dispatch table". These summarizes some rules we had for those modules - generally if we ever wanted to use 3rd library inside, we were asked to dlopen/dlclose it ... and use dlsym to obtain pointer to any function it calls. I can imagine this might cause quite a mess with virtual functions.

As those modules were mainly just dumb loaders or "simple" procedures that were called on data - it is a simple case compared to yours.

I have never tried or heard of hot-loading DLL that would implement the renderer itself (to be honest I never thought of it!), while I can imagine that this may actually work with Vulkan (where you are able to obtain all function pointers to Vulkan functions dynamically), I don't think it will be that easy with Direct3D.

Although it definitely sounds like an idea worth trying (You probably just stole the weekend from me!).

Thing is with OpenGL I’m loading up all functions with GLAD (a bunch of loadprocaddress calls). This seems to work just fine. But with DX and DS I’m not really sure how one would pull out all the functions, since they’re all these wacky COM interfaces. And even if this succeeded, there still might be some static memory used within the APIs that had been cleared to zero due to the DLL reload. This would all be simpler to test out if I knew how to loadprocaddress on all the DX/DS functions I need.

Or maybe even worse if the code address changes for the DX/DS virtual functions. I’m not exactly sure how Windows considers DLL dependencies. If d3d9.dll is linked by my DLL, and mine is reloaded, would d3d9.dll also get reloaded? This would invalidate the vtable of any virtual C++ objects from the DX/DS APIs. I’ve been trying to read up on all the shared lib dependency rules, but am having trouble finding good information on the topic. It seems like if the executable depends on dsound.dll, then when my personal DLL is reloaded the process does not release dsound, since the executable still holds a dependency reference. So my workaround seems a little like working around reference counting... And that seems silly to me.

For now I’m pulling in the appropriate functions with pragma dll (which I suppose deals with import symbols at link time? I’m not exactly sure how import libs and the like work on Windows and what the implications are for DX/DS dependent libs being reloaded), and haven’t done anything special when hotloading my DLL.

https://docs.microsoft.com/en-us/windows/desktop/dlls/about-dynamic-link-libraries

OK I found a good link that describes various DLL behavior on Windows. My assumption about DLL reference counting was correct, and I am indeed using the "load time linking" via import libs.

So there are two options I see:

• Link to the DLL in question in the executable, so that the reference count for that DLL for the entire process can never hit zero. Hitting zero will flush the DLL, causing any function pointers to be invalidated (and other stuff too). This can cause all kinds of problems (and it does), depending on how the internals of DX/DS are implemented.
• Manually load/unload function pointers/vtable pointers from the DLL itself (with dlsym or GetProcAddress). This still might not work if the DLL gets flushed, since DX/DS objects can contain vtables or pointers to static vtables (not implemented as a jump table, like you mentioned above).

The first option is probably the only good one. Though I imagine it might get a little bit tedious to *make sure* the executable references the DLL in question -- for example I would be inclined to not even include DirectX headers in the executable, since I only want to use DirectX in my DLL. But, I don't want to mess with the linker stripping out the "dead references" to d3d9.dll or dsound.dll, so I'll probably just initialize DX/DS in the executable and call it good enough.

The only downside when linking DLL in the executable is for the case when you will use DLL which will use OpenGL/Vulkan context. At this point you will have in memory loaded both - DX and OpenGL/Vulkan. This won't cause any problem though (apart from using a little bit more memory).

The linker will strip dead references, I can definitely confirm this for GCC at least (just build a project with some additional linker dependencies, and do ldd (under Windows you can use Dependency Walker for example) - they will not be mentioned). In that scenario you will need at least an include and at least one call.

A shot in the dark, have you tried initializing COM for your reloaded DLLs? (CoInitializeEx)