Advertisement

DLL engine - How to seperate game.exe from engine library's .h files?

Started by May 08, 2020 05:13 PM
8 comments, last by LorenzoGatti 4 years, 6 months ago

I'm in the process of learning how to move engine code to a DLL and have 2 or 3 simple games that can call into the DLL.

But I can't figure out how to correctly separate a game.exe from having to know about the engine lib's .hpp files.

Here is an example. The game engine will contain a Lua state. I named the main Lua class LuaEL for Lua Embedded Language. Here are the first few lines of the DLL's LuaEL class. Notice it requires lua.hpp from the Lua library headers.

#pragma once
#ifdef KRAKENGINE_EXPORTS
#define KRAKENGINE_API __declspec(dllexport)
#else
#define KRAKENGINE_API __declspec(dllimport)
#endif

#include <lua.hpp>

class KRAKENGINE_API LuaEL
{
	static lua_State* L;
public:
	LuaEL();
	~LuaEL();
	static lua_State *getLuaState();

And here are the first lines of my game.cpp test.

#include <KrakEngine.hpp>
#include <LuaEL.hpp>

int main(int argv, char *argc[])
{
	LuaEL *lua = new LuaEL;

So game.cpp needs to know about the engines LuaEL.hpp, that's fine (I guess). But when it reads it in, the LuaEL.hpp requires the Lua library's lua.hpp also and that file has to be made available to the game. I don't want the games to have to know about the engine's library headers, just the engine's headers.

I don't know if I need different headers somehow for the engine and game. I'm at a loss on how to keep the game code from knowing about the engine's library dependencies but I'm guessing it's just from not knowing how to separate the DLL correctly.

Any information on how to handle this would be great. Or maybe I just need to include those library .hpp files in the game and be happy?

🙂🙂🙂🙂🙂<←The tone posse, ready for action.

fleabay said:
Any information on how to handle this would be great. Or maybe I just need to include those library .hpp files in the game and be happy?

If you want to not need your game to include headers of libraries you use in the engine, you need to forward-declare all types in the headers, and only include library-header inside your .cpp files. For example:

struct lua_State;

class KRAKENGINE_API LuaEL
{
static lua_State* L;
};

And inside LuaEL.cpp, you would “#include <lua.hpp>”. You can lookup everything you need to know about it by simply searching for “c++ forward declarations”. Its a good idea in general to do that instead of including headers everywhere.

If thats not enough, you need to create additional abstractions, where the game only communicates with interfaces/classes that you designed which do not keep visible references to library-types. But I'm not going to address that in-depth as I belive forward-declaring those types should be enough.

Advertisement

So one big question for a library is to decide if it will have any forwards/backwards compatibility. Also consider if a DLL actually gets you anything much over a static library. Choosing to use a DLL prevents many cases of inlining for example, that static libs / multiple c/cpp files will do ("link time code generation"), as well as potentially much bigger restrictions on the use of say templates (why bother having a DLL if most code must be in headers anyway?).

  1. In the simplest case, you say that the programs must be compiled against the exact version of the DLL used
  2. Otherwise you might say it must be the same toolchain, settings, etc., so using C++ classes and standard library stuff across the boundary is OK (malloc/free, new/delete, FILE, std::string, etc.), but versions of the DLL are otherwise backwards compatible.
  3. Rely on some platform conventions outside the C++ standard. e.g. Microsoft basically say how vtable's / virtual functions, will work on Windows, be that MSVC, GCC, CLANG, whatever to use the “C++ style” COM interfaces rather than the C ones (and who uses those now?).
  4. Otherwise your basically limited to more of a C library, and make sure each of your objects has an exported malloc/free style function pair. C++ doesn't have a standard ABI even for basic classes, and certainly not standard library objects / allocators.

Any types in your public interface like that `getLuaState` you would need to create a “wrapper type” to contain it with all the methods you wish to expose.

You can generally forward declare any pointer/reference types generally as mentioned, but often you will need to know object sizes, for example because they appear within your own struct/class definitions, on the stack, used in an array, etc.

You can give up some more performance, turn those fields into pointers, or turn your object into a pointer using virtual functions or PIMPL (point to implementation) to hide those details:

struct ExternalThing;
class FOO_EXPORT  FooA
{
public:
    void hello_world();
private:
    ExternalThing x; // ERROR, forward declaration is not enough!
};
// make the problem field a pointer
class FooB
{
public:
    ~FooB(); // Don't forget the destructor, copy constructor, etc. as well as the default needs the full declaration
    void hello_world();
private:
    // NOTE: Any changes here will break compatibility if any program needs to know sizeof(FooB) (arrays, 
    // stack/auto object storage, new/delete, etc.), but is OK in case 1. once you recompile!
    // If you don't recompile, you won't get a load error (like a missing DLL or symbol), but risk memory corruption.
    std::unique_ptr<ExternalThing> x; // OK, size of pointer is always known
    int y;
    float z;
};
// Make all your private stuff a pointer
class FOO_EXPORT  FooC // PIMPL
{
public:
    FooC();
    ~FooC();
    void hello_world();
private: // NOTE: Since this is just a pointer, Impl can change and maintain compatibility, working for case 2.
    struct Impl;
    std::unique_ptr<Impl> impl;
};
    // in some cpp file
    struct FooC::Impl // OK to make any changes to this and be backwards compatible!
    {
        ExternalThing x;
        int y;
        float z;
    };
    FooC::FooC() : impl(std::make_unique<Impl>()) {}; // extra cost of allocating somehow
    void FooC::hello_world()
    {
        std::cout << impl->x << std::endl; // and extra level of indirection
    }

class IFooC // interface
{
public:
    // NOTE, be very careful with virtual function changes, they will break the vtable and not cause a load error like a missing symbol would.
    // For example in COM you will see them make a IFooC2 or such, even if just adding a function in that version onwards.
    virtual ~IFooC() {}
    virtual void hello_world() = 0;
};
FOO_EXPORT IFooC *create_foo_c();
    // in some cpp file
    class FooC : public IFooC { ... }; // OK to make any chane to this and not recompile
    FOO_EXPORT IFooC *create_foo_c()
    {
        return new FooC();
    }

You can also just leave enough space for the private field manually, but I wouldn't really recommend it and be wary if the object size might change (32 vs 64bit, debug vs release, just new versions even). You could also possibly do similar hacks using `#ifdef` macros within the class declaration, but that is certainly getting pretty risky.


class FooD // put on the stack, arrays, anywhere! no extra pointers!
{
public:
    FooD();
    ~FooD();
    void hello_world();
private:
    std::aligned_storage<64, 8>::type x;
};
    // in cpp
    FooD::FooD()
    {
        static_assert(sizeof(ExternalThing) == sizeof(x));
        static_assert(alignof(ExternalThing) == alignof(x));
        new(&amp;amp;amp;x) ExternalThing("Hello World!"); // be wary of exception safety!
    }
    FooD::~FooD()
    {
        reinterpret_cast<ExternalThing*>(&amp;amp;amp;x)->~ExternalThing(); // manually call the destructor
    };
    void FooD::hello_world()
    {
        std::cout << reinterpret_cast<ExternalThing&amp;amp;amp;>(&amp;amp;amp;x) << std::endl; // cast to the actual type as needed
    };
    

@Juliean @SyncViews Thanks. I will be working on this tonight. I am going to try the forward declaring and/or wrapping those classes as a first try and will keep my options open to the other suggestions.

SyncViews said:
Also consider if a DLL actually gets you anything much over a static library.

I'm doing this for several reasons. For one, I will be creating plugin functionality for a DCC in the future and doing this will help me with that .

🙂🙂🙂🙂🙂<←The tone posse, ready for action.

3rd party plugins? Be aware that quite possibly forces you towards restrictions 3 and 4 with C++, because it often isn't really practical to recompile such things every time (especially if one or both projects are closed source), and possibly more towards 4 as people don't like being told what toolchain and version to use.

Apparently Linux and BSD have a C++ ABI since whatever shipped with GCC 3.2, but check the details. See also the “-Wabi” compiler option, so you can still use a small subset of the C++ syntax, but you would need to investigate details (class virtual functions looks OK, namespaces and regular class functions maybe, exceptions really not sure, anything standard library not OK).

SyncViews said:
3rd party plugins? Be aware that quite possibly forces you towards restrictions 3 and 4 with C++, because it often isn't really practical to recompile such things every time (especially if one or both projects are closed source), and possibly more towards 4 as people don't like being told what toolchain and version to use.

I am familiar with the Maya API and it's problems with version incompatibility and plugins having to be recompiled for new versions. I really like the architecture of the Maya API and have some understanding of the patterns required to make something similar.

I know the Maya API is ancient, but it's the API that I understand the best (really the only). More or less, the Maya application provides interfaces for the DLL plugins to derive from. If there is a better, more modern way to create plugins, I'm open to other options.

🙂🙂🙂🙂🙂<←The tone posse, ready for action.

Advertisement

Using scripts for plugins is one option, since the Lua/Python/etc., as well as .NET and Java. As well as maybe being easier for plugin authors, they don't need such a stable ABI as C/C++ where doing something that touches memory layout can lead straight to corruption territory, as the runtime handles a lot of that stuff.

For C/C++ you just have a lot of compromises as mentioned. You can look to other plugins and libraries. I guess things like Direct3D, OpenGL, Win32 API, etc. are good examples of extreme compatibility. And why say ID3D11Device, ID3D11Device1, ID3D11Device2, ID3D11Device3, ID3D11Device4, and ID3D11Device5 all exist, or why “foo.cbSize = sizeof(FOOTYPE)” is sometimes seen.

Dynamic libraries should ideally use atomic value types and signed integer handles so that they can be called from other languages than C/C++, which can reach a wider audience of developers. This will however prevent automatic garbage collection unless reimplemented in the wrapper using manual counting across the API. To make a DLL library, start by writing example use cases of how a minimalistic use would look, without considering the effort that goes on behind the scenes to make it happen. Closely to making a theatre for the caller who does not need to know the internals.

Otherwise I would just make it into a folder of source code that automatically recompiles to a static library with the same compiler used by the application. (stb_image is just a header to include) This of course requires keeping any dependencies pasted inside the folder. Keep the amount of code to a minimum for compilation time. Making your own custom-purpose script engine and media layer will make the pieces fit together with less type-mangling boiler-plate and unused functionality. Pasted-source libraries are for users who want to easily fix your bugs, so code readability without hacks is then essential.

If you like scripting and don't mind a 10000X slowdown (compared to expert optimized assembly), you can also release the library as an executable reading the entire game as a script (like Flash or Java). Then do like MatLab and include a huge set of vectorized functions to reduce the need for virtual cycles. Instead of looping over 1000 integers and adding them one-by-one, you have a function adding integer 1..N between three buffers using SIMD vectorization. Even multi-threading can be used for the very big buffers if you simulate multiple ALUs in the virtual machine. Then automatically optimize into compound functions like CPython does.

A fourth option is to integrate a just-in-time compiled Java backend to give near C++ speeds and hot reloading of code in your own IDE. This is the hardest, but would be able to compete against Unity and Unreal if done well. This would however be much more likely to attract contributors fixing bugs if you have rolled out the red carpet for new users. Most contributors help with software they actually use.

SyncViews said:

So one big question for a library is to decide if it will have any forwards/backwards compatibility. Also consider if a DLL actually gets you anything much over a static library. Choosing to use a DLL prevents many cases of inlining for example, that static libs / multiple c/cpp files will do ("link time code generation"), as well as potentially much bigger restrictions on the use of say templates (why bother having a DLL if most code must be in headers anyway?).

This problem motivates a rather widespread design pattern: placing the unchangeable game engine in a very large DLL, or even an executable, allowing good optimizations, and placing the entry point for a specific game in a small script or “driver” program that doesn't have, but doesn't need, optimizations and fast calls across DLL boundaries. In extreme but common cases the end user specifies what to play from scratch (e.g. Doom source ports that include no assets of their own and load an open-ended set of mods from files).

Omae Wa Mou Shindeiru

This topic is closed to new replies.

Advertisement