Hi there !

I've been working for a few month on a peculliar game engine from scratch in my spare time, mostly “cuz i can, then why not ?”
I am only using c++ 20 with the std and following the ECS design.

I'm trying to make the core engine totally independant from external libraries (graphics, sysevents, sound, network…).Not sure if this is good but I'll still go all the way and see if it was worth it ^^

My main motivation here are :

• to not bother with visuals and graphic until a later stage of developpment (do i need 3d at the start ? Maybe just 2d while coding the core principle of the gameplay would suffice) (use cheap & quick lib at first then switch to a better one if you need it) (cross plateform ?). That's a lot to think and it's prone to changes so i just prefer doing it later
• Having a “Pure” core game engine which means it would almost with minimum tweaks be crossplateform. And if the “game” part of the code only depends on the engine that's means the same for your game 🙂
  

The approach I took to resolve this is the following :
A system (which hold the game logic) upon registration can request some services which will be provided upon instanciation. (Dependency injection here). The system will then be able to use the requested service interface that hides which library is actually in use (loose coupling).

I can already see two problems happening in a near future during the game developpement (not only the engine) :

• Handling the loading / lifetime of the lib specific ressources (like a sprite, a texture or a soundPlayer) .
• Making optimization of the lib usage which might depend on the core engine.
  

Is there anyone that have done or heard of loose coupling libraries in game engine ?
I did read a few things about it but it was mainly about making the code not overly reliant on Unity or Unreal Engine (which hold no interests to me because the funniest part of the dev is already done).
It would be even better if you had some ressources about it (article, books, podcast, gravure, I take anything you might have :p )

If needed I can provide my PlantUML class diagram of the engine.
Thanks for your time reading and any suggestions you might have 🙂 !

The challenge with doing this kind of loose coupling is picking the right level of abstraction, and since libraries often have significantly different ways of doing things, you can often end up having to abstract complete subsystems in order to make efficient implementations of the library-specific versions. Unless you do significant planning ahead, or implement multiple versions in parallel, you'll most likely have to do a fair bit of refactoring of those interfaces once you add support for new libraries. It sounds like you are doing a hobby project to learn, so that will be a great exercise.

Your game engine without support for graphics and sound is already useful - it's the feature base you'll need to run the game logic on an authoritative server.

Yes, it's a hobby project but who knows maybe it'll be something someday 🙂 .

Indeed that's the biggest challenge about abstraction ! c++ have quite a few mechanisms that are really helpful and I try to tap in it as much as I can.

That's what i thougth too, my current task on the project is refactoring it as multiple libraries and executable, mono-repo style.
I think i'll start by doing an OpenGl & Vulkan (that one looks cool) implementations so has to keep them up to date with the core engine. I saw web assembly too maybe I'll make a 3rd one if I have nothing better to do ^^

Yup the core game engine is the domain and it must stay pure !

Many game engines are developed in ways that don't require graphics, and you generally can run servers as headless processes. You don't need the graphics to process simulations, although you do need elements like physics meshes.

For the specific elements:

SainteCroquette said:
Handling the loading / lifetime of the lib specific ressources (like a sprite, a texture or a soundPlayer) .

From a purely theoretical aspect, there are four core elements. I tend to call them the store, the proxy, the cache, and the loader.

Usually game engines keep the store and the proxy highly visible, and hide the cache and loader as internal details.

There is a store of “things”, depending on engines these can be the loaded scene or the game world, or other names. There are the proxy objects that represent what is loaded. When a system tells the store “give me an object” it can instantly return a proxy without loading the final resource details. The proxy has enough information to manipulate but not necessarily load the hard resources. In most game engines this is hidden from the gameplay programmers, for example, they receive a “game object” with valid data, you can query metadata about a texture but you don't know if the graphic texture is loaded on screen, and if it is, what level of detail it is shown at. The game object is a proxy for the hard resource. The cache has the actual hard resource of the object (or not), your textures may be loaded or not, your meshes may be loaded or not, your sounds may be loaded or not, either way it's the cache that actually stores them. How they are stored and processed may depend on what the resource actually is, audio data is likely stored somewhere different from a surface texture. And finally, you have the loader that can pull them in across whatever systems are necessary, put the textures into hardware memory, put the waveform into audio memory, and similar.

A sprite can keep metadata about the image without putting the image resource into video memory. A sound player can keep metadata about the position in the audio track without having the actual audio being output to speakers.

SainteCroquette said:
Making optimization of the lib usage which might depend on the core engine.

This requires too much knowledge of the real-world use patterns to do much with.

For board games it not too difficult to abstract the gameplay rules and provide an interface to play with them.

You can build a chess engine easily this way, never exposing the logic of how chess is played but providing an interface for making moves and for viewing the state of the board. You could build more complex games this way with effort, there are 4X games ("Civilization clones") where the map of the world, the state of pieces on the world, the technology development, are all accessible through an interface without tying it to display.

This gets more difficult in physics driven systems, in interactive simulations rather than board games, and anywhere that more complex interactions or continuous interactions need to be processed. Unreal does it to a degree which requires a lot of source diving to discover, look at the difference between how it handles headless servers versus how it handles interactive game clients. The server loads physics meshes and meta-information about textures and models, information about audio, information about lighting, but doesn't bother processing the actual presentation. They do a really good job of hiding it from the user, unless you're digging pretty deep into the engine or doing major development.

Thank you for the detailed answer

frob said:

Usually game engines keep the store and the proxy highly visible, and hide the cache and loader as internal details.

There is a store of “things”, depending on engines these can be the loaded scene or the game world, or other names. There are the proxy objects that represent what is loaded. When a system tells the store “give me an object” it can instantly return a proxy without loading the final resource details.

So in your terms, the store is a collection of proxies: the components which hold game data and no logic beyond guards for multi-threading access.

• The components must be directly accessible at a low level in the engine
• The collection is handled and own by the core of the engine
• At a higher level when coding the actual gameplay of the game it is hidden behind some kind of GameObject which the engine will care to construct from references to the component (proxies)

Then the cache which hold only resource specific data in memory where every resource has it's associated loader.
If I'm not wrong here the cache should be unique for each service. That would means that the entire lifecycle of the resource becomes the responsibility of the service (the loader maybe ?). This should probably means that it would be near impossible to do fine tunes to handle loading / unloading on the fly for open-world games for example.

frob said:

SainteCroquette said:
Making optimization of the lib usage which might depend on the core engine.

This requires too much knowledge of the real-world use patterns to do much with.

Do you mean that it would be game specific and not core engine / lib specific?

frob said:

You can build a chess engine easily this way, never exposing the logic of how chess is played but providing an interface for making moves and for viewing the state of the board. You could build more complex games this way with effort, there are 4X games ("Civilization clones") where the map of the world, the state of pieces on the world, the technology development, are all accessible through an interface without tying it to display.

That's what I want to achieve, and not only for display (events, sound, network [probably the hardest]…)

frob said:

This gets more difficult in physics driven systems, in interactive simulations rather than board games, and anywhere that more complex interactions or continuous interactions need to be processed. Unreal does it to a degree which requires a lot of source diving to discover, look at the difference between how it handles headless servers versus how it handles interactive game clients. The server loads physics meshes and meta-information about textures and models, information about audio, information about lighting, but doesn't bother processing the actual presentation. They do a really good job of hiding it from the user, unless you're digging pretty deep into the engine or doing major development.

One of my first goal once i have most part of the engine ready was to delve a bit in physics systems & simulations :p
I'm not sure how would that get more difficult here. Do you mean by using physics libraries external to the engine ? Or that it get time consuming to get the right architecture to keep it clean without scarifying performances ?

I think i'll start by doing an OpenGl & Vulkan (that one looks cool) implementations so has to keep them up to date with the core engine. I saw web assembly too maybe I'll make a 3rd one if I have nothing better to do ^^

If you want to do a webassembly build, you might also have a look at WebGPU, which is a modern abstraction that is far less verbose than Vulkan. It's supported in browsers (obviously), but there are also a couple of desktop implementations such as https://dawn.googlesource.com/dawn/+/refs/heads/main/README.md​

SainteCroquette said:

My main motivation here are :

• to not bother with visuals and graphic until a later stage of developpment (do i need 3d at the start ? Maybe just 2d while coding the core principle of the gameplay would suffice) (use cheap & quick lib at first then switch to a better one if you need it) (cross plateform ?). That's a lot to think and it's prone to changes so i just prefer doing it later

Developing a game engine with no graphics sounds difficult. I would do at least some graphics up front. That's not to say it has to have PBR or anything, but IMO seeing something on the screen seems pretty necessary for anything but a text-based game engine.

For one how do you intend to debug it? Especially with physics, it's hard to know what's going on, by only looking at numeric output. In fact, I had to build a “replay” feature in my engine so when I hit a bug, I could reliably reproduce it. That involved both graphics and a GUI (in my case Dear IMGUI) to implement.

The other thing is, seeing output on the screen is a great motivator. Game engines are a lot of work and it's nice to be able to take a breather, admire your accomplishments and show them off a bit.

You are right it is a must to have a minimal graphic interface for developping & debugging that kind of things.
What I meant by not bothering with it is to make a really naive implementation of some lib (openGl for example) with just enough to develop core feature of the engine / game. Once you are satisfied with game logic you can start delving into animations, shaders or whatever you want to have something nice on the screen. At this point you can consider what are your specific needs and do a proper implementation that would support these features. You might want to change lib to go to vulkan for example and it must not break the engine as it have no dependency relationship with it

SainteCroquette said:
Then the cache which hold only resource specific data in memory where every resource has it's associated loader. If I'm not wrong here the cache should be unique for each service. That would means that the entire lifecycle of the resource becomes the responsibility of the service (the loader maybe ?). This should probably means that it would be near impossible to do fine tunes to handle loading / unloading on the fly for open-world games for example.

There is no “should be”. It ends up conceptually being whatever works for the system. Each system has lifetime of their own objects. Something unspecified needs to resolve when things are removed, if the game engine says “load this texture, too” but there is no room for it in the video card, something somewhere needs to decide which gets priority, which one is present in the cache and which one is removed. If that means swapping LOD on either one, or removing something else entirely that is now unused, that's managed somewhere. Space is finite.

SainteCroquette said:
Do you mean that it would be game specific and not core engine / lib specific?

You can make informed algorithm selection as an optimization without seeing actual use, but without actual knowledge and measurement you're likely focusing your efforts on the wrong thing. You can spend a ton of time on a system that isn't on the critical path.

You can make your network socket system lightning fast, fast enough to use on high performance stock trading systems, but it is meaningless if the game is doing simple asynchronous chat on a game like chess. An amazing fast pathfinding system, for paths that are calculated for a single object once every ten seconds. A highly efficient physics system used for a pong game. Repeat for whatever system you've got.

In game engines, over time there is enough for broad patterns to emerge, but that quickly becomes based on actual use, not speculative use.

SainteCroquette said:
I'm not sure how would that get more difficult here. Do you mean by using physics libraries external to the engine ?

It is iterative. Very often you don't know what you need until after you've built a few. “Build one to throw away” is a common philosophy, what you learn lets you create a more complex system. Of course, Second System Syndrome is also a real possibility, so often it isn't until version 3 that things eventually becomes viable.

Good system design is good and admirable, sometimes it's better to ignore it and just get something done quickly. How do you know what “good” looks like until you've experimented around it, and also discovered what “bad” looks like?

Brian Sandberg said:

If you want to do a webassembly build, you might also have a look at WebGPU, which is a modern abstraction that is far less verbose than Vulkan. It's supported in browsers (obviously), but there are also a couple of desktop implementations such as https://dawn.googlesource.com/dawn/+/refs/heads/main/README.md​

Didn't knew that one thanks ! Noted that down 🙂

frob said:

There is no “should be”. It ends up conceptually being whatever works for the system. Each system has lifetime of their own objects. Something unspecified needs to resolve when things are removed, if the game engine says “load this texture, too” but there is no room for it in the video card, something somewhere needs to decide which gets priority, which one is present in the cache and which one is removed. If that means swapping LOD on either one, or removing something else entirely that is now unused, that's managed somewhere. Space is finite.

I perfectly understand the concept, I just have hard time having a long term vision. I'll follow the advice and not spend too much time on the architecture design, and only delve deeper when i'll have metrics to do actual comparisons.

frob said:

It is iterative. Very often you don't know what you need until after you've built a few. “Build one to throw away” is a common philosophy, what you learn lets you create a more complex system. Of course, Second System Syndrome is also a real possibility, so often it isn't until version 3 that things eventually becomes viable.

Good system design is good and admirable, sometimes it's better to ignore it and just get something done quickly. How do you know what “good” looks like until you've experimented around it, and also discovered what “bad” looks like?

Right, so all that's left to code right and see.
I'll try doing some kind of devlog if anyone is interested seeing how it's going