So I've decided to create a simple game engine for learning purposes & I've decided to use minimum possible third-party libraries(but of course I'll use some of them for topics where I know nothing or very little). I've also decided not to use STL at all and minimum possible help from CRT(e.g. no malloc or free, no default new/delete etc).

But after this said, now I have 2 main questions at this point of time:

1. How to link(in terms of C++ building process) engine, editor & game itself. Many engines have games as an instance of some class(e.g. Application) that has several methods to control game flow(OnStartup, OnWorldLoaded, OnGameTick, OnShutdown, etc). Game also can create & register methods for custom events(keyboard, mouse, window, network, etc). But afterward I have no idea where I have to put an editor in this system(because the editor should not be a part of the engine, because the engine has to be able to work without an editor in release builds). Also another question is should I make a game as dll and engine as exe or they have to be linked statically(this may be a good solution for public builds, but what about Debug mode & problem with editor linking).
2. The second big question now is memory management. I want to do it myself instead of using default malloc/free or new/delete. There are few reasons for this: low & high-level profiling, different allocators for different tasks and so on. But I haven't found any good materials on this topic. Most of them use strategy "allocate a big block of memory during startup & use it as it is", but I don't think this is a good idea, especially when we talk about modding possibility because then we don't know how much memory the game will use. I don't want to support consoles at all, so the main platform, for now, is Windows with its virtual memory and maybe later I'll try to add support of Linux & OSX. Now I have 2 main strategies of memory management in mind: create a GlobalHeap class(which will be a singleton) & do all allocations from it(and probably create different allocators for containers or resource system or rendering system) and another way of doing things is to create some kind of allocator hierarchy, where there is a PlatformHeapAllocator, which manages allocations from platform layer, and BlockAllocator, which works just like PlatformHealpAllocator, but it gets memory from another allocator instead of platform layer, and StackAllocator for per-frame allocations or PoolAllocator for fixed-size allocations. I've tried to find how modern AAA game-engines do this, but due to my lack of experience in this topic I haven't found anything suitable. In fact, memory management is the reason why my previous try to make a game engine failed before it event started.

So after all this being said, I hope you will share your thoughts on this topic and help me to do right choice, I'll be thankful.

Poseydon42 said:
In fact, memory management is the reason why my previous try to make a game engine failed before it event started.

Then my only advice is: immeditely stop worring about memory management and drop the idea of needing a sophisticated handcrafted solution. Why? Because its not mandatory.
In my own game engine, I only use custom allocation schemes in very specific use-cases (script compilation; per-frame allocations happening in the renderer), which are very local, very easy and don't need some overarching management.
Other than that, you really can easily get away without manuallly managming memory, if you keep in mind a few simple points:

1. Don't allocate unless you need to. Prefer local variables, store members by value etc…
2. Make use of C++s modern features. Make use of std::move to avoid copies, be aware of RVO/guaranteed copy ellision (which also usually just means: use local variables)
3. Use linear containers (vector+reserve, array) where possible. Be aware of intrinsic properties of all containers, ie… you don't need to store an object as pointer in std::map<key, value>, as “value” will never change its location unlike vector
4. Don't allocate in tight-loops or within your game(logic) loop. Try to precalculate things on startup as much as possible.
5. Stop worrying too much about allocations in one-time, out-of-line steps like loading assets, parsing files etc…
6. Optimize for allocations if you notice that performance is undesirable, and then only after measuring and finding conclusively that allocations are really the problem

;TL;DR there's nothing wrong with wanting global memory managment, but when its literally preventing you from making progress, screw it. You can make games and engines without it, and without automatically crippling your performance. If you thoughtfully use C++ as it is intendet and don't treat it as if it were Java or C# (= don't “new” everything), then there are way less allocations than you might think.

Poseydon42 said:
How to link(in terms of C++ building process) engine, editor & game itself. Many engines have games as an instance of some class(e.g. Application) that has several methods to control game flow(OnStartup, OnWorldLoaded, OnGameTick, OnShutdown, etc). Game also can create & register methods for custom events(keyboard, mouse, window, network, etc). But afterward I have no idea where I have to put an editor in this system(because the editor should not be a part of the engine, because the engine has to be able to work without an editor in release builds). Also another question is should I make a game as dll and engine as exe or they have to be linked statically(this may be a good solution for public builds, but what about Debug mode & problem with editor linking).

Now there are many different ways to do it. My engine is a little over-generic for what most people would want to do, so I can only give you slight hints.
Personally, I have the main engine in a DLL, which is used by both a “player” (= application that runs the game, which the user will run), and the “editor”. Since I use a plugin-architecture, there is also a shared “editor” API dll.
Both player and editor simply use the engine to display the game in different forms. Player will render the game as its main feature, while editor will add lots of controls and render the game as part of one of its windows. Lots of features are reused from the engine to get the editor to running, ie. the asset/resource-managment for loading assets needed by the editor.

but when its literally preventing you from making progress, screw it.

Oh, maybe I said it wrong. In fact, I've actually managed to create my custom heap allocator and make it about 2-3 times faster than CRT malloc&free. The problem actually was that I haven't got any idea whether I was doing this in right way, not in coding. I want to implement it on my own for learning purposes so I asked this question. It does not really prevent me from doing other parts of game engine, but I was spending about 30% of my time for doing programming searching articles about memory management.

About second question, I like your way of doing things and I'll try to make my engine in about the same way, but I've spent some time exploring source code of UE4 and other engines like Godot and most of them use architecture where game is just a plugin for engine & engine itself is a core of application that manages everything, game just adds some very specific functionality on top of it. So what do you think about this method? I'd apreciate if you'd help me again.

Poseydon42 said:
Oh, maybe I said it wrong. In fact, I've actually managed to create my custom heap allocator and make it about 2-3 times faster than CRT malloc&free. The problem actually was that I haven't got any idea whether I was doing this in right way, not in coding. I want to implement it on my own for learning purposes so I asked this question. It does not really prevent me from doing other parts of game engine, but I was spending about 30% of my time for doing programming searching articles about memory management.

Ok, then you can (mostly) ignore what I said, though the general tips towards avoiding allocations still hold on. But if you are actually fine with what you are doing, more the power to you ?

Poseydon42 said:
About second question, I like your way of doing things and I'll try to make my engine in about the same way, but I've spent some time exploring source code of UE4 and other engines like Godot and most of them use architecture where game is just a plugin for engine & engine itself is a core of application that manages everything, game just adds some very specific functionality on top of it. So what do you think about this method? I'd apreciate if you'd help me again.

Yeah, I have pretty much the same architecture. All engine, editor and player are game-agnostic, and each game is one plugin like any other (with an editor-part that is compiled out for final builds). Plugins can add asset-types, systems as well as data/asset-files.
It makes things quite flexible, but also requires a really open architecture, to being able to code your entire game w/o touching the core parts of the engine. But if thats up your alley, sure, can be done, personally I'm working with it with.

The first thing to consider is what kind of library you're making and what audience you're targeting. Keep in mind that these sheep and wolf generalizations are the extreme ends of the scale and some use cases are in between.
* If it's supposed to be a complete platform which doesn't require additional dependencies, you can make your own code generators or virtual machines and hide it in a fancy IDE or game maker. Alternatively make an ActiveX wrapper for .NET. Users of these libraries expect you to hold their hand all the way. They don't mind learning a new language as long as it reminds them of something they already know. This is what makes Arduino so good, because it looks like C/C++ but gets rid of all the ugly and fragile boilerplate code.
* If it ends up wrapping lots of other libraries, you might want to move those dependencies to SDK examples so that the developer can choose other libraries based of platform availability. Then you really have to hide your own types internally and respect that the user probably already adopted the math types of other frameworks. Taking two integers as arguments is then a lot more appreciated than mangling from one library's 2D integer vector to another with a quadratic growth of conversion functions. Not having any custom types is what makes FastCV so easy to just plug in to existing projects. These users (at least think that they) know what they're doing and will be annoyed by safety and performance overhead. They will not accept having your library controlling the main function and call flow, nor carry around a context that's not even storing anything for the sake of being object oriented. They might be reluctant to admit that something's hard to use and silently use another framework instead of asking for help.

Second is how to make your core logic reusable independent from external changes. You don't want your math framework to depend on something in an SDK which might not be there in the next version.
* Abstraction layers are easier to link with as a core part of your library, for matured dependencies like SIMD vectorization that will last for a few more decades or can be emulated.
* Dependency injections are easier to modify for temporary things like window managers by inheriting a class and calling the operating system for each basic task.

Try to have as little as possible exposed in headers, include other headers in the cpp files if possible. Any bloated list of system specific headers should have a wrapper to only include them once in a cpp file. I prefer classes hidden inside of cpp files for fast compilation, with global functions forming a backwards compatible API on the outside to prevent memory corruption when accidentally linking to a library with an outdated header or vice versa.

If an algorithm is very advanced and useful, consider making an interface that can be called without the rest of the framework for future reuse. This is much easier to do using functional programming because you don't need your class factory, meta type or garbage collector included just to call a single function.

I also stay away from std::vector because indexing with unsigned integers can cause bugs in more advanced algorithms where you need to loop backwards without underflow in the stop criteria. Just another source of annoyance that slows down development by having to think about it. std::vector also require you to pre-allocate everything manually unless you want terrible performance. Let's be honest, nobody uses std::vector with manual pre-allocation even half of the time. You can easily create a wrapper around std::vector with your preferences for initial buffer size and expansion rate. Then optimize it later in one place where the backend can easily be swapped or reimplemented.

If you start with your own arena allocator, tools like Valgrind or your IDE's debugger won't be able to help you, because you're technically never out of bound when the system only sees a big chunk of shared memory. You can however implement your own pointer template classes with compiled memory protection that's on in debug mode and off in release. These can detect bound errors even inside of a bigger allocation.
https://github.com/Dawoodoz/DFPSR/blob/master/Source/DFPSR/base/SafePointer.h

Using the Variable Length Array extension may be controversial for the potential stack overflow risk of not knowing the size at run-time, but a fixed size array containing the same elements would always be equal or bigger in practice even if it's bounded by a constant in theory. Avoiding VLA using fixed size arrays is like making a static array at the size of the world's population when making a database for the members of a niche computer club. VLA is very fast, and safe when used with common sense. Everything on the stack has the risk of causing stack overflows, so just keep it compact with small integer types.

Using short lived heap allocations is a big no when your code is called 2,000,000,000 times per second. Always consider if your algorithm can avoid storing temporary arrays by using another scan order or inlined callbacks.

I'm very grateful about your answers, they really help a lot.

you can make your own code generators or virtual machines and hide it in a fancy IDE or game maker.

I don't really like this way of doing things because it comes to a huge problem when you need to add a specific functionality just to your game, but all code of engine itself is hidded behind some kind of scripting language. I think engine code have to be opened to game developer so he can use all of its functionality, even if it is unsafe. Scripting language may exist in way that UE4 has its blueprints - most of time it's used by level designers to add interaction to game levels, but all game code itself(game rules, custom assets, etc) is written in engine native language.

I like the other way you said this can be done, I think it's really cool when game developer can include some other 3d party libraries inside existing engine with just few lines of code. I'll try to create as much functionality as possible inside engine itself, but who knows what other people need in their projects.

I'll also try your other advices, especially the way you can decrease compile time. Also about std::vector, this is one reason why I don't like STL, because it's very multipurposal and not optimized for time consistent applications that games are. Memory leaks don't have to be a problem at all as I am going to implement reflection based GC system(something like UE4 does). Allocation from heap will mostly be performed during level loading or performed in another thread(not game logic one), all other allocations will be static or made in fast one-frame allocator.

In AAA games, they are allocating a big block of memory at start as you already mentioned, which is then managed by some kind of custom heap scheduler. The one and only reason to manage allocations by ourselves instead of let the OS do all the stuff is fragmentation. Allocating and deallocating different sized bunches of heap memory is causing the virtual tables to fragmentize and increase in size if there is more memory requested than free regions exist between allocated blocks.

There are two major techniques to prevent this while both operate on large heap blocks:

• Allocations happen in different sized ‘buckets’ which belong to their own heap manager. If you allocate certain amount of memory, it will be stored in one of those buckets where buckets have a minimum and maximum range they manage. One also could say they allocate in chunks where chunks are different in size from bucket to bucket. This makes it easier to fill the gaps between allocated blocks by wasting some bytes if the requested memory block is smaller than the maximum the bucket can handle
• Garbage Collection is the technique used in modern languages like Java or C# and requires to manage memory in pages. If objects get released (disposed), the GC is told to mark the block as not needed anymore and on the next run, it will be removed from heap. The GC also reranges memory blocks to close gaps between allocations and so it is difficult if you need fixed memory locations or store/ pass plain pointers to 3rd party API calls. They can be reranged at any time the GC decides to and in most languages you have to either pin the memory (which is a huge impact in how the GC works) or handle access violations

I personally am not as strict as you with the no-CRT policy, however, I avoid using the STL too but still need to access malloc/ free in the background (and also offer it to my users as low.level API) to access OS heap at all. Instead I have a strict allocator policy whenever a class or function needs access to memory, I force the user to pass an allocator or have to register a main allocator as default during startup. It not just helps to keep track of the memory but also saved my ass a couple of times from memory leaks while developing new features.

I wrote my own STL like container classes

• Array, inspired by C# array. Array is a flexible sized array (as opposite to the static sized array in C++) which can resize at every time by calling the function. It makes liefe easier because it is the base ‘class’ of every other container I have. I know about the penalities that come from inheritance but except for an additional pointer (to the vtable), my class has a quite small memory footprint of just 2 * sizeof(void) + 4 which is either 12 (x86) or 20 bytes (x64). I believe this is a low price for offering more usability and safety at all
• Vector, insipred by C# List<T>. Based on Array and expanding the class for another 4 byte integer value to keep track of the amount of items currently stored. Vector works like a C# List, you can add/ remove and insert/remove at with some memory reordering
• HashSet is an performance optimized hash based storage that is also based on Array. I'm using fast lookup algorythm on a linear block of memory instead of maintaining hash buckets and grow in POT steps
• HashMap, based on HashSet and a key-value-pair data struct
• Stack/ Queue also based on Array

The most benefit of doing this is that all data container also force allocators to be used and they all can be passed into functions that require just Array as an argument.

How to setup your engine design is highly depending on your desired usecase as already mentioned. Do you expect it to be closed source like Unity, where you are able to create content during a monolythic editor application and player, or do you plan to work all open source and have your user compile the source along with it's game code?

I designed my whole project to be the later one. My engine is compiled together with the game code into a single executeable to be run, it can also be compiled with the game code into a library. The editor is a C# application that binds to the C++ engine and game code via p/invoke and offers a clean interface to the udnerlaying system.

As everything is open source and can be modified, the user has full control of the game systems and editor capabilities. I don't expect non-programmers to use my system, a designer should design, an artist should do art and a programmer should maintain the game and engine, this is my philosophy and I absolutely hate it if non-programmers think they can just Google, Copy/ Paste some code pieces from the net and expect to get a well working game

In AAA games, they are allocating a big block of memory at start

I'm doing something like this. At startup, my allocator requires some big block of memory from platform layer, but during runtime it still can require more memory or free it. I don't know how can I predict memory usage for game on different platforms with different game situations(especially if it is some kind of MMORPG or open-world singleplayer sandbox, where game events are unpredictable at all). Now, my allocator gets 2 GB of memory during startup via VirtualAlloc & uses it, but it can get more memory in chunks of smaller size(128 MB now, if I remember correctly).

Allocations happen in different sized ‘buckets’ which belong to their own heap manager. If you allocate certain amount of memory, it will be stored in one of those buckets where buckets have a minimum and maximum range they manage. One also could say they allocate in chunks where chunks are different in size from bucket to bucket. This makes it easier to fill the gaps between allocated blocks by wasting some bytes if the requested memory block is smaller than the maximum the bucket can handle.

It looks like a good idea. But do I have to make allocations of same size inside one bucket or they can be sized from size A to size B. I think I'll have 3 buckets for now: one for allocating object, from 1 byte to 1 KB, second for data sets(1 KB to 512 MB) and another for assets(512 KB up to whatever). But do they have to be fixed-size allocations or I can do them with different sizes & storing this size in some kind of allocation header?

I'm planing to make my engine completely open-source, but I want that user will be able to download prebuilt binaries of engine with editor, and compile engine himself only for public monolithic builds.

Getting 2GB at start sounds like guessing, I instead recommend to allocate certain pages (calculate in pages, not in GB) and if you ran out of pages for an allocator, acquire the next couple ones and release them if you are done.

Buckets are usually sized to fit in certain range, for example small ones like several 100 bytes, some kb, medium sized ones from several kb up to 1mb and larger ones. It fdepends on your game, if you have a lot of small objects just use more buckets for smaller size

Don't know if this helps but you have low level engines, as high level usely people only talk about high level : https://en.wikipedia.org/wiki/List_of_3D_graphics_libraries