Hey,

I'm currently starting next iteration on my engine project and have some points I'm completely fine with and some other points and/or code parts that need refactoring so this is a refactoring step before starting to add new features. As I want my code to be modular to have features optional installed for certain projects while others have to stay out of sight, I designed a framework that starting from a core component or module, spreads features to several project files that are merged together to a single project solution (in Visual Studio) by our tooling.

This works great for some parts of the code, naming the Crypto or Input module for example but other parts seem to be at the wrong place and need to be moved. Some features are in the core component that may belong into an own module while I feel uncomfortable splitting those parts and determine what stays in core and what should get it's own module. An example is Math stuff. When using the framework to write a game (engine), I need access to algebra like Vector, Quaternion and Matrix objects but when writing some kind of match-making server, I wouldn't need it so put it into an own module with own directory, build script and package description or just stay in core and take the size and ammount of files as a treat in this case?

What about naimng? When cleaning the folder structure I want to collect some files together that stay seperated currently. This files are foir example basic type definitions, utility macros and parts of my Reflection/RTTI/Meta system (which is intended to get ipartially t's own module as well because I just need it for editor code currently but supports conditional building to some kind of C# like attributes also).

I already looked at several projects and they seem to don't care that much about that but growing the code means also grow breaking changes when refactoring in the future. So what are your suggestions/ oppinions to this topic? Do I overcomplicate things and overengeneer modularity or could it even be more modular? Where is the line between usefull and chaotic?

Thanks in advance!

Of course, rewriting is an opportunity to learn from experience and make things better in many ways.  Just be aware of second system syndrome.  Don't worry too much though, your third attempt is usually that much better again.

Do you really have a "core component"? Module dependencies form a direct acyclic graph; there can be any number of high level consumers of many dependencies, such as the different engine variants of each game (using almost everything) or the mentioned matchmaking service (probably using network server and player database modules).

On a more technical level, you should try to split currently large and incoherent header files into more manageable and faster to compile fine-grained ones, but it doesn't mean you should split everything else: related headers and the respective implementations can reside in the same VS project, the engine can be consolidated into a few static link libraries (probably only one), and you probably need only one source control repository. Only independent projects should be completely separated. 

On 9/18/2018 at 10:54 AM, Shaarigan said:

and/or code parts that need refactoring so this is a refactoring step before starting to add new features

It is root of problems. SCRUM and other pseudo-methodologies do not work with complexive projects. Only metodology for complexive mathematicaly related projects such as game engine is waterfall. Just separate task to subtask and make each component complitely and universally before start another one. By this way yo will never need a refactor already made components. Really any iterational development have a exponential growth of complexity with linear growth of functionality due to refactoring of temporary-made components. Canonical development strategy mathematically proven give a exponential growth of functionality with linear growth of complexity due to universal realisation and reusing of each subcomponents. 

On 9/18/2018 at 8:54 AM, Shaarigan said:

An example is Math stuff. When using the framework to write a game (engine), I need access to algebra like Vector, Quaternion and Matrix objects but when writing some kind of match-making server, I wouldn't need it so put it into an own module with own directory, build script and package description or just stay in core and take the size and ammount of files as a treat in this case?

How you separate your modules can be argued different ways, however you should make a distinction between LIBRARY functionality and modules specific to your engine / app that actually contain data. Things like Vector, Quaternion, Matrix, templates all should go in libraries (static or just pull in the source as a dependency) that you can bring in any where you need them. This is like the way you might use e.g. STL.

I personally prefer to have modules separated by projects. However in C++ managing dependencies betwen projects can be a nightmare. Monolithic seems to be more preferable in C++ due to lacking of necessary tools I guess.

11 minutes ago, mr_tawan said:

However in C++ managing dependencies betwen projects can be a nightmare

Just create folder called include and place any shared headers to it folder. For code that not just a inlined templates and require to add a dependencies make folder libs, a static libs project that have all depndencies compiled in and specify it on #pragma comment (lib, "") on related header by concept one header=one lib .  Add all libs to solution in case you need to rebuild libs later. Add a include and lib path to projects settings and output targets of libs projects.  And no any nightmare any more. Just include required header and enjoy.

2 hours ago, LorenzoGatti said:

Do you really have a "core component"?

Yes, the core component contains anything that is used by other components that are build on top of it. Crypto module for example uses streams and long integer math from Core, Input uses the event implementation from Core, Network uses streams, threading and other stuff from Core ... and so on.

2 hours ago, Fulcrum.013 said:

It is root of problems. SCRUM and other pseudo-methodologies do not work with complexive projects.

I don't think so! Whatever methodology your opinion is my project fits in, I worked that way for long time now, thinking about several features and refactoring while the project grows. The clue of refactoring is that you discover treats while developing and using the framework that needs to be changed, then you change them.

Not doing so will result either in a year or two of heavy planning phase or you'll end up in a situation where a specific use-case dosen't fit into your project. You will need to refactor then.

1 hour ago, lawnjelly said:

Things like Vector, Quaternion, Matrix, templates all should go in libraries (static or just pull in the source as a dependency) that you can bring in any where you need them.

A module is similar to a library. It is defined by it's topical use, has it's own sub-directory, project file and build step. This way it is ensured that there won't be any circular dependencies between modules so that anything stays inside a strict dependency graph starting at Core as the root module node and spreading along anything else.

I already have anything in static libs also.

1 hour ago, Fulcrum.013 said:

Just create folder called include and place any shared headers to it folder. For code that not just a inlined templates and require to add a dependencies make folder libs, a static libs project that have all depndencies compiled in and specify it on #pragma comment (lib, "") on related header by concept one header=one lib .  Add all libs to solution in case you need to rebuild libs later. Add a include and lib path to projects settings and output targets of libs projects.  And no any nightmare any more. Just include required header and enjoy.

The reason why Cpp prohects look that crappy is putting anything in a seperate include folder. I don't know how often I worked on projects that have had those strict structure of holding header files in the one path and implementation in the other. This is just one thing, a time consuming missconception when getting into the code.

I keep away from those constelations instead keeping header and implementation file near each other and instead make a few more topical sub-folders in my projects. Crawling through a single folder of hundrets of files that is anightmare!

#pragma comment (lib, ...)

Is another bad design choice! Working on multiple platforms or just using another compiler will crash your whole code base. Instead I manage anything of these by small custom build script files you just add the lib reference to and thats it, also anything I build is placed into the same output folder prefixed by architecture

49 minutes ago, Shaarigan said:

Not doing so will result either in a year or two of heavy planning phase or you'll end up in a situation where a specific use-case dosen't fit into your project.

It fit anycase due to universality of components. For new behaviors/subtasks just new components added that very often use existing componets under hood. Refactoring is just a result of not enought task field analise. Especially it applicable to mathematic related tasks.

49 minutes ago, Shaarigan said:

This way it is ensured that there won't be any circular dependencies between modules

thanks to header files circular dependencies is not a problem.

49 minutes ago, Shaarigan said:

putting anything in a seperate include folder

Separate folders for headers, libs and libs from project specific code folder allow to have multiple implementations for each header and have common libs easily accesible for different projects . For example target specific implementations usualy done by changing libs folder in target settings. Also you can specify as many as you wan additional include and libs folders.

49 minutes ago, Shaarigan said:

Is another bad design choice! Working on multiple platforms or just using another compiler will crash your whole code base

MSVC uses __clang-based compiler. It compile thru LLVM code that uses sparse native codegenerators. It mean that it no need to change compiler for changing a target. It only require to change adapter libs that hide differences in OS api. Also compilers have to ignore unknown pragmas. So  ever building on compiler than has no support of it pragma only require to add libs to project manually.

49 minutes ago, Shaarigan said:

nstead I manage anything of these by small custom build script files you just add the lib reference to and thats it, also anything I build is placed into the same output folder prefixed by architecture

And have a chaos in folders like that you already unable to manage and asking how to avoid it?

52 minutes ago, Shaarigan said:

is that you discover treats while developing

It newer will work for mathemathic realted tasks. You have to find self-consistent set  of operations under your primitives before any coding. Othervice it just will never work. But when it work once it will work forever. Ever in case you need enlarge something you have again find extended  self-consistent set and it never require refactoring.