Alright, let me try to organize some thoughts on this subject. Modularized design is one of these things that everyone tosses around as of course that must be a good idea, but very few people actually succeed at doing. As has been noted, C++ is particularly poor for modular design compared to most later language designs, but that's far from the only issue or even the major one.
Whenever code interacts, there are assumptions involved about how both sides behave. It's tempting to say that oh, B strictly depends on module A and so module A can be cheerfully reused, interchanged, etc without having to worry about B. It's also easy to say that things won't ever end up straddling the boundary, or making assumptions about the internal behavior of either piece. This is where we start to head into the land of leaky abstractions, systems that ostensibly live as independent modules and yet expose much more of their implementation than expected or intended.
Eventually at the scale of something like a game, we tend to end up with modules that are entirely co-dependent, at least at a behavioral level and sometimes at a full code level. The physics and rendering code ends up with hacks, taps, or magic variables that exist to enable behaviors and structures that simply did not modularize in any meaningful way. I've seen people argue that the entire concept of code reuse (especially in the context of OOP) is overblown and more of a fantasy than reality, though I think that's extreme.
That's all well and good at an academic level, but you'd probably like some advice about what to do about it. I'm going to give the same advice I give for many design problems: don't do anything at all. The biggest mistake developers routinely make is designing systems and interfaces for what they imagine they would like to solve instead of what they're actually solving. Focus on the exact things you need to actually produce the exact product you're making, and put just enough design effort in so that everything doesn't go to hell if you need to reshape it later. KISS and YAGNI are undervalued approaches. Take your matchmaking server that doesn't need math code. Suppose the math code goes in anyway. Who cares? What has this cost you? What problem has it created? What problem does removing it solve?
Don't get lost trying to solve problems you don't actually have. Create a module when you have need to share blocks of code smoothly across multiple products, and then you'll have the necessary context to design the module properly. There's no sense making a module out of something that only has one use case in the first place.
