C++ compile times are necessarily longer than they ought to be (longer than comparably complex code in more modern languages by orders of magnitude). This is unfortunate, but there are steps you can take to reduce compile times
Much of those costs are also the source of the strengths. I don't think it is "longer than they ought to be". Yes, it takes more time, but that is a cost paid for an actual benefit.
The performance killers are well documented. Opening up every #include is time consuming, and features like template expansion cause compilation time to grow rapidly. However, the benefits from those compilation costs can be amazing.
One of the strongest features of the language is that the compiler digs deeply into every function and searches for things to elide and eliminate, nested calls to inline, code to relocate and merge, and evaluate an enormous number of potential substitutions in order to save a few nanoseconds at runtime. Trying to do the same thing with more modern languages, the deeply-nested call trees that ultimately resolve to a single data read, the logic that in some builds vanishes completely, these are opposite of the modern language functionality. It is possible to get some of that benefit with JIT compilation and hotspot analysis done at run time, but that moves the cost to a different time that is often unacceptable in games. Sure it is fine if one of the load-balanced business servers in a server rack slows down for a moment for a hotspot optimization, not so much if the game stutters.
The compilation model used by C and C++ and several other older languages include features that are completely at odds with features in more modern languages. You cannot perform the heavy optimizations, the heavy stripping of dead code, extracting logic from common classes or from deeply-inlined calls, the complete removal of unused logic and functions, while at the same pulling in features like complete reflection of classes.
So yes, the compilation times can be much faster in modern languages. For fast iteration that is a good thing. But since games are still pushing computers to their limits, there are many times when those extra compile times provide major benefits that cannot be recouped in modern languages.
It is a tradeoff. Use the right tools.
The same for switching from ue4 to unity. They are similar tools, but they have differences. If one is a better fit for the project at hand, use that one.