5 senior developers for 3 years will not get anywhere near a "AAA capable game engine."

A game ENGINE is something a lot more complex than just the bit that runs the game. Art production for AAA games is a much more expensive (and less reusable) process than the programming itself, so most of the effort for AAA development goes into custom art tools to improve the art production pipeline. It's not an engine if it doesn't have high quality scene builders, high quality physics builders, high quality navigation visualization, high quality collision debugging, high quality Maya plug-ins, high quality cut scene editors, high quality hair and cloth visualizers, high quality particle effect editors, and so on. (Add procedural tools for forests, ground cover, terrain, cities, skies, time of day, ....)

And, here's a poorly kept secret: The big engines today are not built from scratch! They integrate other middleware for anything from physical simuation to particle effects to open world forest generation. They hire people who have spent lots of time developing the skills and knowledge in some small area (micro-reflection based specular? atmospheric diffraction based skylights? dual-quaternion based character animation?) and have them either bring their code, or re-implement what they already did once, into the engine. They not only have worked on their engines for twenty years, and now have hundreds of developers, but they draw on all of the available open source, commercial source, research papers, and community contributions available in the world.

With a specific art style chosen, and a specific environment kind designed, you may be able to have 5 senior developers assist a AAA team in a linear RPG, on top of existing technology. That sounds reasonable. Stick with UE4, pay them the 5% (or whatever you can negotiat) when you're done and shipping, and make sure to hire people with good work ethic, good experience, and not too many personal axes to grind. It's well worth it!