Hey thanks heaps for this. I understand what you are saying and I think it is the "switch-over" that would be the most complex task to solve. I don't have any intentions of using loads of textures etc - it would be very minimalistic (think early 1990's). As long as I can transition from space to world that would be the ticket.. which I guess would have to be done when the proximity of the sphere is such that it no longer appears to have a rounded surface?
It may be worth me having a look at Unity. Unless I can combine multiple engines, I don't think Unreal/UDK will work for me.. (perhaps I'm wrong) If its possible to quickly load up one of these engines and switch it over via some code of my own then perhaps that could work.. but I there would be lag in doing so.. unless the Unreal level loading can be done via code/script somehow instead of using its 2D level loading tiled map.
Unity IMO is always worth having a look at, as its free to try and actually build your game with anyway.... I don't exactly get why you rule out UDK/UE4 though?
a) Maybe somebody already wrote similar systems to what might be available in Unity (be aware that I am not 100% how much the systems I have heard of in Unity really apply to your use case)
b) These Engines will offer the same possibilities to come up with your own system if you want to go the DIY route.
Also, UDK is basically legacy, You should have a look at UE4 and their subscription option. In the end, you get a more up to date engine with AFAIK better licensing options.
About your game idea and coming up with complex systems: If there is one thing certain, then it is that in the 1990's, a use case like yours would have involved the player getting close to the planet, choosing to land somehow, followed by a loading screen, and then loading the the onsurface scene.
If you really want to concentrate on the retro look, don't waste your time with a orbit-to-surface transition system. I will only look out of place, and the worst of it is, it might consume more time (at least if you have to do it yourself) than the rest of the game.
It is one of these nice to have things that make a game look really good and all, but you can waste months developing on without it having a tangible impact on gameplay. Like night/day cycles in most games. Yes, its atmospheric, but let me tell you, if you go all the way, its a ridicolous figth with realtime shadow system limitations (ugh, flickering), onesided terrain polygons, and careful tweaking of light color and strength changes over time. I know, I tried it, I got it to work, I wasted months on it. Not much of a game to show after those months, but boy, were the sunsets you could watch while doing nothing gorgeous ;)
Get your game running, work with load screens and multiple scenes, and don't worry about all the next-gen technological fringe stuff. Until you get a third party tool/plugin that does all the work for you, its not worth the waste in time.
Of course, if you MUST have it (you want to have realtime Firstperson view battles in space, in orbit, in the airspace and on the ground and everywhere inbetween (so that a fighter can escape to space with smooth transitions).... yes, it is possible to do. But think long and hard if that part of your game design is worth the technological struggle you might have to gfight to get it working.
EDIT: re-reading your question, you might have asked more WHEN to set the transition between space and surface scene, rather than opting for a smooth transition system.
I don't think the polygons will be your problem. that can be dealt with easy with an LOD system, and a sphere start to look extremly smooth even with a poly count that is rather medium by todays standarts (Several thousand tris/quads should suffice... remember, some next-gen character models are multiple 10'000 polygons, and some HD vehicle models over 100'000 polygons big).
Your bigger problem will be texture resolution. AFAIK the biggest Texture resolution possible on today GPUs is 4096x4096.... which is plenty, if the planet just fills the screen in Full HD.
Go closer to the Planet model, or go UHD, and the texture might get blurry.
So either make the transition just before your ship gets so close that the player can spot the texture getting blurry, model the highest LOD with multiple textures (not very wise, this will use a LOT of VRAM) or use some clever shader to hide the blurriness of the main texture. Look up "detail textures", they are basically doing just that.