This is quite "ancient stuff" you are talking about. The NeoGeo console is older than the SNES and Genesis, being released in 1987 in Japan, and the Arcade board that spawned it might be older still.
True, Games were developed for 17 years for it, still, back in these days I guess nobody really cared about engines, this was the time when consoles were still programmed in Assembler (though the SNES at least seemed to accept higher level languages, or so I read).
And seeing how the graphics tech of most SNK titles was really "old school" (as Eck says, all sprite based), and SNK just had incredibly talented artists and big budgets (those games were expensive as hell in the days, seeing how they were mostly sold to arcades), I don't even think there was a real need for an engine yet.
Your best be to finding out would be to get ahold of the source of an SNK game. Sadly I do not think SNK ever released any of their games to the public domain, and seeing how they still try to milk 20 year old games for cash (the prices of the games for the PSP NeoGeo Station are ridicolous... I mean I gladly pay 30-50 bucks for an old retro cartridge for SNES. But 10-20 bucks for a digital emulation? Are they out of their mind?), I don't think they ever will. Until SNK kick the bucket (and they do quite fine it seems), you will not see their sources being released to the public.
But really, as far as I can tell, the games are pretty simple technically. For the moves I guess Ecks solution sounds like it would do the trick, for the graphics just work the same way you would work for any recent 2D game (either write from scratch, or use a 2D engine... there is even a good fighting game template / engine available for the unity engine, though it is not free)...
For the AI you need to get a little bit more inventive. That is, to make it not "too smart". It has a BIG tacitcal advantage (perfect knowledge of enemies next move 1ms after player triggers it, and perfect execution of any special move), so you need to develop the ability to make mistakes, as well as add the ability to read the players next move before he executes (as some moves need to be countered before the player inputs it, very quick moves for example). The last point thought is something any good fighting game player might give you input on.