This will be what it will look like, but with virtual monitor with realtime texture:

The real problem, which game engine can get MAME window and convert it i

MAME is open source, so you could simply look up its source code and find where its drawing happens.

You can browse this source code here:

http://www.mamedev.org/source/

Still, you'll soon realize it was built on a low level, and it won't be as simple as hooking up a global at initialization time. You'll have to go deeper than that, unfortunately.

You could also take a look at SDLMAME (http://rbelmont.mameworld.info/?page_id=163). I haven't looked up its code, don't really know how much functionality it has, but I guess it would be more accessible than MAME itself...

The MAME source linked by @dejaime looks like it supports D3D 8/9, DirectDraw or GDI for drawing (on Windows OS). I don't know which and when etc.. but there are methods for capturing each of these if they are used.

Alternatively, a custom build of MAME that provides an appropriate extension point (if it doesn't already exist) for grabbing the render output. MAME does have support for recording video of gameplay or taking screenshots - you might be able to leverage this somehow.