I don't even know if this would work or has ever been attempted, but one day this idea came to me. Out of the desire to see truly photo-realistic 3d models in real-time gameplay, the initial idea that occurred to me was, what if one were to capture a live actor performing movements, with video cameras around him/her. Then extract that video footage of the actor's skin surface textures and wrap it around a 3d model of a human (if this is even possible). Here's an image of what I sort of had in mind from a top down perspective.



I've thought a lot about this and can see problems, limitations and disadvantages for someone to undertake such a project. Such as, would this have to be done in a controlled environment, like a studio where lighting can be adjusted accordingly, to different lighting situations? I mean wouldn't the only real reason to do something like this be, to get the different light and shadow information on the actor when they are performing different movements, along with the skin textures? What about the possibility of 3d (or even 2d) plane gameplay? Also, I wanted to make mention that part of my idea involved emulating your real life surrounding in the game, and make it a fighter, if necessary.

Could any of this actually work? What haven't I realized? I'm quite positive there to be massive holes in this concept, that I haven't thought of. What about limitations in terms of character attire, amount of lights, cosmetic effects, backgrounds, props, etc...

But someone explained to me that currently the reason we aren't able to get photo-realistic real-time graphics on current hardware is because we can't support the cooling CPU's would require to handle such on screen rendering, or something along these lines. But he said this most likely will be possible in the next 20 years or so. I'm just trying to think of ways to cheat that time :P

This is a pretty standard technique, though it is usually used only to capture the motion for an animation. It is extremely common in CG movies (eg: Lord of the Rings's Gollum) to have the motion captured from live actors, even down to the facial expressions. However, for that they used more precise techniques than simply pointing some cameras at actors. Usually the actor wears, say, black with specific white points marked and software will recreate the location of all the points.

Simpler versions of this are used too. I remember seeing a video about motion capture for the original Prince of Persia which was basically taking a video of a guy doing the various moves and then drawing the pixel art over that.

Here's the Wikipedia article on it.

What you're thinking of is basically taking motion capture of a live actor and then placing that point data on to a 3d mesh.

It's not as easy as "extract video footage of the actor's skin surface textures". You'd have to take high res photographs of the actor at many angles and then do a lot of photoshop work to create UVmaps out of the photos... and even then you'd still need a bit of artistic skill to clean it up so things don't get all stretched out over the polygons.

But yeah, all this has been done before.

I think the closest thing that has been done so far would be the technique they used for the movie The Curious Case of Benjamin Button. They have a making of video here:

">

But that's only with a head... for a full body scan the rig would be far more expensive... I think that's why they chose to use real actors and just do face replacement... Though you could also get away with doing 3d meshes with mocap applied and put the face directly on the 3d model.

Something like this was fairly common in film during the late 1990s. For example the film Fight Club did stuff like this for the sex scene and other shots in the film...And of course the original Matrix filmed the "bullet time" effects with multiple cameras, interpolateing (basicly a morphing effect) the inbetweens to smooth out the shots.

NSFW - Fight Club effect scene useing 5 cameras:


Couldn't find the traveling shot stuff in Fight Club that were strikeingly photo real. But the process is simular to Disney's Deep Canvas stuff used in Tarzan:


Only Fight Club used multiple photographs to texturemap simple CG models instead of the digital paining stuff seen in that Deep Canvas video.

Another explanation of some of those Fight Club effects:
http://www.digitalpostproduction.com/Htm/Features/Buf/BufPage2.htm

Okay, I kind of understand the explanations. To me, the videos didn't explain how everything worked in great detail...but if they did, maybe it's just my lack of knowledge on computer graphics that have me baffled. But either way, I feel like I'm trying to put the pieces together.

But anyways, now that I know something similar to what I originally suggested has been done before...have such techniques been used for games? If not, how come? How much would a full body scan rig cost?

Thanks for the replies and video examples everyone.

Quote: Original post by Swordmaster
have such techniques been used for games?

It's been standard practice since at least xbox and PS2 on almost every 3D title and even on RTS games [smile]. Basically standard practice for about 6+ years for capturing animation.

Quote: Original post by Swordmaster
How much would a full body scan rig cost?

You can do simple ones with spandex, ping-pong balls, an HD camera and some video software on the cheap. The pro setups are probably in the hundreds of thousands of dollars. A lot of companies go to motion capture studios and you basically rent the room & gear by the hour with your stunt men. You act out your animations, their people then do their magic with the data and deliver it to you. Fun times.

-me

Quote: Original post by Swordmaster
now that I know something similar to what I originally suggested has been done before...have such techniques been used for games? If not, how come? How much would a full body scan rig cost?

A lot. Lots of games use body-scan facilities to help with character creation - I don't know how much these places charge, but we only do it if the publisher agrees to foot the bill.

These use a laser to measure the 3D shape as well as optical equipment to capture the colours.

The really high-end facilities have arrays of light sources all around the subject, which are timed to pulse in synch with the recording equipment, which in conjunction with polarising filters, allows you to seperate out the specular lighting from the diffuse lighting (which helps out your texture artists a lot).
Here's a DIY example of seperated diffuse/specular images.

Quote: Original post by Palidine

Quote: Original post by Swordmaster
have such techniques been used for games?

It's been standard practice since at least xbox and PS2 on almost every 3D title and even on RTS games [smile]. Basically standard practice for about 6+ years for capturing animation.

Quote: Original post by Swordmaster
How much would a full body scan rig cost?

You can do simple ones with spandex, ping-pong balls, an HD camera and some video software on the cheap. The pro setups are probably in the hundreds of thousands of dollars. A lot of companies go to motion capture studios and you basically rent the room & gear by the hour with your stunt men. You act out your animations, their people then do their magic with the data and deliver it to you. Fun times.

-me

Hmmm....I always envisioned the only thing that fell into motion capture was actually capturing motion and not the actual appearance of the actor. I just really hope we are talking about the same thing :)

But in any event, why then do games real-time graphics, not look as real as the "Fight Club" and "The Curious Case of Benjamin Button" videos posted in this thread, if such techniques are being used?

Quote: Original post by Swordmaster
Hmmm....I always envisioned the only thing that fell into motion capture was actually capturing motion and not the actual appearance of the actor. I just really hope we are talking about the same thing :)

Oh it's definitely just capturing motion. No point to capturing appearance since games are extremely low-resolution when compared to reality. Perhaps that's a future technology...

Quote: Original post by Swordmaster
But in any event, why then do games real-time graphics, not look as real as the "Fight Club" and "The Curious Case of Benjamin Button" videos posted in this thread, if such techniques are being used?

Movies to not have to render in real-time, games do. It takes often several minutes or longer to render a single frame of movie quality graphics. But the movies also do not capture appearance. For example here is footage of what they actually filmed & captures versus what you see on screen in pirates of the carribean:

http://www.spike.com/video/pirates-of-caribbean/2812960

Actually here's a good idea of what they do for games:

">

-me

Quote: Original post by Hodgman

Quote: Original post by Swordmaster
now that I know something similar to what I originally suggested has been done before...have such techniques been used for games? If not, how come? How much would a full body scan rig cost?

A lot. Lots of games use body-scan facilities to help with character creation - I don't know how much these places charge, but we only do it if the publisher agrees to foot the bill.

These use a laser to measure the 3D shape as well as optical equipment to capture the colours.

The really high-end facilities have arrays of light sources all around the subject, which are timed to pulse in synch with the recording equipment, which in conjunction with polarising filters, allows you to seperate out the specular lighting from the diffuse lighting (which helps out your texture artists a lot).
Here's a DIY example of seperated diffuse/specular images.

Thanks for the explanation on the scan rigs. And I checked out that link and only understood up to the part about what specular and diffuse lighting is. The rest of it honestly had me scratching my head. Could you possibly explain to me what it is that's happening within those images?