Hey guys,

Are lightmaps still the best way to handle static diffuse irradiance, or is SH used for both diffuse and specular irradiance now?

Also, do any modern games use direct light in lightmaps, or are all direct lighting handled by shadow maps now?

Finally, how is SH usually baked?

Thanks!

4 hours ago, KarimIO said:

Are lightmaps still the best way to handle static diffuse irradiance, or is SH used for both diffuse and specular irradiance now?

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

For specular you are typically out of memory to store high detailed enviroment at every lightmap texel, that's why a sparse set of high resolution enviroment probes is common for this. (A 3 band SH has less detail than a cubemap with 6 * 2*2 texels, a 2 band SH is similar to 6*1 texel cubemap - not enough for sharp reflections)

In contrast, sampling diffuse indirect lighting from a sparse set of enviroment probes would be much worse than using lightmaps. Lightmaps have high resolution in the surface so they can store accurate local details, like shadows and darkening in corners. High resolution directional information is not necessary.

But there are many games that use either sparse lobes or dense lightmap approach for everything, others use sparse 3D texture bricks instead 2D lightmaps, many do not use precomputed lighting at all.

4 hours ago, KarimIO said:

Finally, how is SH usually baked?

Typically you generate n rays evenly distributed to any direction, for each ray calculate the incoming light (e.g. by path tracing), and sum up. Finally you normalize SH and done. 

So you could do something like contribute each texel of a cubemap to a SH as well, just calculate direction to each texel and a weight (texels at corners are more dense than at a face center, the weight should compensate this effect)

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

6 hours ago, JoeJ said:

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

But then this isn't a lightmap, this is just SH projected onto surfaces

6 hours ago, JoeJ said:

For specular you are typically out of memory to store high detailed enviroment at every lightmap texel, that's why a sparse set of high resolution enviroment probes is common for this. (A 3 band SH has less detail than a cubemap with 6 * 2*2 texels, a 2 band SH is similar to 6*1 texel cubemap - not enough for sharp reflections)

Yeah I was specifically talking about irradiance, or indirect light.

6 hours ago, JoeJ said:

In contrast, sampling diffuse indirect lighting from a sparse set of enviroment probes would be much worse than using lightmaps. Lightmaps have high resolution in the surface so they can store accurate local details, like shadows and darkening in corners. High resolution directional information is not necessary.

Okay so this kinda does answer my major part of the question.

My current plan is to use:

CSM + Regular Shadow Maps for direct light shadows, normal-mapped lightmap (Valve's technique, but with only two targets: ambient color and direction) for indirect diffuse for static geometry, SH for indirect specular on all geometry + indirect diffuse on dynamic geometry, "The Last Of Us"'s occluders for indirect dynamic shadows, and parallax cubemaps, planar reflections, and SSAO for direct specular (planar reflections being mostly used for water).

I just want to verify that these techniques are reasonable and won't cause issues.

6 hours ago, JoeJ said:

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

I saw this, but from what I understand, spherical gaussians are just broken down SH (where directions are not intrinsically calculated, rather they're provided). They don't seem to be much of a solution.

1 hour ago, KarimIOAH said:

7 hours ago, JoeJ said:

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

But then this isn't a lightmap, this is just SH projected onto surfaces

The SH data already is generated from the surface, so no need to projection. Your original question was lightmap or SH, but you can store SHs in the lightmap. (which would look better than just ambient with primary light direction, but if that's worth the memory depends on the game.)

1 hour ago, KarimIOAH said:

My current plan is to use:

CSM + Regular Shadow Maps for direct light shadows, normal-mapped lightmap (Valve's technique, but with only two targets: ambient color and direction) for indirect diffuse for static geometry, SH for indirect specular on all geometry + indirect diffuse on dynamic geometry, "The Last Of Us"'s occluders for indirect dynamic shadows, and parallax cubemaps, planar reflections, and SSAO for direct specular (planar reflections being mostly used for water).

I just want to verify that these techniques are reasonable and won't cause issues.

That's a large collection of hacks... of course it will cause issues i guess, 

But it looks pretty state of the art to me so see how it goes... (I do not understand the distinction between SH indirect specular and cubemaps for reflections - you might want to explain in more detail.)

2 hours ago, KarimIOAH said:

 

8 hours ago, JoeJ said:

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

I saw this, but from what I understand, spherical gaussians are just broken down SH (where directions are not intrinsically calculated, rather they're provided). They don't seem to be much of a solution.

I agree it does not matter that much but i see advantages of SG vs. SH. What matters is the idea to store the entire enviroment at dense locations on the surface to achieve highest quality, and to be able to calculate full BRDF from that data without the need to read from some cubemaps, do some screenspace raytracing, do some other traces in volume data, and whatever other hacks we come up with. 

1 hour ago, JoeJ said:

The SH data already is generated from the surface, so no need to projection. Your original question was lightmap or SH, but you can store SHs in the lightmap. (which would look better than just ambient with primary light direction, but if that's worth the memory depends on the game.)

Makes sense. The only annoying bit is that I think you'll store half the information aiming into the texel rather than away from it, which seems like a waste.

1 hour ago, JoeJ said:

That's a large collection of hacks... of course it will cause issues i guess, 

Well all of realtime CGI is hacks But what do you mean more specifically? Do you see any problems with it?

1 hour ago, JoeJ said:

(I do not understand the distinction between SH indirect specular and cubemaps for reflections - you might want to explain in more detail.)

Well, you've got both diffuse and specular, each based off direct and indirect light. For specular, I want to handle direct with SSAO + Cubemaps, and ambient with SH. Because low-res cubemaps aren't very good for ambient.

20 minutes ago, KarimIOAH said:

Makes sense. The only annoying bit is that I think you'll store half the information aiming into the texel rather than away from it, which seems like a waste.

Yes, that's one argument for SG (you could align all its directions to the hemisphere) or tiny spheremaps (which is what i do but in realtime: 4x4 texel enviroment per lightmap texel. Even 2x2 would be better than primary light direction if the scene is colorful.)

It's also possible to modify the SH idea so it only covers the hemisphere, but i guess the win is not worth the effort to fully understand the math

31 minutes ago, KarimIOAH said:

For specular, I want to handle direct with SSAO + Cubemaps, and ambient with SH.

I'm still confused. I would define 'direct specular' as the reflection of light sources, which can not be stored in SH practically.

I would define 'ambient specular' as the reflection of the lit scene excluding light sources.

So how and why would you blend between cubemap and SH representation of the same thing? Why SH at all and not just cube mip? (Or 4 samples in higher cube mip, to fight the cubes axis-alignment.)

Maybe this indicates a flaw in your design, but maybe i just miss something (i know more of generating the data than using it.)

53 minutes ago, JoeJ said:

Yes, that's one argument for SG (you could align all its directions to the hemisphere) or tiny spheremaps (which is what i do but in realtime: 4x4 texel enviroment per lightmap texel. Even 2x2 would be better than primary light direction if the scene is colorful.)

It's also possible to modify the SH idea so it only covers the hemisphere, but i guess the win is not worth the effort to fully understand the math

I don't think you'll be saving much with SG, considering you still need to include the direction.

Anyhoo, you were right, I just got confused I'm actually working on something else so I didn't get my facts straight, but yeah, SSAO/Cubemaps for indirect specular, SH for indirect diffuse, maybe with an option for lightmaps instead. Maybe in addition, some options for dynamic indirect diffuse to be calculated, whether through voxel-GI or otherwise.

Thanks, JoeJ!

11 hours ago, KarimIOAH said:

SSAO/Cubemaps for indirect specular, SH for indirect diffuse, maybe with an option for lightmaps instead. Maybe in addition, some options for dynamic indirect diffuse to be calculated, whether through voxel-GI or otherwise.

Sounds good. I liked CryEngines dynamic voxel GI approach - IIRC it's about reflective shadowmaps using voxels for occlusion, so still expensive but voxels require only one bit. Details on their manual pages.

On the precalculated side this one is interesting (static geometry but dynamic lighting): https://users.aalto.fi/~silvena4/Projects/RTGI/index.html

I was considering only using those kinds of approaches on smaller areas, if at all (RSM, VoxelGI, and the like would be supported in volumes) because they're so expensive. But I'll check out your link, thanks

I just wanted to chime in on a few things, since I've lost too much of my time to this particular subject.

• I'm sure plenty of games still bake the direct contribution for at least some of their lights. We certainly did this for The Order, and did it again for Lone Echo. Each of our lights has flags that control whether or not the diffuse and indirect lighting is baked, so the lighting artists could choose to fully bake the light if was unimportant and/or it would only ever need to affect static geometry. We also always baked area lights, since we didn't have run-time support for area lights on either of those games. For the sun shadows we also bake the shadow term to a separate lightmap. We'll typically use this for surfaces that are past the last cascade of dynamic runtime shadows, so that they still have something to fall back on. Here's a video if you want to see what it looks like in practice: https://youtu.be/zxPuZYMIzuQ?t=5059. 
• It's common to store irradiance in a light map (or possibly a distribution of irradiance values about a hemisphere in modern games), but if you want to to compute a specular term then you need to store a radiance distribution in your lightmap. Radiance tells you "if pick a direction, how much light is coming in from that direction?" while irradiance tells you "if I have a surface with a normal oriented in this direction, what's the total amount cosine-weighted light that's hitting the surface?". You can use irradiance to reconstruct Lambertian diffuse (since the BRDF is just a constant term), but that's about it. Any more complicated BRDF's, including specular BRDF's, require that you calculate Integral(radiance * BRDF) for all directions on the hemisphere surrounding the surface your'e shading. How to do this efficiently completely depends on the basis function that you use to approximate radiance in your lightmap.
• If you want SH but only on a hemisphere, then you can check out H-basis. It's basically SH reformulated to only exist on the hemisphere surrounding the Z axis, and there's a simple conversion from SH -> H-basis. You can also project directly into H-basis if you want to. I have some shader code here for projecting and converting. You can also do a least-squares fitting on SH to give you coefficients that are optimized for the upper hemisphere. That said I'm sure you would be fine with the Last of Us approach of ambient + dominant direction (I believe they kept using that on Uncharted 4), but it's nice to know all of your options before making a decision
• You don't necessarily have to store directions for a set of SG's in a lightmap. We assume a fixed set of directions in tangent space, which saves on storage and makes the solve easier. But that's really the nicest part of SG's: you have a lot of flexibility in how you can use them, as opposed to SH which has a fixed set of orthogonal basis functions. For instance you could store a direction for one SG, and use implicit directions for the rest.