So, in real life, incoming dot normal at the silhouette is always 0.  With smooth shaded meshes, it never is, not naturally, not outside of contrived situations.  (Not with flat shaded meshes either, I guess.)

And incoming dot normal is one of the bedrocks of CG.  Probably the equal of 4x4 matrix multiplication.  Problems with silhouette normals show up in Fresnel, in diffuse lighting, in environment mapping....  everywhere.  But I can't really find anybody talking about it.  (Maybe I'm not Googling the right terms.)

Obviously, the problem decreases as poly count goes up, eventually reaching a point where it's dwarfed by other silhouette problems (like translucency or micro-occlusion) that CG doesn't handle well either.  But, if I'm reasoning correctly, normal maps don't improve the problem-- they're as likely to exacerbate it as improve it, and the exacerbations are, aesthetically speaking, probably worse than the improvements are better.

I've tried playing with crude fixes-- basically, rotating normals toward incoming by a percentage, or of course clamping incoming dot normal (like we all have to do) to prevent it from bending behind the mesh.  Nothing I've tried looks good.  I suppose the best option might be to rotate normals to perpendicular to incoming at the silhouette and then interpolate to the nearest inflection point  of something like screen space depth to preserve curvature, but the math for how to do that is beyond me, and I'm not sure it would look any better.  Or maybe, instead, somehow, adjust the drawn silhouette to match the silhouette defined by incoming dot normal?  Not even sure how that would work, not if the normal was pointing away from incoming.

I don't know-- is this a solvable problem?  Has anyone tried other stuff and given up, pursued anything that was promising but too expensive, anything like that?  Are there any papers I'm missing?  It's really surprising to me that I can't find anyone else talking about this.

(Apologies if I chose the wrong subforum for this.  I considered art forums, but I felt that people frequenting the programming forums would have more to say on the subject.)

Yeah on highly tesellated meshes it's usually not too noticeable, but on low poly meshes with realistic lighting models, it can produce horrible artefacts... The most common "solution" I've seen is to simply bias the N dot L / N dot V calculations so that, say -0.1 becomes 0.0, shifting the horizon slightly further ...beyond the horizon?

Another solution is to blend from the smooth normal to the per-triangle flat normal at grazing angles.

Some recent related work (not the geometry issue, but same issue from normal maps):

https://blogs.unity3d.com/2017/10/02/microfacet-based-normal-mapping-for-robust-monte-carlo-path-tracing/

Also, yes, this seems like it should be a fundamental and well known problem in computer graphics, but I'm not sure what keywords to use when searching for solutions either ?

I guess under systems like REYES, it wasn't a problem because every triangle was flat shaded (and meshes tesellated enough that you couldn't notice)...

10 hours ago, Hodgman said:

Another solution is to blend from the smooth normal to the per-triangle flat normal at grazing angles. 

Wow, that works a lot better than I expected.  Never would have thought of that, thanks!

Checking out the paper right now.  Video looked interesting.