Some questions about (world-space) normal encoding/decoding for GBuffers (lighting and post-processing):

All spherical encoding samples, I found so far use atan2 to compute the arctangent of phi in the [-pi, pi) range. How does this work for a normal of (0,0,1) since both the mathematical and HLSL atan2 are undefined for (x=0, y=0)?

Can sphere-map encoding (decoding) directly operate on world-space normals or should I first convert to (from) view-space?

Does one in practice use and get away with a DXGI_FORMAT_R8G8_UNORM or rather stick with a DXGI_FORMAT_R16G16_UNORM for accuracy (for both spherical and sphere-map parameterizations)?

The typical sphere-map transforms that I've used only require sqrt, not atan, e.g. https://aras-p.info/texts/CompactNormalStorage.html#method04spheremap

Spheremap or angle-based methods tend to have very obvious precision biasing though -- e.g. some latitudes look great while others look awful... It works fine in world-space, but I guess if you know where the good/bad parts are, you could use view-space normals and try to position the bulk of them in a good part of the transform...

At 8_8_8, the only good format I found was Crytek's BFN (page 38), which is great, but requires a LUT fetch during encoding.

The best format that I found for 8_8 was octohedral encoding (1, 2), but it still shows major quantization artifacts for mirror-like surfaces -- roughly equal in quality to naive/unpacked 8_8_8 x/y/z SNORM storage of normals. It's fairly easy to reason about -- the 8_8 domain gives 65536 possible encodings, octohedral splits the sphere into two hemispheres, and each hemisphere into four triangles, at which point there's only 8k possible normals in that triangle, roughly equal to a 90x90 grid. At 16_16, octohedral is perfect though (rougly a 23k x 23k grid per triangle ). If you're doing manual G-buffer packing within 32-bit integer channels, you could use something in the middle, such as 12_12 octohedral

9 minutes ago, Hodgman said:

The typical sphere-map transforms that I've used only require sqrt, not atan, e.g. https://aras-p.info/texts/CompactNormalStorage.html#method04spheremap

Sorry for the confusion, but I am talking about two different encodings in my questions: spherical and spheremap (method #3 and #4 :P).

1 hour ago, matt77hias said:

atan2 to compute the arctangent of phi in the [-pi, pi) range. How does this work for a normal of (0,0,1) since both the mathematical and HLSL atan2 are undefined for (x=0, y=0)?

Atan2(0,0) is defined in C/C++ and its value is 0. However I don't know if that will behave that you would expect it...

8 minutes ago, Silence@SiD said:

Atan2(0,0) is defined in C/C++ and its value is 0. However I don't know if that will behave that you would expect it...

If HLSL uses the same convention then that would be ideal:

float2 EncodeNormal_Spherical(float3 n) { // (0.0f, 0.0f, 1.0f)
    const float phi       = atan2(n.y, n.x); // 0.0f hopefully :D
    const float cos_theta = n.z; // 1.0f
    return SNORMtoUNORM(float2(phi * g_inv_pi, cos_theta)); // SNORMtoUNORM (0.0f, 1.0f)
}


float3 DecodeNormal_Spherical(float2 n_unorm) {
    const float2 n_snorm = UNORMtoSNORM(n_unorm); // (0.0f, 1.0f)

    const float phi = n_snorm.x * g_pi; // 0.0f
    float sin_phi; // 0.0f
    float cos_phi; // 1.0f
    sincos(phi, sin_phi, cos_phi);

    const float cos_theta = n_snorm.y; // 1.0f
    const float sin_theta = CosToSin(cos_theta); // 0.0f

    return float3(cos_phi * sin_theta, // 0.0f
                  sin_phi * sin_theta, // 0.0f
                  cos_theta); // 1.0f
}

To be sure, I asked the compiler team: https://github.com/Microsoft/DirectXShaderCompiler/issues/1245

And unfortunately, there is no guarantee. A check needs to be added, making spherical encoding quite expensive . (Though, I am really happy with the Octahedron )

Note that if you store tsnms as (BC5_)_UNORM 0 won't be represented exactly. So the encoding only becomes undefined, if you explicitly pass a (0,0,1) normal.