I'm a newbie in DX12 and trying to visualize Stencil buffer onto screen. I've been copied stencil plane slice into a DXGI_FORMAT_R8_UINT texture, but when HLSL samples this texture it just returns 0, and finally my render result is a black screen.

Here is the texture info from PIX:

and the shader code is simple:

Texture2D    gDiffuseMap : register(t0);


SamplerState gsamAnisotropicWrap  : register(s4);

float4 PS_PixelOverdraw(VertexOut pin) : SV_TARGET
{
    // gsamAnisotropicWrap is a static sampler
    int diffuseAlbedo = gDiffuseMap.Sample(gsamAnisotropicWrap, pin.TexC);
    float3 litcolor = diffuseAlbedo / 255.0f;
    return float4(litcolor, 1.0f);
}

when I debug pixel history at uv coordinate (0.321875, 0.5625), it just return 0 from Sample expression:

When sampling an UINT-texture format like you did, the result will be a “float”, or “float4” to be precise. The sampler will already do the conversion for you, so “Sample(…).x” is a value in the range of 0.0f to 1.0f. You convert this to an int, effectively turning it into a 0, which you then divide. Try to use just the return-value from the sampler, this should present you with the correct value.

If you want to slightly optimize the shader for the case of the texture only being one value, you can try declaring it's type as float:

Texture2D<float>    gDiffuseMap : register(t0);

This won't change ther rules I described above, and it's probably only a minor gain at best. You could also declare it as Texture2D<uint>, then it would return the value you originally expected, and you'd have to clip it back with the divide. However, Sample does not work on an explicit uint-type texture, you'd instead have to Load it's value from a pixel-index pair, instead of a texcoord… just don't bother for the case that you have, this is just for clarification.

@Juliean thanks for reply. I just tried change return type of sample to float4, but get a weird value almost zero:

but the input-texture has a red channel of ‘4’ in this location:

Hm, then there might be something different going on too. I'm not too familiar with the resource-bindings of DX12 myself, so there could be an issue there. But here, you could actually try to declare the texture as “Texture2D<uint>” and use Load instead of Sample, to see if it's an issue with the samplers type-conversion, or with the value that the shader receives.

To do that, first you declare the Texture as “Texture2D<uint>”. Then, for changing the Sample to Load, you need to pass Load an int3(), where the first two parameters are the pixel indices. You can get those by multiplying the texture-coordinate with the texture size (either pass it as a constant or use GetDimensions). The 3rd parameter is the mip-level, which is null. Then you should receive the value between 0 and 255, if the texture is correctly bound/read.