In my D3D11 project, I have an HLSL pixel shader (Shader Model 5.0) wherein I have a branch where I iterate over a bound multisampled depth texture pixel's sub-samples to find a specific (nearest depth) sample index, which I then use to sample from another bound multisampled texture. These two matched depth + color textures are the output of a previous multisampled render.

However, the output indicates that it is not taking/returning the correct sub-sample. In this case the pixel sub-samples include background (red) pixels and foreground (brown) pixels, and the shader appears to be returning red.

Doing a capture using the Graphics Debugger and inspecting the captured frame, this confirms that the output for that pixel for that draw call is indeed red. However, when I debug the pixel shader, it appears to be executing correctly. The logic and sub-sample selection is working properly, and the value being returned is the correct one (brown). Literally at "return finalColor;", finalColor is the desired brown color.

I can alter the shader to manually set the sampled color to something obvious (bright green) directly after the multi-sample Load, and this is reflected in the output as expected.

So this leads me to believe that there is some subtle issue regarding multi-sampled texture (sub)sampling that I am experiencing, which the Graphics Debugger stepping somehow avoids. To be clear, all of my sample retrievals are via Load, so there is no filtering in play.

While the Graphics Debugger stepping seems to disagree with the actual final output, the runtime output result is consistently incorrect regardless of my shader compilation settings (full release/optimization, versus full debug (no optimization/prefer branches)).

In my debug configuration I have all DX runtime + HLSL warnings as errors, full validation, and the runtime does not make a peep compiling or during execution.

Any ideas on what might be going wrong? Anyone seen anything similar or have general gotchas for multisampled texture loads?

I determined the root cause of the runtime error, which was that a preceding post-processing pass had rendered (alpha-blended) a fullscreen quad onto the MSAA render target, thus overwriting/resetting the underlying pixel data. While my initial idea was sound (and worked when disabling said post-processing pass), this led me to take a slightly different approach to my MSAA edge problem that doesn't involve needing the MSAA color sub-sample data.

However, this does raise an interesting question, which I have yet to find an answer to (or even discussed, really): What is the expected effect of doing a blend operation onto an MSAA pixel (how are the existing/resultant sub-samples affected)? It seems to me there are two possibilities:

1. When a blend must occur on a pixel, that pixel is resolved and then the blend operation is performed (and any resultant new MSAA sub-samples are generated, if it's on an edge). This is the effect which appeared to be happening for me at runtime. As an example, if we are alpha blending a red pixel onto an existing MSAA pixel that has 50% black sub-samples, and 50% white sub-samples, this is resolved into a gray pixel, the alpha blend is performed against that, and the new sub-samples for that pixel are now all pinkish gray (as this new pixel falls entirely within the primitive being rendered).
2. The blend could be performed on the individual sub-samples, preserving the original MSAA “edge” sub-samples. This would mean the sub-samples now have red+black and red+white blended sub-samples, which you can still retrieve individually. This is the effect that seemed to be happening when debugging in the Graphics Analyzer.

#1 seems the most logical and consistent outcome, as that would align with what happens when you simply overwrite a pixel (non-blended). However, the Graphics Analyzer behavior leads me to believe that it's certainly possible it may be doing #2, perhaps as a not-guaranteed or unspecified behavioral or implementation-dependent side-effect.

Does anyone have any ideas or insight into what I saw and how that might play into MSAA blending behavior?