I am a bit confused of the difference between transparency/alpha blending and alpha-to-coverage, since Frank D. Luna put both techniques in one section of 3D Game Programming with DirectX 11, letting it appear that the latter technique is superior. I have some questions:

I guess the superiorness is only related to the smoothing of blocky edges and not to the handling of transparency?

Since the alpha in alpha-to-coverage technique is used as coverage instead of the coverage determined at the polygon level:

• should opaque geometry be rendered without alpha-to-coverage, since we will otherwise have always 100% coverage (killing the purpose of MSAA in the first place)?
• do you loose quality for transparent geometry (which have a texture with an alpha channel), since the coverage determined at the polygon level is not used? (as opposed to MSAA with alpha blending)

Transparency/alpha blending requires sorting and rendering back-to-front.

• Is this the case for alpha-to-coverage as well?
• Suppose the alpha of a fragment is 50%, what happens with the source (fragment) and destination color? Is there still some kind of blending with the background or is the color intensity reduced by half?

Alpha to coverage shouldn't be used for opaque or alpha-blended objects; it's useful for alpha-tested objects. Normally alpha-tested objects gain no benefit whatsoever from MSAA, because the pixel shader runs at pixel-frequency (not sample-frequency) so either the pixel is rejected (no samples written) or the pixel is accepted (all samples written if it's a polygon interior / some samples written if it's a polygon edge).

A2C allows those pixels that are in the interior of a polygon but the edge of an alpha-cutout to only write to a fraction of the samples, which approximates the same effect as a real MSAA polygon edge (which would require super high polygon meshes instead of alpha mask textures), or the same effect as if you'd used an alpha-blended edge (which would require back-to-front sorting).

44 minutes ago, Hodgman said:

but the edge of an alpha-cutout to only write to a fraction of the samples

And how is determined which of the subpixels will be covered and which will not be covered in case of an 0 < alpha < 1?

I also assume that the normal coverage mask is not used with alpha-to-coverage. So when using alpha-to-coverage you need to chose between AA and transparency for alpha-tested objects (e.g. texture with alpha channel that is opaque at its edges)?

The regular MSAA sample mask from the rasterizer is AND'ed with the A2C sample mask. The A2C sample mask is created using some kind of implementation defined algorithm that quantizes the alpha and enables a corresponding portion of the samples.
e.g. for 4x MSAA, maybe 87-100% uses 1111, 62-87% uses 0111, 37-62% uses 0011, 12-37% uses 0001 and 0-12% uses 0000.

On very modern GPU's, you can actually compute a sample coverage mask yourself in your pixel shader instead of using "fixed function" A2C modes to do it for you.

14 hours ago, Hodgman said:

The regular MSAA sample mask from the rasterizer is AND'ed with the A2C sample mask.

Ok, thanks for the explanation and clarification.