Looking through my backlog of half written articles, I realized I have a couple more topics in my series about antialiasing, which I had entirely forgotten about! Oops...
Actually, though, today is a good day to finish this article. Forget your ghosts, ghouls, and goblins, for I have something Really Truly Very Scarily Horrible indeed. Yes, boys and girls, today marks the return of the ALIASING MONSTER!
A shader is really just a program that computes what color pixels should end up on the screen. Depending on the details of what this program does, it could cause aliasing, or it could use any imaginable technique (including any of the things I previously discussed) to avoid aliasing.
For instance, consider a pixel shader that implements a vector rasterization algorithm. Invoked by C# code that draws a simple square: spriteBatch.Begin(0, null, null, null, null, effect);
spriteBatch.Draw(flatWhiteDummyTexture, new Rectangle(0, 0, 128, 128), Color.White);
spriteBatch.End();
If we apply this pixel shader: float2 circleCenter = { 0.5, 0.5 };
float circleRadius = 0.4;
float spriteSize = 128;

float4 PixelShaderFunction(float4 color : COLOR0, float2 uv : TEXCOORD0) : COLOR0
{
float distanceFromCenter = length(circleCenter - uv);
float distanceFromCircle = abs(circleRadius - distanceFromCenter) * spriteSize;

float alpha = (distanceFromCircle < 0.5) ? 1 : 0;

return color * alpha;
}

technique Technique1
{
pass Pass1
{
PixelShader = compile ps_2_0 PixelShaderFunction();
}
}
We get a circle:
But it is an aliased circle. We can fix this by changing the alpha computation in our pixel shader to: float alpha = saturate(1 - distanceFromCircle);
Which gives a nicer, antialiased result:
Zoomed in so you can see the difference more clearly:
Ok, so this is a contrived case. And the way I implemented this antialiasing is specific to this particular circle algorithm. It isn't especially useful to say "depending on what they do, some shaders may cause aliasing, but there may be specialized ways you can change them to antialias their computations" :-) Is there some more general principle that be extracted here?

Vertex Shader Aliasing

Perhaps surprisingly, vertex shaders are not usually a source of aliasing problems, at least so long as your triangles remain large enough to avoid geometry aliasing.
The vertex shader is responsible for computing two things:

• Position of each vertex on the screen, which causes geometry aliasing if the resulting triangles are too small
  
• Values such as colors and texture coordinates, which are interpolated and then passed to the pixel shader
  
  Regardless of whether these interpolated values are looked up from vertex buffers or computed by whatever crazy piece of math you can imagine, it is basically impossible for them to cause aliasing. Remember that aliasing occurs when we resample a signal containing frequencies above the Nyquist threshold. We can think of color or texture coordinate channels as a waveform:
  
  
  • Frequency is determined by the distance between vertices (aka triangle size)
    
  • Changing the value of the color or texture coordinate alters the amplitude of the waveform, but not usually its frequency
    
  • Changing values can reduce frequency if several adjacent vertices share identical values, but can never increase it above the limit determined by triangle size
    
    This means that, having taken care to avoid small triangles that would cause geometry aliasing, you need not worry about aliasing elsewhere in your vertex processing.
    Note that, although choice of color or texture coordinates cannot introduce aliasing that was not previously present, it can affect how noticeable previously existing aliasing is in practice. If you had a model with nasty geometry aliasing, but colored the whole thing subtle shades of grey, the aliasing might not be too offensive. Change the colors to a rainbow of primary shades, and even though we altered the amplitude but not frequency of our color signal, the geometry aliasing will now be clear for all to see. It was there all along, but our choice of colors can make the problem more or less obvious. Regardless, such problems are best tackled by fixing the geometry aliasing at source (which means using larger triangles) rather than trying to paper over them by changing vertex color or texture coordinate mappings.
    
    Pixel Shader Aliasing
    
    The pixel shader is where things get interesting. This is a function that takes interpolated values from the vertex shader, applies an arbitrary computation, and produces an output color for a single screen pixel. The previous section explained how, as long as your triangles are not too small, the pixel shader input values will not be aliased, but it is both possible and common for pixel shader computations to introduce entirely new aliasing of their own.
    Whenever a pixel shader applies a computation to an input value it is useful to ask yourself, does this alter the frequency of the signal, or change its amplitude, or both? If the former, aliasing may ensue, but if the latter, we are safe.
    Consider these common shader operations: color = textureColor * lightColor;
    
    alpha *= 2;
    
    result = lerp(baseTexture, environmentMap, fresnelAmount);
    
    intensity = dot(normal, lightDirection);
    These are all linear computations, which means they change amplitude but not frequency, and so will not cause aliasing.
    These, on the other hand, are not linear. They have the potential to increase the frequency of the output signal above that of the input, so aliasing must be a concern any time you encounter a shader that does such things: color = (alpha < 0.5) ? red : green;
    
    color = lookupTable[alpha];
    
    result = pow(nDotL, 16);
    The act of indexing into a lookup table is especially important to graphics programmers, because this is exactly what we do each time we sample a texture map! Fortunately, GPUs provide sophisticated mechanisms to avoid aliasing when performing such lookups. If your pixel shader contains other nonlinear math that is causing aliasing artifacts, a good solution is often to replace these computations with a texture lookup, thus bringing the power of bilinear filtering and mipmaps to bear on the problem.
    
    Specular Light Aliasing
    
    One of the most everyday and yet pernicious examples of shader aliasing is the humble specular light. This is so common and inoffensive that we built it into the standard BasicEffect lighting model, and yet specular lighting involves a power computation which is inherently nonlinear, and thus a source of aliasing. The higher you crank the specular power setting, the shiner the object looks, and also the worse the aliasing becomes.
    Surprisingly for something that is so widely used, there is no universally accepted solution for specular aliasing problems. Many people just turn down their specular intensity or specular power for whichever models show the worst artifacts, put up with minor remaining flaws, and call it good.
    A full discussion of how to antialias specular lighting is beyond the scope of this article. The Self Shadow blog has a great summary of the topic.
    
    Source