I finally got AA working for primitives in the fragment shader! I've wanted to add this for awhile once I migrated my software renderer to a proper GPU-driven one, but I was a bit surprised to find a relative lack of resources on doing it in the fragment shader as opposed to more typical solutions like super-sampling, MSAA, FXAA, etc. However, there is one thing that I still don't quite understand how/why it works.

My approach is basically:

- determine whether fragment/pixel lies on the edge within some range
- if it does, calculate alpha by performing smoothstep based on the pixel's position within the range

Here is the shader (Metal): https://gist.github.com/cfloisand/64c5965b6babf6915cf001cef0f3a922

(All coordinates are in NDC.)

The part in question is with the edge testing. To test whether a pixel is within the range of an edge, I calculate the dot product with the perpendicular to the edge. Naturally, the length of the edge will affect the value of the dot product, so I store this value in the Edges data that I pass to the fragment shader as edgeMaximas, and this value is calculated as the length of the edge divided by half the height of the render target.

e.g.

edges.edgeMaximas[0] = length(v0, v1) / renderTarget.height / 2.f; edges.edgeMaximas[1] = length(v1, v2) / renderTarget.height / 2.f; ...

Why does this calculation work? Specifically dividing by half the height of the render target. I must admit, I arrived at this through experimentation (and maybe a bit of intuition.. :) ), but I'm not confident I know why it works. I've tested at various resolutions and with various sizes of triangles, and it all works!

As you can see, the results are very good (better than I expected!):