Hello, I am making a Software renderer, and I noticed that there is a screen shaking like effect happening. shown in the first video:

first_video.mov

I tried doing filling without clipping against the small square, and the result is much better, yet, there is some sort of shaking on the edges as shown in the second video: second_video.mov

Finally I tried drawing the clipped triangles outline, and the shaking is more apparent as shown in the third video: third_video.mov

My initial assumption was the line drawing was not accurate enough, so I tried both DDA and Bresenham line drawing without any success.

Additionally, I tried different triangulation schemes, I noticed that when I increase the number of created triangles (such as when I triangulate a square using the center as a new vertex instead of using one of the corners) the shaking gets worse. That led me to believe to the issue is related to shared edges in the triangles, and indeed in my triangle filling algorithm, I tried to remove the shared edge and got a bit better results (but it introduced different artifacts, some pixels were missing): https://git.sr.ht/~aliabdulkareem/SoftwareRenderer/tree/FixTrianglulation/item/src/drawing.cpp#L450

That did not however remove it entirely (I believe I should remove the shared edges from the triangulation phase as well, not entirely sure how to do that, one idea is when I triangulate the triangles after the first one, I always more pixel in the direction of the two newly formed edges)

Also it does not explain entirely the second video shaking when there was a shared edge in the diagonal of a square, but it was fine, while the square edges were shaking despite not sharing an edge.

I am a bit puzzled as how to debug this as I can at least think of three different reasons:

1- Triangulation is wrong

2- Floating point inaccuracy

3- Aliasing (I heard that in animations, aliasing becomes very clear, not sure if that is the case)

Full source code: https://git.sr.ht/~aliabdulkareem/SoftwareRenderer/tree/FixTrianglulation

If you look at triangle outlines in your first video you can see that you have some kind of issue filling it properly:

Something is not filling something properly.

My guess is that happens because of your floats with --H or ++H style of loops in your DrawTop/BottomTriangle. Because they will not compare exactly on equality for ending condition. Calculate it in integers instead, so you will hit ending conditions exactly.

Updating XLeft/XRight in floats probably also contributes to same problem.

And be careful when you're passing (s32) casted floats to your DrawHLine, because that is with truncation. Meaning values like 4.9 will be truncated to 4. But 4.9 would probably means you want to fill pixel with coordinate 4, because 0.9 covers most of pixel.

To debug these kind of issues, find values for which it looks wrong. Stop the rotation - draw fixed triangle. And then step through code examining values. Because you'll be drawing fixed triangle, the values every time will be the same.

In general you should do rasterization in fixed point values. See Fabian's posts on triangle rasterization which covers sub-pixel accuracy: https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlusion-culling-index/

I'd suggest to do your videos in mp4 file format, then they will play inline in other browsers (mov is Apple/Safari thing).