Gahh, I think I solved the issue I was having. I'm using the GLM math library (but slowly replacing it with my own structs, functions, etc.), and apparently matrix multiplication in GLM is reverse of what I was expecting. To apply a rotation matrix (that I was generating from my quaternion) to the model matrix I was doing
But I really had to do
This issue only popped up when I switched to quaternions because I was previously using a built-in GLM rotation function instead of calculating the matrix myself and doing the multiplication. Thanks for the help everyone, I think I got it working now :D