What are your references when writing matrix and quaternion code as far as the math go?
I often spend a lot of time swapping operations around until I get the result I need.
Is there a good book that I could use, something specifically written for programmers?

"Essential Mathematics for Games and Interactive Applications: A Programmer's Guide" (James M. Van Verth and Lars M. Bishop) is ok.

"Mathematics for 3D Game Programming and Computer Graphics (Eric Lengyel) is also good.
It has never alternative of it by name "Foundations of Game Engine Development, Volume 1: Mathematics".

"Real-Time Collision Detection" (Christer Ericson) also is ok.