today I was trying to unproject my mouse position to see where it is in world space.
Here it is the unproject function:
//project is the same exact matrix passed to opengl
Vec3 Unproject( Renderer* renderer, Camera* camera, matrix4x4 project, matrix4x4 backward, Vec2 screenP, r32 distanceFromCameraZ)
//compute ndc X and Y based on screen coordinates
Vec2 screenCenter = 0.5f * renderer->screenDim;
r32 ndcX = (screenP.x - screenCenter.x) * (2.0f / renderer->screenDim.x);
r32 ndcY = (screenP.y - screenCenter.y) * (2.0f / renderer->screenDim.y);
//let's see what's the clipZ of the targetZ
Vec4 probeZ = V4( camera->Position -worldDistanceFromCameraZ * camera->ZAxis, 1.0f );
//for debugging, let's force the worldZ of the probe to be on the ground plane, so that our points will always be on the ground plane
probeZ.z = 0;
probeZ = project * probeZ;
r32 clipZ =probeZ.z;
//"backward" perspective divide to obtain the clip coordinates
Vec4 clip = clipZ * V4(ndcX, ndcY, 1.0f, 1.0f);
//"backward" projection to obtain world coordinates
Vec3 world = (backward * clip).xyz;
It seems to work fine when the camera is perfectly aligned with the world Z Axis (0, 0, 1): (you can see the red rectangle perfectly fits the screen)
but as soon as I start rotating the camera it doesn't work anymore, as it seems it is still using the straight up camera.
Assuming the problem is not in the camera matrixes (checked multiple times both the forward and back matrixes), I think the problem should be _somewhere_ in this function, but I really can't see it.
Can you guys give me a little help with this?