Quick sanity check for my mouse selection process and a question about ray casting in general:

I'm calling XMVector3Unproject twice, once with z dimension at ViewportMinZ (0.0) and once at ViewportMaxZ (1.0).

I find the diff between near plane z and target entity z (which is fixed and known for now), turn that into a ratio over the total z distance between near and far plane, then multiply that ratio by the differences in x,y between the near and far plane to find the x,y at the target z depth.

It looks something like this:

vec3 pos = {0};
pos.z = (zDepth - rayPoints[0].z);
vec3 diff = rayPoints[1] - rayPoints[0];
ASSERT(diff.z != 0.0f);
float zRatio = pos.z/diff.z;
ASSERT(zRatio >= 0.0f && zRatio <= 1.0f);
pos.xy = rayPoints[0].xy + zRatio*diff.xy;
return pos;

This works splendidly for my current wonky prototyping, but I made this up on my own instead of copying something someone else did so I'm checking whether I should be doing this differently?

ALSO, for ray casting in general (for when I don't know the depth of my target), would I do the same thing as I'm doing here but then brute force iterate over the positions of all entities in the scene testing their z depths to see if they're in the ray to aggregate a collection of entities that the ray passes through? Or is there a better approach?

Thanks, Robert

Assert is not useful for division by zero checks, because of the low chance of being caught in debug mode. The best way to prevent rare runtime exceptions from causing crashes at the end user, is by handling all the cases to begin with and testing the edge cases in regression tests. Conditional move instructions only take one CPU cycle, so don't be afraid of using the ? operator in C/C++ when handling special cases. No branching will be performed even if you write it as a trivial if statement with an assignment inside.

The standard way of making ray casts is by having broad-phases with height-fields (ground), grids (dense) and trees (sparse), so that one can skip whole groups of objects that have a combined hull not intersecting with the line. No need to involve the view frustum or depth buffer, because the line might have nothing to do with the player's line of sight and you will end up writing a general intersection eventually.

You also need to filter line intersections based on item groups to allow passing through different items based on what is being tested with the ray intersection. Usually done efficiently using bit masks. If you then have a group of items containing glass and grass, their bit masks are combined with bitwise or operations. Then sight rays may have a bit mask checking for the grass and concrete bits, masking the item group with bitwise and. If the result is not zero, then one of the occluding groups was present (grass) and the group will check the items and sub-groups for their masks before performing the line intersections on individual items.

static const uint32_t group_glass = 1u << 0;
static const uint32_t group_grass = 1u << 1;
static const uint32_t group_stone = 1u << 2;
static const uint32_t groupMask_occluding = group_grass | group_stone;
static const uint32_t groupMask_bulletHit = group_glass | group_stone;
static const uint32_t groupMask_bulletStop = group_stone;

If you are using it for bullets or checking where one can walk for AI, you also need a thickness on the line. This can be added to the thickness of any convex shape being intersected.

Thanks, Dawoodoz! That gives me plenty more to research and think through.