Hi I am trying to implement ray-casting in my engine so that I can pick objects in 3D. I am able to cast rays by clicking the screen and they are projected into the world with relatively ok accuracy in terms of their origin point. However, the problem is that no matter which direction I cast the ray, it always goes through the origin of the world (0,0,0), see the screenshot below:

As you can see, all the rays diverge to the middle of the world, even though that is not the point on the screen where I clicked. Below is the code I use to generate the ray. Note that I am not normalizing the direction vector because I found that when I do that, the ray is cut off and doesn't go all the way to the middle. What am I doing wrong? Thanks

D3DXVECTOR3 rayStart, rayDir;

	rayStart = m_Camera->GetPosition();
	cout << m_screenWidth << " " << m_screenHeight << endl;
	rayDir.x = ((2.0f * (float)mouseX) / (float)m_screenWidth) - 1.0f;
	rayDir.y = ((2.0f * (float)mouseY) / (float)m_screenHeight) - 1.0f;
	rayDir.z = 1.0f;

	D3DXMATRIX viewMatrix, projectionMatrix, inverseMatrix;
	m_Camera->GetViewMatrix(viewMatrix);
	m_D3D->GetProjectionMatrix(projectionMatrix);

	D3DXMatrixMultiply(&inverseMatrix, &viewMatrix, &projectionMatrix);
	D3DXMatrixInverse(&inverseMatrix, NULL, &inverseMatrix);

	D3DXVec3TransformNormal(&rayDir, &rayDir, &inverseMatrix);
	//D3DXVec3Normalize(&rayDir, &rayDir);

	Line* line = new Line;
	line->Initialize(m_D3D->GetDevice(), 0, rayStart.x, rayStart.y, rayStart.z, rayDir.x, rayDir.y, rayDir.z, 1.0f, 0.0f, 1.0f);
	lines.push_back(line);

	for (int i = 0; i < meshes.size(); i++) {
		bool intersect = false;
		
		intersect = TavianAABBIntersect(rayStart, rayDir, meshes[i]);
		if (intersect) {
			selected = meshes[i];
			break;
		}
	}

@adam7 I took a look at some 3d Picking I did a while back:

Usually I see picking routines that use the ‘Unproject()’ function, but I rolled my own. It looks like you're using the inverse of the projection matrix, I'm not sure that's what you need. It's been a while since I looked at this:

----------------------------------------

//Scale the X&Y so they're e(-1,+1)

//The projection matrix represent the Angles of the frustum:

//proj(0,0)=ratio*tan(fov/2) and proj(1,1)=tan(fov/2)

//zDir=1

//

//Unproject() could do this also

float fXDir = 2.0f*MousePos.x / fWidth - 1.0f;

float fYDir = -(2.0f*MousePos.y / fHeight - 1.0f);

fXDir /= pCamera->GetProjection()._11;

fYDir /= pCamera->GetProjection()._22;

vStart = XMFLOAT3(0.0f, 0.0f, 0.0f); //initially centered at the origin

XMVECTOR vStartPos(XMLoadFloat3(&vStart));

vDirection = XMFLOAT3(fXDir, fYDir, 1.0f); //move inward at an Dir

XMVECTOR vDir2(XMLoadFloat3(&vDirection));

//--------------------------------------------

//Calc direction

//Take from screen space to worldspace

XMMATRIX matView(XMLoadFloat4x4(&pCamera->GetView()));

XMMATRIX matInverseView;

matInverseView = XMMatrixInverse(nullptr, matView);

vDir2 = XMVector3TransformNormal(vDir2, matInverseView); //We're changing a direction (vector) not a point!

vStartPos = XMVector3TransformCoord(vStartPos, matInverseView);

vDir2 = XMVector3Normalize(vDir2);

XMStoreFloat3(&vDirection, vDir2);

XMStoreFloat3(&vStart, vStartPos);

if you want to unproject your screen point to world space you will also need to divide the vector by its w component after applying the inverse transformation matrix see: https://feepingcreature.github.io/math.html​

So here your rayDir is bad, its initial w component should be equal to 1.

Also you want to launch rays from your camera to your screen point placed at the far plane, you may prefer a ray from your screen point in the near plane to your screen point at the far plane.

Vector2 ndc;
ndc.x = ((2.0f * (float)mouseX) / (float)m_screenWidth) - 1.0f; 
ndc.y = ((2.0f * (float)mouseY) / (float)m_screenHeight) - 1.0f;
Vector4 screenNear = Vector4(ndc.x, ndc.y, -1, 1);
Vector4 screenFar = Vector4(ndc.x, ndc.y, 1, 1);
Vector4 worldNear = inv_mvp * screenNear;
Vector4 worldFar    = inv_mvp * screenFar;
worldNear.xyz /= worldNear.w;
worldFar.xyz /= worldFar.w;
Line* line = new Line;
Line→init(worldNear.xyz, worldFar.xyz);

Thanks your responses. I have changed the line D3DXVec3TransformNormal(&rayDir, &rayDir, &inverseMatrix); to D3DXVec3TransformCoord(&rayDir, &rayDir, &inverseMatrix); which seems to have ‘unlocked’ the rays and now they go roughly in the direction they are cast in. My problem now is that the rays aren't cast directly from the screen but the position of the camera object which causes the ray direction to not be consistent with the point on the screen that is clicked. @thibault ober I will try your solution as this seems like the correct way to fix this problem. Can you clarify what is the inv_mvp matrix in your code? Is it one of the view/projection matrices or is it the result of multiplying both? Thanks.

mvp stands for ModelViewProjection, so here it's the multiplication of both.