refer to image … i was clicking at red position and the pick is registered at the vertical pole, where i was setting nearest point on height map to height of 256, see in green…

this is my code for picking …

UnprojectMouseCoord()
{
if (!m_TerrainInitilized)
{
return false;
}

Vector3 mouseNear = Vector3(m_MouseX, m_MouseY, 0.0f);
Vector3 mouseFar = Vector3(m_MouseX, m_MouseY, 1.0f);

mouseNear = m_Viewport.Unproject(mouseNear, m_ProjectionMatrix, m_Camera->GetViewMatrix(), m_WorldMatrix);
mouseFar = m_Viewport.Unproject(mouseFar, m_ProjectionMatrix, m_Camera->GetViewMatrix(), m_WorldMatrix);

m_RayDirection = mouseFar - mouseNear;
m_RayDirection.Normalize();

m_RayOrigin = mouseNear;

return true;
}

this is my all black zero height , height map, where i am picking the point on map using integer cast of plane - ray intersection point and finding the index of point in map …

Plane p( Vector3(0.0f), Vector3(0.0f, 1.0f, 0.0f));
float odot = p.DotNormal(rayOrigin);
float ndot = p.DotNormal(rayDirection);
float t = -1.0f * odot / ndot;

//perpendicular to plane
if (ndot == 0.0f)
{
return false;
}

Vector3 pos = rayOrigin + t * rayDirection;

int x = pos.x;
int z = pos.z;
if (x < 0 || z < 0 || x > m_TerrainWidth-1 || z > m_TerrainHeight-1)
{
return false;
}

uint64_t index = (m_TerrainHeight * z) + x;
m_HeightMap[index].position.y = 256.0f;
return true;

but it is incorrect and i am not getting the exact point , where my mouse has clicked on terrain. I am using directxTK for maths and geometry wrapper.

This depends on what your m_Viewport implementation does.

E g, the code you posted looks OK (although hard to read without a code block) but the math error probably lives elsewhere.

You should print a log message with input mouse x, input mouse y, output mouse near, output mouse direction, and output terrain raycast hit result, each time the user clicks.

That way, you can work through the numbers and see what's going on, clicking in each of the corners and the center of the screen, to get some numbers to match up.

i did the debugging and ray origin seems to be correct … but the ray direction is at angle and not where i clicked … confused……

m_Viewport is just directxtk viewport … which has correct fields …

https://github.com/microsoft/DirectXTK/wiki/Viewport

OK the issue was on resize the incorrect window rect dimension was passed to viewport ….

Would like to continue further this discussion for better way to Terrain - Ray intersection …

suggestion are welcomed…