What am I doing wrong when trying to calculate a ray from the camera?

glm::vec4 mouseClip = {ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 0, 1};

glm::mat4 inv = glm::inverse(game::camera->projection * game::camera->transform);

glm::vec4 p = inv * mouseClip;
p.x /= p.w;
p.y /= p.w;
p.z /= p.w;

glm::vec3 direction = glm::normalize(glm::vec3{p});

I can guarantee that mouseClip's x and y is correct (from [-1, -1] to [1, 1]).

glm::vec3 intPos, intNor;
if(glm::intersectRaySphere(glm::vec3{game::camera->transform[3]}, direction, glm::vec3{0, 0, 0}, 1, intPos, intNor)) {
    printf("intersect\n");
}

intersectRaySphere does as it says, here is it's documentation.

Problem is that.. well.. basically everything is off. There is nothing I can accurately describe, it seems as if the camera's transformation is always exaggerated. I know this is all very vague, please ask questions if you have any.

I tried using glm::unProject instead, but issues still exist:

glm::vec3 direction = glm::normalize(glm::unProject(glm::vec3{ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 1}, game::camera->transform, game::camera->projection, glm::vec4{0, 0, 640, 480}));

Issue is common due to window dimensions and mouse relative position.

Where windiw with border can be 640x480 the actual glwindow may be smaller and the mouse coordinate can be calculated from window left top including border if you managed to fix that and have still bad results then you could calculate the ray from near z plane knowing left top corner as 0,0 you can clac the ray knowing mouse relative to window coord sth like this:

inline vec3 GetDirectionFromScreen(mat4 modelview, TSpecCamera * FPP_CAM,  int x, int y, int sw, int sh, float fov, float z_near, float aspect)//, float z_far)
{

mat4 mvm = modelview;
mvm.Transpose();
vec3 dirX, dirY;
	dirX.x = mvm.m[0];
	dirX.y = mvm.m[4];
	dirX.z = mvm.m[8];

	dirY.x =	mvm.m[1];
	dirY.y =	mvm.m[5];
	dirY.z =	mvm.m[9];

//	float aspect = float (SCREEN_WIDTH) / float (SCREEN_HEIGHT);
	float a = fov / 2.0;
float cotangent = 1.0 / tan( a * imopi );

float ax = z_near / cotangent;

float screen_w = 2.0*ax;

float screen_h = screen_w;// * yratio;

screen_w = screen_w * aspect;

float scr_coord_x = float(x) / float(sw);
float scr_coord_y = float(sh - y) / float(sh);

vec3 dir = FPP_CAM->ReturnFrontVector();

//move to lower left corner
vec3 start_pos = (FPP_CAM->pos + dir * z_near) + (-dirX * (screen_w / 2.0)) + (-dirY * (screen_h/2.0));

vec3 start = start_pos + (dirX * (screen_w * scr_coord_x)) + (dirY * (screen_h * scr_coord_y));

return Normalize( vectorAB(FPP_CAM->pos, start) );

}

Its an old code not sure if fov was fovy imopi stands for pi/180

Coords are.in pixels as i think

As I said, I can guarantee the mouse position is correct, that's not the issue.

I apparently have passed a few wrong arguments to glm::unProject, here is the updated code:

glm::vec3 direction = glm::unProject(glm::vec3{ev.cursor.x, 480 - ev.cursor.y, 1}, glm::mat4{1}, game::camera->projection * game::camera->transform, glm::vec4{0, 480, 640, -480});
direction = glm::normalize(direction - glm::vec3{game::camera->transform[3]});

And now it works for almost all cases. It starts to break if there's any Y translation, or any rotation at all, or if I perform glm::translate and glm::rotate in the wrong order. Those values seem to be always exaggerated too much for seemingly no reason.

The Y translation making it incorrect? That was a non-issue, I was inverting the mouse's Y coordinate twice, durr.
But rotation in the view matrix is still making it incorrect, said rotation (on any axis) also seems to be exaggerated, as if it's scaled up.

Fixed the issue. And, as ALWAYS it's the stupidest issues you can't even imagine.

Check the unProject documentation:

Quote

detail::tvec3<T> glm::gtc::matrix_transform::unProject(detail::tvec3< T > const& win,
		detail::tmat4x4< T > const& model,
		detail::tmat4x4< T > const& proj,
		detail::tvec4< U > const& viewport 
)

 

The second argument? That's the modelview matrix, not just the view matrix! So I just ended up passing the inverse of the view matrix there, and it works fantastically now.

It does not make any sense since model matrix is the object matrix not camera one. Maybe you think about cam_translation_mat*cam_rot_mat?

It's GLM's documentation that's misleading. The program now works without issues.

It seems to be correct in 0.9.9:

unproject