Hi, first post on this site. I have mostly been able to find solutions to my problems by google searching, and eventually finding a solution. I even search for a few days until I find what I am looking for. However, this time I've been looking for over several months, and have been trying several different tutorials, and other options for my problem; for which none of them have been working for me.

I am trying to create a vec3 value by clicking onto the screen. I'm just looking for a simple formula that I can use based on the matrices given by opengl. I looked at gluUnProject, but it doesn't do what I want to achieve.

I've been attempting to follow this tutorial, but it doesn't seem to work for me. I've been getting -1.#IND values for my last matrix multiplication. Any help would be nice. I'm using glm math library.

http://antongerdelan.net/opengl/raycasting.html

void render()
{
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    glLoadIdentity();

    GLint viewport[4];
	GLdouble projMatrix[16];
	GLdouble modelMatrix[16];
        glGetIntegerv(GL_VIEWPORT, viewport);
	glGetDoublev(GL_PROJECTION, projMatrix);
	glGetDoublev(GL_MODELVIEW, modelMatrix);

//    m_world->render(); // <-- Only renders the grey ground.

/** STEP 0: 2d viewport coordinates **/
	int mX, mY;
	getWindow()->getMouse()->getMousePos(mX, mY);

	mX = max(mX, 0);
	mY = max(mY, 0);
	mX = min(mX, viewport[2]);
	mY = min(mY, viewport[3]);

/** STEP 1: 3d Normalised Device Coordinates **/

	float x = (2.0f * mX) / viewport[2] - 1.0f;
	float y = 1.0f - (2.0f * mY) / viewport[3];
	float z = 1.0f;
	vec3 ray_nds = vec3(x, y, z);

/** STEP 2: 4d Homogeneous Clip Coordinates **/

	vec4 ray_clip = vec4(ray_nds.x, ray_nds.y, -1.0f, 1.0f);

/** Step 3: 4d Eye (Camera) Coordinates **/
	double a = projMatrix[0];
	double b = projMatrix[5];
	double c = projMatrix[10];
	double d = projMatrix[14];
	double e = projMatrix[11];

	mat4 invProjMat4 = mat4(1.0 / a, 0.0f,	  0.0f,		0.0f,
				0.0f,	 1.0 / b, 0.0f,		0.0f,
				0.0f,	 0.0f,	  0.0f,		1.0 / e,
				0.0f,	 0.0f,	  1.0 / d, -c / (d * e));
	vec4 ray_eye = invProjMat4 * ray_clip;
	ray_eye = vec4(ray_eye.x, ray_eye.y, -1.0, 0.0f);

/** Step 4: 4d World Coordinates **/
	
	double m[16];
	gluInvertMatrix(modelMatrix, m);
	mat4 invViewMat4 = mat4(m[0], m[1], m[2], m[3],
			   m[4], m[5], m[6], m[7],
			   m[8], m[9], m[10], m[11],
			   m[12], m[13], m[14], m[15]);
	vec4 v = invViewMat4 * ray_eye;
	vec3 ray_wor = vec3(v.x, v.y, v.z);
	ray_wor = normalize(ray_wor);

	int glVersion[2] = {0, 0};
	glGetIntegerv(GL_MAJOR_VERSION, &glVersion[0]); 
	glGetIntegerv(GL_MINOR_VERSION, &glVersion[1]);

    stringstream msg[12];
    msg[0] << "Raycasting Demo";
    msg[1] << "Mouse";
    msg[2] << "mX: " << mX;
    msg[3] << "mY: " << mY;
    msg[4] << "ray_nds: " << ray_nds.x << ", " << ray_nds.y << ", " << ray_nds.z;
    msg[5] << "ray_clip: " << ray_clip.x << ", " << ray_clip.y << ", " << ray_clip.z << ", " << ray_clip.w;
    msg[6] << "ray_eye: " << ray_eye.x << ", " << ray_eye.y << ", " << ray_eye.z;
    msg[7] << "ray_wor: " << ray_wor.x << ", " << ray_wor.y << ", " << ray_wor.z;
// ...
    msg[10] << "OpenGL: " << glVersion[0] << ", " << glVersion[1];
    msg[11] << "Etc";

	for (int i = 0; i < 12; i++) {
    m_font->printString(msg[i].str(), 10.0f, viewport[3] - i * 14.0f - 20.0f);
	}

    stringstream fpsMessage;     
    fpsMessage << "Raycasting Demo " << std::setprecision(3) << m_FPS;
    m_font->printString(fpsMessage.str(), 20.0f, 20.0f);
}

I am trying to create a vec3 value by clicking onto the screen. ... I looked at gluUnProject, but it doesn't do what I want to achieve.

1. What do you want the vec3 to represent?

2. What doesn't gluUnProject not do that you need it to do?

Can you describe what you're attempting to do (rather than how you're trying to do it)?

Just a guess:

From the topic title, it sounds like you may be attempting a "pick" algorithm. If so, you haven't mentioned what in your scene you want to pick. If you're attempting to create the ray to do picking or raycasting, then, generally speaking, you use gluUnproject twice - once to get the world position of the mouse screen position at the near plane (call it rayNear), and a second time to get the world position of the mouse screen position at the far plane (call it rayFar).

The picking ray is then comprised of two vectors: rayNear (the ray origin) and the direction of the ray: rayDir = normalize( rayFar - rayNear ). Think of the mouse position on the screen as the tail of an arrow. The tail of the arrow is at rayNear (in world coordinates), and the direction that the arrow is pointing is rayDir (a direction in the world).

For sample code of what buckeye wrote, see MouseToWorldRay. It is C# opentk, but is basically the same

https://github.com/jeske/SimpleScene/blob/master/SimpleScene/Util/OpenTKHelper.cs

*facepalm* I seem to have misread gluUnProject weeks ago, and have just been avoiding it after that. I tried it again after looking at that code, and it's exactly what I was looking for. I just need to modify it to get both the zNear plane intersection point, and the zFar plane intersection point by changing the winZ values from 0 to 1, and make my vector.

Thanks a bunch.

#include <glm/glm.hpp>
using glm::vec3;

// Use the ScreenToClient method, before passing to the method, to convert the mouse co-ordinates given by GetCursorPos to window co-ordinates.
vec3 getOGLPos(int x, int y) {
	GLint viewport[4];
	GLdouble modelview[16];
	GLdouble projection[16];
	GLfloat winX, winY, winZ;
	GLdouble posX, posY, posZ;

	glGetDoublev(GL_MODELVIEW_MATRIX, modelview);
	glGetDoublev(GL_PROJECTION_MATRIX, projection);
	glGetIntegerv(GL_VIEWPORT, viewport);

	winX = (float)x;
	winY = (float)viewport[3] - (float)y;
	glReadPixels(x, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

	gluUnProject(winX, winY, winZ, modelview, projection, viewport, &posX, &posY, &posZ);

	return vec3(posX, posY, posZ);
}

// The call
    POINT pos;
    GetCursorPos(&pos);
    ScreenToClient(hwnd, &pos);
    vec3 point= getOGLPos(pos.x, pos.y);