Hi!

I am making a RTS game with OpenGL 2 and I am at the point where I am only adding in game mechanics. As any RTS game will likely need this, I am trying to implement mouse picking from the screen coordinates to the world coordinates. I tried many different solutions with view and projection matrices using matrix math, and I couldnt get it to work, and then I found gluUnProject, and so far, that has been the closest to working. It follows the mouse mostly except for one small thing: the coordinates are slightly off. So far, the only things I have done to the camera that could cause this is that I rotated the camera itself 45 degrees on the X axis so that I have a 3D isometrical-type view of the map. When I try offsetting the model by about positive 30, it seems to stay with the mouse until I get near the edge of the screen, and that is where it starts going too far positive. I am currently using the Win32 API for my display, so I am not completely sure if that could have an effect on it.

I have spent a lot of time working on this, and I was just wondering if anyone here could help me out with my issue.

Here is my mouse cursor code (it is a modified version of the NEHE mousepicking code):

Vector3 GetOGLPos(double mvmatrix[16], double projmatrix[16], int viewPort[4], HWND hWnd)
{

	//GLint viewport[4];
	GLdouble modelview[16];
	GLdouble projection[16];

	GLfloat winX, winY, winZ;
	GLdouble posX, posY, posZ;

	GetCursorPos(&p);                   // Gets The Current Cursor Coordinates (Mouse Coordinates)
	ScreenToClient(hWnd, &p);


	glDisable(GL_DEPTH_TEST);

	glGetDoublev(GL_MODELVIEW_MATRIX, modelview);
	glGetDoublev(GL_PROJECTION_MATRIX, projection);
	glGetIntegerv(GL_VIEWPORT, viewPort);

	winX = (float)p.x;
	winY = (float)viewPort[3] - (float)p.y;
	//glReadPixels(winX, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

	gluUnProject(winX, winY, 0.0, modelview, projection, viewPort, &posX, &posY, &posZ);
	
	gluUnProject(winX, winY, 1.0, modelview, projection, viewPort, &posX, &posY, &posZ);

	glEnable(GL_DEPTH_TEST);


	return Vector3(posX, posY, posZ);
}

Keep in mind that variable p is a point that represents the mouse cursor position. It supports both an X and a Y value. p.x is the mouse's X coordinates and p.y is the mouse's Y coordinates.

Thank y'all in advance!

-Ryan

Does your GetOGLPos function use the same matrices as when you're rendering? It doesn't look like it's using the relevant arguments, besides viewPort and hWnd.

(Also, wouldn't it be nicer for it to take the p directly, instead of juggling hWnd stuff here, and apparently also having the responsibility of updating p's value?)

@supervga I tried this, but my number goes to “nan”. I originally thought this was an issue because I was converting the number to a string with std::to_string, so I converted it to an integer, but that didnt fix it, as the number went from 0 to 2147483647 as soon as I moved the mouse… I wondeed why this could be happening, and I realized I forgot to change the glGetDoublev calls to my modelview matrix and projection matrix. However, when I did that, I got not a game bug, but an entire exception - an access violation. Perhaps this is because I already did this:

	glGetFloatv(GL_MODELVIEW_MATRIX, (GLfloat*)modelviewMatrix);
	glGetFloatv(GL_PROJECTION_MATRIX, (GLfloat*)projectionMatrix);

before this function is called. Keep in mind that modelviewMatrix and projectionMatrix are my actual matrices, and not parameters.

This is how I use the function:

Vector3 mouseWorldCoords = GetOGLPos((double *)modelviewMatrix, (double *)projectionMatrix, (int *)viewPort, hwnd);

And last but not least, even though this is probably not necessary, here is the modified version of GetOGLPos:

Vector3 GetOGLPos(double mvmatrix[16], double projmatrix[16], int viewPort[4], HWND hWnd)
{

	//GLint viewport[4];
	GLdouble modelview[16];
	GLdouble projection[16];

	GLfloat winX, winY, winZ;
	GLdouble posX, posY, posZ;

	GetCursorPos(&p);                   // Gets The Current Cursor Coordinates (Mouse Coordinates)
	ScreenToClient(hWnd, &p);


	glDisable(GL_DEPTH_TEST);

	glGetDoublev(GL_MODELVIEW_MATRIX, mvmatrix);
	glGetDoublev(GL_PROJECTION_MATRIX, projmatrix);
	glGetIntegerv(GL_VIEWPORT, viewPort);

	winX = (float)p.x;
	winY = (float)viewPort[3] - (float)p.y;
	//glReadPixels(winX, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

	gluUnProject(winX, winY, 0.0, mvmatrix, projmatrix, viewPort, &posX, &posY, &posZ);
	
	gluUnProject(winX, winY, 1.0, mvmatrix, projmatrix, viewPort, &posX, &posY, &posZ);

	glEnable(GL_DEPTH_TEST);


	return Vector3(posX, posY, posZ);
}

I noticed that if I used glReadPixels, the coordinates would be accurate for a short time on the Z axis, but as the coordinates got closer and closer to x = 1, they would start flickering a bit to different coordinates, and then I would get the weird Z offset issue.

I am not sure if I should just go back to using glReadPixels, as the coordinates were correct in one area, or if I should just continue using both of the gluUnProject functions.

In other words, it worked better when I was using the projection and modelview matrices that I had in the function itself.

I'm a bit confused about your two lines

glGetFloatv(GL_MODELVIEW_MATRIX, (GLfloat*)modelviewMatrix);

vs.

GetOGLPos((double *)modelviewMatrix, (double *)projectionMatrix, (int *)viewPort, hwnd);

Are you sure you really really REALLY want to use the same array of however you defined that in your code once as a float pointer and once as a double pointer (which is also double in sise of the expected float range)?

This doesn't look right to me nor is it recommendet because hardware is working with float numbers anyways. Maybe the cause of your exception as you tried to write outisde of the acquirred memory region

@shaarigan

Thank you for replying!

GetOGLPos is the function that i initializes all of the mousepicking stuff. And what is confusing you about my use of glGetFloat? I have reverted my code back to where I had no exceptions. Do you think I should go back to the code that returns an exception? Did that seem a bit more sensible?

-Ryan

@shaarigan and @supervga I have come up with a possible solution. Maybe it is that I rotated the camera on one matrix and I did not rotate it on the other matrix.