Hello. For the last two weeks, I've been struggling with one thing... 3D object picking. And I'm near getting it right! Works great when facing front! With a first person style camera, I can go up, down, forward, backward, strafe left, straft right, and it works. Problem is, when I rotate the camera, the other end of the ray that is not the mouse end goes off in another place other than the camera, completely throwing it off! So I'm going to go step by step, and see if you guys can spot what went wrong.

The first step was to normalize the mouse device coordinates, or in my case, touch coordinates:

    public static float[] getNormalizedDeviceCoords(float touchX, float touchY){
        float[] result = new float[2];
        result[0] = (2f * touchX) / Render.camera.screenWidth - 1f;
        result[1] = 1f - (2f * touchY) / Render.camera.screenHeight;

        return result;
    }

which in turn is converted into Homogeneous Clip Coordinates:

float[] homogeneousClipCoords = new float[]{normalizedDeviceCoords[0], normalizedDeviceCoords[1], -1f, 1f};

The next step was to convert this Homogeneous Clip Coordinates into Eye Coordinates:

    public static float[] getEyeCoords(float[] clipCoords){
        float[] invertedProjection = new float[16];
        Matrix.invertM(invertedProjection, 0, Render.camera.projMatrix, 0);

        float[] eyeCoords = new float[4];
        Matrix.multiplyMV(eyeCoords, 0, invertedProjection, 0 ,clipCoords, 0);

        float[] result = new float[]{eyeCoords[0], eyeCoords[1], -1f, 0f};

        return result;
    }

Next was to convert the Eye Coordinates into World Coordinates and normalize it:

    public static float[] getWorldCoords(float[] eyeCoords){
        float[] invertedViewMatrix = new float[16];
        Matrix.invertM(invertedViewMatrix, 0, Render.camera.viewM, 0);

        float[] rayWorld = new float[4];
        Matrix.multiplyMV(rayWorld, 0, invertedViewMatrix, 0 ,eyeCoords, 0);

        float length = (float)Math.sqrt(rayWorld[0] * rayWorld[0] +
                                        rayWorld[1] * rayWorld[1] +
                                        rayWorld[2] * rayWorld[2]);
        if(length != 0){
            rayWorld[0] /= length;
            rayWorld[1] /= length;
            rayWorld[2] /= length;
        }

        return rayWorld;
    }

Putting this all together gives me a method to get the ray direction I need:

    public static float[] calculateMouseRay(){
        float touchX = MainActivity.touch.x;
        float touchY = MainActivity.touch.y;

        float[] normalizedDeviceCoords = getNormalizedDeviceCoords(touchX, touchY);
        float[] homogeneousClipCoords = new float[]{normalizedDeviceCoords[0], normalizedDeviceCoords[1], -1f, 1f};
        float[] eyeCoords = getEyeCoords(homogeneousClipCoords);
        float[] worldCoords = getWorldCoords(eyeCoords);

        return worldCoords;
    }

I then test for the Ray / Sphere intersection using this with double precision:

    public static boolean getRaySphereIntersection(float[] rayOrigin, float[] spherePosition, float[] rayDirection, float radius){

        double[] v = new double[4];

        double[] dir = new double[4];

        // Calculate the a, b, c and d coefficients.
        // a = (XB-XA)^2 + (YB-YA)^2 + (ZB-ZA)^2
        // b = 2 * ((XB-XA)(XA-XC) + (YB-YA)(YA-YC) + (ZB-ZA)(ZA-ZC))
        // c = (XA-XC)^2 + (YA-YC)^2 + (ZA-ZC)^2 - r^2
        // d = b^2 - 4*a*c

        v[0] = (double)rayOrigin[0] - (double)spherePosition[0];
        v[1] = (double)rayOrigin[1] - (double)spherePosition[1];
        v[2] = (double)rayOrigin[2] - (double)spherePosition[2];

        dir[0] = (double)rayDirection[0];
        dir[1] = (double)rayDirection[1];
        dir[2] = (double)rayDirection[2];

        double a = (dir[0] * dir[0]) +
                   (dir[1] * dir[1]) +
                   (dir[2] * dir[2]);
        double b = (dir[0] * v[0] +
                    dir[1] * v[1] +
                    dir[2] * v[2]) * 2.0;
        double c = (v[0] * v[0] +
                    v[1] * v[1] +
                    v[2] * v[2]) - ((double)radius * (double)radius);

        // Find the discriminant.
        //double d = (b * b) - c;
        double d = (b * b) - (4.0 * a * c);
        Log.d("d", String.valueOf(d));

        if (d == 0f) {
            //one root
        }
        else if (d > 0f) {
            //two roots

            double x1 = -b - Math.sqrt(d) / (2.0 * a);
            double x2 = -b + Math.sqrt(d) / (2.0 * a);

           Log.d("X1 X2", String.valueOf(x1) + ", " + String.valueOf(x2));

            if ((x1 >= 0.0) || (x2 >= 0.0)){
                return true;
            }

            if ((x1 < 0.0) || (x2 >= 0.0)){
                return true;
            }
        }

        return false;
    }

After a week and a half of playing around with this chunk of code, and researching everything I could on google, I found out by sheer accident that the sphere position to use in this method must be the transformed sphere position extracted from the model matrix, not the position itself. Which not one damn tutorial or forum article mentioned! And works great using this. Haven't tested the objects modelView yet though. To visually see the ray, I made a class to draw the 3D line, and noticed that it has no trouble at all with one end being my mouse cursor. The other end, which is the origin, is sort of working. And it only messes up when I rotate left or right as I move around in a FPS style camera. Which brings me to my next point. I have no idea what the ray origin should be for the camera. And I have 4 choices. 3 of them worked but gave me the same results.

Ray Origin Choices:

1. Using just the camera.position.x, camera.position.y, and camera.position.z for my ray origin worked flawlessly straight due to the fact that the ray origin remained in the center of the screen, but messed up when I rotated the camera, and moved off screen as I was rotating. Now theoretically, even if you were facing at an angle, you still are fixated at that point, and the ray origin shouldn't be flying off away from the center of the screen at all. A point is a point after all.

2.Using the cameras model matrix (used for translating and rotating the camera, and later multiplied to the cameras view matrix), specifically -modelMatrix[12], -modelMatrix[13], and -modelMatrix[14] (note I am using negative), basically gave me nearly the same results. Only difference is that camera rotations play a role in the cameras positions. Great facing straight, but the ray origin is no longer centered at different angles.

3.Using the camera's view matrix didn't work at all, positive or negative, using 12, 13, and 14 in the matrix.

4.Using the camera's inverted view matrix (positive invertedViewMatrix[12], invertedViewMatrix[13], and invertedViewMatrix[14]) did work, but gave me what probably seemed like the same results as #2.

So basically, I'm having difficulty getting the other end of the ray, which is the ray origin. Shooting the ray to the mouse pointer was no problem, like I said. With the camera end of the ray being off, it throws off the accuracy a lot at different camera angles other than straight. If anyone has any idea's, please let me know. I'm sure my math is correct. If you need any more information, such as the camera, or how I render the ray, which I don't think is needed, I can show that too. Thanks in advance!

You need a near plane definition and screen size and camera position, idea is that your near plane defines your actual screen: your view frustum starts from there and its actally the size of your opengl buffer

inline vec3 GetDirectionFromScreen(mat4 modelview, TSpecCamera * FPP_CAM,  int x, int y, int sw, int sh, float fov, float z_near)//, float z_far)
{

mat4 mvm = modelview;
mvm.Transpose();
vec3 dirX, dirY;
	dirX.x = mvm.m[0];
	dirX.y = mvm.m[4];
	dirX.z = mvm.m[8];

	dirY.x =	mvm.m[1];
	dirY.y =	mvm.m[5];
	dirY.z =	mvm.m[9];

	float aspect = float(SCREEN_WIDTH) / float(SCREEN_HEIGHT);
	float a = fov / 2.0;
float cotangent = 1.0 / tan( a * imopi );

float ax = z_near / cotangent;

float screen_w = 2.0*ax;

float screen_h = screen_w;// * yratio;

screen_w = screen_w * aspect;

float scr_coord_x = float(x) / float(sw);
float scr_coord_y = float(sh - y) / float(sh);


vec3 dir = FPP_CAM->ReturnFrontVector();

//move to lower left corner
vec3 start_pos = (FPP_CAM->pos + dir * z_near) + (-dirX * (screen_w / 2.0)) + (-dirY * (screen_h/2.0));

vec3 start = start_pos + (dirX * (screen_w * scr_coord_x)) + (dirY * (screen_h * scr_coord_y));

return Normalize( vectorAB(FPP_CAM->pos, start) );
}

You dont need modelview matrix unless you know right front and up vectors, however this is an old code and i do a transpose there - if you use opengl style matrix then dont call transpose i guess

I'll give it a shot, even though my near z is 1

In the mean time, I'm constructing a camera array so there are 2 cameras to see where the hell my ray origin is going. Gonna see it as a 3D line. It's pretty hard to see it when its on ya.

Nevermind. That wasn't really needed. I should slap myself. Turns out that when I update the matrices of any of the 3D objects, I was multiplying the X values by a stupid aspect ratio that wasn't needed at all, throwing off the values completely since the picking wasn't using an aspect ratio. So I took aspect ratios out of the equation and it worked. Can't believe it, this whole time!

Worked like a charm when using camera.position x y and z values for the picking origin and ray origin of my 3D line, but I noticed it was a teeny bit off at certain angles, although acceptable. Just for the hell of it, I replaced the origin's camera position with the cameras inverted view matrix, and low and behold, it was pixel perfect accurate at all angles, even when my picking method was using floats instead of doubles!!!

You multiply your vector by inverse projection matrix, but then discard Z/W values and put -1,0 in there. Z=-1 is only correct if your zNear is 1. Setting W=0 makes it ignore the translation component of inverse view matrix, which is what you need if you just want the ray direction. The origin of the ray is, indeed, your camera position. It is the same as the translation column (12,13,14) in inverse view matrix.

You should keep the Z so it works for every zNear. The rest seems fine, and you seem to have figured out the problem by now.

Glad you got it working! Just posting up a reference in case anyone else comes along in this thread: https://github.com/RandyGaul/cute_headers/blob/master/cute_math.h#L630-L649

void compute_mouse_ray( float mouse_x, float mouse_y, float fov, float viewport_w, float viewport_h, float* cam_inv, float near_plane_dist, v3* mouse_pos, v3* mouse_dir )
{
	float aspect = (float)viewport_w / (float)viewport_h;
	float px = 2.0f * aspect * mouse_x / viewport_w - aspect;
	float py = -2.0f * mouse_y / viewport_h + 1.0f;
	float pz = -1.0f / tanf( fov / 2.0f );
	v3 point_in_view_space( px, py, pz );

	v3 cam_pos( cam_inv[ 12 ], cam_inv[ 13 ], cam_inv[ 14 ] );
	float pf[ 4 ] = { getx( point_in_view_space ), gety( point_in_view_space ), getz( point_in_view_space ), 1.0f };
	tgMulv( cam_inv, pf );
	v3 point_on_clipping_plane( pf[ 0 ] , pf[ 1 ], pf[ 2 ] );
	v3 dir_in_world_space = point_on_clipping_plane - cam_pos;

	v3 dir = norm( dir_in_world_space );
	v3 cam_forward( cam_inv[ 8 ], cam_inv[ 9 ], cam_inv[ 10 ] );

	*mouse_dir = dir;
	*mouse_pos = cam_pos + dir * dot( dir, cam_forward ) * vfloat( near_plane_dist );
}