All,

Used http://www.opengl-tutorial.org/miscellaneous/clicking-on-objects/picking-with-a-physics-library article as base for collision detection, I've got following issue.
Few inputs before problem itself
1. I am initializing my projection and view matrices once, within following piece of code

projectionMatrix = glm::perspective(90.0f, 4.0f / 3.0f, 0.1f, 100.0f);
   
 
   float horizontalAngle = 3.14f;
   float verticalAngle =0.0f;/// problems here 
 
   glm::vec3 direction(
      cos(verticalAngle) * sin(horizontalAngle), 
      sin(verticalAngle),
      cos(verticalAngle) * cos(horizontalAngle)
   );
 
   glm::vec3 position = glm::vec3( 0.0, 10.0, 32.0 ); 
 
   glm::vec3 right = glm::vec3(
      sin(horizontalAngle - 3.14f/2.0 ), 
      0,
      cos(horizontalAngle - 3.14f/2.0 )
   );
 
   // Up vector
   glm::vec3 up = glm::cross( right, direction );
 
   viewMatrix =  glm::lookAt(
                        position,          
                        position+direction, 
                        up                  
                     );

2. I do not change view and projection matrix based on cursor movements as it was in mentioned tutorial.
3. I am using btConvexHullShape adding a points to it.
4. Here is the code of ray calculation

 ScreenPosToWorldRay(
   int mouseX, int mouseY,             // Mouse position, in pixels, from bottom-left corner of the window
   int screenWidth, int screenHeight,  // Window size, in pixels
   glm::mat4 ViewMatrix,               // Camera position and orientation
   glm::mat4 ProjectionMatrix,         // Camera parameters (ratio, field of view, near and far planes)
   glm::vec3& out_origin,              // Ouput : Origin of the ray. /!\ Starts at the near plane, so if you want the ray to start at the camera's position instead, ignore this.
   glm::vec3& out_direction            // Ouput : Direction, in world space, of the ray that goes "through" the mouse.
){
 
   // The ray Start and End positions, in Normalized Device Coordinates (Have you read Tutorial 4 ?)
   glm::vec4 lRayStart_NDC(
      ((float)mouseX/(float)screenWidth  - 0.5f) * 2.0f, // [0,1024] -> [-1,1]
      ((float)mouseY/(float)screenHeight - 0.5f) * 2.0f, // [0, 768] -> [-1,1]
      -1.0, // The near plane maps to Z=-1 in Normalized Device Coordinates
      1.0f
   );
   glm::vec4 lRayEnd_NDC(
      ((float)mouseX/(float)screenWidth  - 0.5f) * 2.0f,
      ((float)mouseY/(float)screenHeight - 0.5f) * 2.0f,
      0.0,
      1.0f
   );
 
 
 
   // The Projection matrix goes from Camera Space to NDC.
   // So inverse(ProjectionMatrix) goes from NDC to Camera Space.
   glm::mat4 InverseProjectionMatrix = glm::inverse(ProjectionMatrix);
   
   // The View Matrix goes from World Space to Camera Space.
   // So inverse(ViewMatrix) goes from Camera Space to World Space.
   glm::mat4 InverseViewMatrix = glm::inverse(ViewMatrix);
   
   glm::vec4 lRayStart_camera = InverseProjectionMatrix * lRayStart_NDC;    lRayStart_camera/=lRayStart_camera.w;
   glm::vec4 lRayStart_world  = InverseViewMatrix       * lRayStart_camera; lRayStart_world /=lRayStart_world .w;
   glm::vec4 lRayEnd_camera   = InverseProjectionMatrix * lRayEnd_NDC;      lRayEnd_camera  /=lRayEnd_camera  .w;
   glm::vec4 lRayEnd_world    = InverseViewMatrix       * lRayEnd_camera;   lRayEnd_world   /=lRayEnd_world   .w;
 
 
   glm::vec3 lRayDir_world(lRayEnd_world - lRayStart_world);
   lRayDir_world = glm::normalize(lRayDir_world);
 
 
   out_origin = glm::vec3(lRayStart_world);
   out_direction = glm::normalize(lRayDir_world);
}
 

5. Detection routines

out_direction = out_direction*1000.0f;
 
        btCollisionWorld::ClosestRayResultCallback RayCallback(btVector3(out_origin.x, out_origin.y, out_origin.z), btVector3(out_direction.x, out_direction.y, out_direction.z));
        dynamicsWorld->rayTest(btVector3(out_origin.x, out_origin.y, out_origin.z), btVector3(out_direction.x, out_direction.y, out_direction.z), RayCallback);
        if(RayCallback.hasHit()) {
 
            int i =  (int)RayCallback.m_collisionObject->getUserPointer();
            printf("DETECTED index = %d \n",i);
        }
        else {
            printf("NOT DETECTED \n");
        }
 
 

So, on the screenshot (see below) you can see red area which is actual object that should be picked on clicking it, but it does not. Only when I am clicking inside green area I am getting satisfied results.

[attachment=27651:Capture.PNG]

Please help me on this.

PS
May assume there is an issues with my view matrix - but does not know what might be an issue.

projectionMatrix = glm::perspective(90.0f, 4.0f / 3.0f, 0.1f, 100.0f);


float horizontalAngle = 3.14f;
float verticalAngle =0.0f;/// problems here

glm::vec3 direction(
cos(verticalAngle) * sin(horizontalAngle),
sin(verticalAngle),
cos(verticalAngle) * cos(horizontalAngle)
);

glm::vec3 position = glm::vec3( 0.0, 10.0, 32.0 );

glm::vec3 right = glm::vec3(
sin(horizontalAngle - 3.14f/2.0 ),
0,
cos(horizontalAngle - 3.14f/2.0 )
);

// Up vector
glm::vec3 up = glm::cross( right, direction );

viewMatrix = glm::lookAt(
position,
position+direction,
up
);

Please explain this part.

It seems like you're trying to do too much at one time. You most likely need to comment everything out, and add in one piece at a time, verifying that it's actually working.

Normally, just pasting in code and asking "Fix this." doesn't get a lot of responses. We are all busy debugging our own code.

For the look at, I would start simple. Put a square at (0, 0, 0), backup 2 units, and make your lookat something like

eye ( 0.0f, 0.0f, 2.0f )

center( 0.0f, 0.0f, 0.0f )

up (0.0f, 1.0f, 0.0f)

If something that simple doesn't work then you went too fast.

thanks on replying on it. Actually putted like this and still having an issue, such a feeling that shape i am trying to put on my object got mirrored.

or ray direction calculation is wrong

projectionMatrix = glm::perspective(90.0f, 4.0f / 3.0f, 0.1f, 100.0f);

viewMatrix = glm::lookAt(
glm::vec3(0.0f,0.0f,2.0f),           
glm::vec3(0.0f,0.0f,0.0f), 
glm::vec3(0.0f,1.0f,0.0f)  
  );

I know something is wrong with my view matrix but no idea what exactly - maybe you can give some hints how to properly establish it in such a case ?

This

int mouseX, int mouseY, // Mouse position, in pixels, from bottom-left corner of the window

and this

((float)mouseY/(float)screenHeight - 0.5f) * 2.0f, // [0, 768] -> [-1,1]

is often something I stumble over. Try to mirror this to

 ((1.0f-(float)mouseY/(float)screenHeight) - 0.5f) * 2.0f, // [0, 768] -> [-1,1]

and check again. Does it work better ? The source of your problem could be your mouse position, it should be

- relative to your rendered screen (not absolut, check for border etc.)

- it should be aligned with the NDC (if the mouse position of the left,bottom corner is not 0,0, you need to apply a transformation like above).

Great, thanks pointing me in right direction.

It is fixed now

Actually fix was to multiple Y NDC on -1 - as you mentioned just to reflect fact of mirroring

((((float)mouseY/(float)screenHeight - 0.5f) * 2.0f) *(-1.0f)   ),

thanks again.