So i have a bounding box of an object (it can be rotated etc), now i wish to change the bounding box size - to do so i click on the screen near the box side and try to drag it, however i am unable to find proper side, it just feels like its random.

my apporach is that i first store the base of the bounding box (where its center is in 0,0,0), then after click i make a ray form screen to world and determine, closest point on that ray to the object position (center of the bounding box). then i perform polygon ray intersection (from box center towards closest point) this gives me the hit, it doesn't seem to work in my case (btw i take into account all rotations and translations)

here is the screen:

so maybe theres a better way to find the side?

From the description you gave, I think you are trying to select a face using the closest point on a ray to the bounding box? If I read correctly, I think that is actually what is causing your issue because the closest point on a box to a ray that is _not_ intersecting the box will almost always be on an edge and not a face. So in your case, it is likely that the face that gets chosen is simply a product of which face you check first that shares that edge.

If you want to be able to select a face without clicking directly on it, it would require a different approach. One approach could be to have a plane that passes through the center of the bounding box and is parallel to the screen(normal points at the center of the screen/camera), then intersect that plane with the ray you are casting from the screen. That intersection point could then be your closest point. You could still have some wonky behaviour with that though because the plane will always be intersecting edges, and still have a significant chance of selecting an edge.

Is there a reason you can't just ignore the close to the box part of your algorithm and just click on the face? it will be significantly easier to implement.

that ray plane intersection looks reasonable however to find the side i'll have to know how to 'rotate back' my object i assume multiplying rotation matrix by inverse of it wont give me identity ?, and no there will be no case when i pick the face i always hit the area around it.

looks like inverse matrix will do exact thing i want, coding...

yeah it looks like it works as expected now thanks for help