SO i want to do some kind of arcball rotation with usage of angles (pitch, yaw, roll)

however i need to convert screen coords to sphere coords,

so i make a ray from screen to world, and now i need to find proper coordinate on sphere ( the closest point that lies on sphere edge to the ray)

like in this pic: (top view)

Add two vectors to your eye ray to construct an orthogonal basis. A lengthy but straight forward calculation gives you what ever numericals you need. Consult numerous post on this site to check that your version of the formulas is optimal.

Sorry, but I don't have time to figure out the precise math right now. At global level, the simplest starting point seems to be the right figure.

Set up a line orthogonal to the eye ray. It must also go through the center of the circle (let's call the center of the circle C).
Compute the point where the orthogonal line crosses with the ray line. Let's call that point P.
Compute the distance between P and C.

If it is longer than the radius of the circle, you are in the right picture situation.
Compute the point at the circle at the orthogonal line then (from C towards P, with length the radius of the circle).

If the distance is shorter, you have the left situation. Let's call the points where the ray line intersects with the circle A and B.
P is in the middle of line-piece A-B.
Distance between C and P is known already. Distance between C and both A and B is the radius of the circle. Pythagoras will tell you the length of line-pieces A to P (and B to P which is equally long).
Finally, compute length of eye to A, and length of eye to B, and take the smallest distance.

yeah i did sphere ray intersection test if nothing was hit then i do closest poin on line test then get the fdirection from sphere center to that point normalize it and multiple by radius,

still however i have no idea how to use that data, the best thing would be if rotation matrix is identity and points -z forward +y up

now i will have to convert that to rotation i need to compute yaw pitch and roll angles out of these two points ( i reckon camera position will be in equation too. because it needs somehow to know where i hooked and where to i rotate, and then when theres rotation applied and i change camera position i can still compute points on sphere but how then apply that to modified yaw pitch roll, that confuses me as hell