1 hour ago, komilll said:

However I found out, that it works only for location (0, 0, 0). Do you know what might be a problem?

When you refer to only working at the origin, I'm not sure if you mean it only works when the ray originates at the origin, or when the model is positioned at the origin. In any case, in TestOBBIntersection(), you don't appear to be factoring in the model position, which seems like it could be related. Also, I don't know the details of the math library you're using, but keep in mind the distinction between positions and vectors when applying transforms.

@Zakwayda I cleaned code a little and here is a result:

bool TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)
{
	//Create inverse world matrix
	XMMATRIX worldMatrix = XMMatrixIdentity();
	//worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixScaling(model->GetScale().x, model->GetScale().y, model->GetScale().z));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationX(model->GetRotation().x * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationY(model->GetRotation().y * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationZ(model->GetRotation().z * 0.0174532925f));
	//worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z));
	const XMMATRIX inverseWorldMatrix = XMMatrixInverse(NULL, worldMatrix);

	//Transform origin point and direction ray vector
	const XMVECTOR originTransformed = XMVector3Transform({ origin.x, origin.y, origin.z }, inverseWorldMatrix);
	const XMVECTOR dirTransformed = XMVector3Transform({ dir.x, dir.y, dir.z }, inverseWorldMatrix);
	origin = { originTransformed.m128_f32[0], originTransformed.m128_f32[1], originTransformed.m128_f32[2] };
	dir = { dirTransformed.m128_f32[0], dirTransformed.m128_f32[1], dirTransformed.m128_f32[2] };

	//Draw debug ray
	m_raycastModel->Initialize(m_D3D, origin, { origin.x + dir.x * 100.0f, origin.y + dir.y * 100.0f, origin.z + dir.z * 100.0f });

	//Call AABB test for origin, ray transformed to OBB-space
	return TestAABBIntersection(lb, rt, origin, dir, dist);
}

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	float pointX, pointY;
	XMMATRIX projectionMatrix, viewMatrix, inverseViewMatrix;
	XMVECTOR direction;

	pointX = x;
	pointY = y;

	m_D3D->GetProjectionMatrix(projectionMatrix);
	pointX = pointX / projectionMatrix.r[0].m128_f32[0];
	pointY = pointY / projectionMatrix.r[1].m128_f32[1];

	m_Camera->GetViewMatrix(viewMatrix);
	inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	direction = {
		(pointX * inverseViewMatrix.r[0].m128_f32[0]) + (pointY * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(pointX * inverseViewMatrix.r[0].m128_f32[1]) + (pointY * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(pointX * inverseViewMatrix.r[0].m128_f32[2]) + (pointY * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	const XMFLOAT3 dir = { direction.m128_f32[0], direction.m128_f32[1], direction.m128_f32[2] };
	return MouseRaycastResult{m_Camera->GetPosition(), dir };
}

XMFLOAT3 GetMinBounds(ModelClass * const model) const
{
	return{ model->GetBounds().minX + model->GetPositionXYZ().x, model->GetBounds().minY + model->GetPositionXYZ().y, model->GetBounds().minZ + model->GetPositionXYZ().z };;
}

XMFLOAT3 GetMaxBounds(ModelClass * const model) const
{
	return{ model->GetBounds().maxX + model->GetPositionXYZ().x, model->GetBounds().maxY + model->GetPositionXYZ().y, model->GetBounds().maxZ + model->GetPositionXYZ().z };;
}

Brief explanation - I got ray pure direction (not inverse, not normalized) from RaycastToModel. Origin is simply camera world-space position.
Then origin point and ray direction are transformed by inverse world matrix in TestOBBIntersection. lb/rt are equal to GetMinBounds and GetMaxBounds.

It still works for AABB in any position. OBB only for model position in (0, 0, 0). Camera origin (ray origin) might be doesn't matter.

Commented lines - scale is always (1, 1, 1) so that's identity matrix, we don't care. Uncommenting position multiplication makes even AABB not to work and OBB even for (0, 0, 0).

There's more there than I can easily sort out, but I'll point out one thing. XMVector3Transform() treats the input vector as a point (with w = 1), which probably isn't what you want for the direction vector. XMVector3TransformNormal() may do what you want there (in other words, use XMVector3Transform() for the origin and XMVector3TransformNormal() for the direction).

I suspect that may be an issue, but it may not be the only issue. There are many places that picking or raycasting (or any similarly complex geometric algorithm) can go awry, and there could easily be other issues at play here. But, I'd at least give the XMVector3Transform()/XMVector3TransformNormal() suggestion a try.

I can think of some other things that might make it easier to get this sorted out, but I'll refrain from offering them unless asked, as they're somewhat tangential and may not constitute the kind of feedback you're interested in.

@Zakwayda That's exactly where the problem was. Using XMVector3TrassformNormal for vector (not point) solved the problem. Thank you for all of your help. You're great!

For all of you looking for final solution:

XMFLOAT3 GetMinBounds(ModelClass * const model) const
{
	return{ model->GetBounds().minX, model->GetBounds().minY, model->GetBounds().minZ };
}

XMFLOAT3 GetMaxBounds(ModelClass * const model) const
{
	return{ model->GetBounds().maxX, model->GetBounds().maxY, model->GetBounds().maxZ };
}

MouseRaycastResult RaycastToModel(ModelClass* const model)
{
	//Go to [-1, 1] coordinates
	float x = GetCurrentMousePosition().first;
	float y = GetCurrentMousePosition().second;
	if (x > 1.0f)
		x = 1.0f;
	else if (x < -1.0f)
		x = -1.0f;

	if (y > 1.0f)
		y = 1.0f;
	else if (y < -1.0f)
		y = -1.0f;

	XMMATRIX projectionMatrix, viewMatrix;

	//Transform by projection matrix
	m_D3D->GetProjectionMatrix(projectionMatrix);
	x = x / projectionMatrix.r[0].m128_f32[0];
	y = y / projectionMatrix.r[1].m128_f32[1];

	//Create inverse matrix
	m_Camera->GetViewMatrix(viewMatrix);
	const XMMATRIX inverseViewMatrix = XMMatrixInverse(nullptr, viewMatrix);

	//Find ray direction in world space
	const XMVECTOR direction = {
		(x * inverseViewMatrix.r[0].m128_f32[0]) + (y * inverseViewMatrix.r[1].m128_f32[0]) + inverseViewMatrix.r[2].m128_f32[0],
		(x * inverseViewMatrix.r[0].m128_f32[1]) + (y * inverseViewMatrix.r[1].m128_f32[1]) + inverseViewMatrix.r[2].m128_f32[1],
		(x * inverseViewMatrix.r[0].m128_f32[2]) + (y * inverseViewMatrix.r[1].m128_f32[2]) + inverseViewMatrix.r[2].m128_f32[2]
	};

	//Origin is equal to camera position in world space;
	//Dir is transformed, not-normalized, not-inversed ray
	const XMFLOAT3 dir = { direction.m128_f32[0], direction.m128_f32[1], direction.m128_f32[2] };
	return MouseRaycastResult{ m_Camera->GetPosition(), dir };
}

bool GraphicsClass::TestAABBIntersection(XMFLOAT3 lb, XMFLOAT3 rt, XMFLOAT3 origin, XMFLOAT3 dir, float& distance)
{
	assert(lb.x <= rt.x);
	assert(lb.y <= rt.y);
	assert(lb.z <= rt.z);

	const XMFLOAT3 dirfrac = { 1.0f / dir.x, 1.0f / dir.y, 1.0f / dir.z };

	const float t1 = (lb.x - origin.x)*dirfrac.x;
	const float t2 = (rt.x - origin.x)*dirfrac.x;
	const float t3 = (lb.y - origin.y)*dirfrac.y;
	const float t4 = (rt.y - origin.y)*dirfrac.y;
	const float t5 = (lb.z - origin.z)*dirfrac.z;
	const float t6 = (rt.z - origin.z)*dirfrac.z;

	const float tmin = max(max(min(t1, t2), min(t3, t4)), min(t5, t6));
	const float tmax = min(min(max(t1, t2), max(t3, t4)), max(t5, t6));

	// if tmax < 0, ray (line) is intersecting AABB, but the whole AABB is behind us
	if (tmax < 0)
	{
		return false;
	}

	// if tmin > tmax, ray doesn't intersect AABB
	if (tmin > tmax)
	{
		return false;
	}
	distance = tmin;
	return true;
}

bool GraphicsClass::TestOBBIntersection(ModelClass* model, XMFLOAT3 origin, XMFLOAT3 dir, XMFLOAT3 lb, XMFLOAT3 rt, float & dist)
{
	//Create inverse world matrix
	XMMATRIX worldMatrix = XMMatrixIdentity();
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixScaling(model->GetScale().x, model->GetScale().y, model->GetScale().z));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationX(model->GetRotation().x * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationY(model->GetRotation().y * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, DirectX::XMMatrixRotationZ(model->GetRotation().z * 0.0174532925f));
	worldMatrix = DirectX::XMMatrixMultiply(worldMatrix, XMMatrixTranslation(model->GetPosition().x, model->GetPosition().y, model->GetPosition().z));
	const XMMATRIX inverseWorldMatrix = XMMatrixInverse(NULL, worldMatrix);

	//Transform origin point and direction ray vector
	const XMVECTOR originTransformed = XMVector3Transform({ origin.x, origin.y, origin.z }, inverseWorldMatrix);
	const XMVECTOR dirTransformed = XMVector3TransformNormal({ dir.x, dir.y, dir.z }, inverseWorldMatrix);
	origin = { originTransformed.m128_f32[0], originTransformed.m128_f32[1], originTransformed.m128_f32[2] };
	dir = { dirTransformed.m128_f32[0], dirTransformed.m128_f32[1], dirTransformed.m128_f32[2] };

	//Draw debug ray
	m_raycastModel->Initialize(m_D3D, origin, { origin.x + dir.x * 100.0f, origin.y + dir.y * 100.0f, origin.z + dir.z * 100.0f });

	//Call AABB test for origin, ray transformed to OBB-space
	return TestAABBIntersection(lb, rt, origin, dir, dist);
}

Glad to hear you were able to get it working ?