I'm writing a game with RTS-like mechanics. It has a perspective view but from high up looking down and slightly forwards. I'm writing my own engine in C++ based on OpenGL (ES) 2.0 with glm. I want the user to be able to move around the map by dragging the terrain, but I don't know how to create the right effect:- as if a point on the terrain is stuck to the pointer or user's finger and follows it faithfully around the screen.

What I've tried so far is, at each motion event, "unproject" the pointer coordinates to ground level in world space [1], and compare that to the values read from the previous motion event or when the drag started. I then move the camera's look-at point by the same amount in the opposite direction. The trouble with this is that, even when working correctly, there will be some discrepancy because it doesn't take perspective into account: dragging a point near the top of the screen requires a larger camera movement etc. (I seem to have a bigger problem anyway: my terrain always moves considerably slower than the mouse).

The other issue is that I want to implement pinch-zooming on touch screens. Ideally the view should zoom and pan simultaneously to make both points on the terrain appear to be stuck to the digits used in the gesture. I can't think how to achieve that.

In both cases there are limits to how far the camera can be moved, which I would deal with by making the pointer/fingers appear to lose their grip, but regain it at the new point(s) under the pointer/digit(s) if the user then reverses the gesture in a way that takes the camera back away from its limit. I think that should be fairly easy once I understand the principles behind getting the basic motion correct.

I know the basics of matrix and vector arithmetic and I'm reasonably confident I can learn enough to understand how to do the above things properly, but I don't know what I need to learn, and I haven't been able to turn up suitable search results. Can anyone recommend some good reading material for solving these problems?

[1] To do this I use glm::unproject with viewspace z at -1 and 1 to get two points on the near and far clipping planes in world space, work out how far z = 0 lies along that line as a linear interpolation, and interpolate x and y by the same proportion. Is that correct?