I'm working on some basic AI for a fighter game, and I'm trying to get the ships to fly into a formation. To calculate the amount of torque I need to apply to each ship so that it turns towards the desired heading I'm using a PID controller very similar to the one in the first answer here: http://answers.unity3d.com/questions/199055/addtorque-to-rotate-rigidbody-to-look-at-a-point.html

So, I have the current heading of the ship, and the desired heading, and I need to pass an error value into the PID controller so that it can correct the heading.

In the linked example, the poster uses the cross product of the desired heading and current heading. This works perfectly if the angle between the headings is < 90 degrees, if it's greater than 90 degrees then the cross product is flipped and the PID controller starts correcting to the opposite direction than I want!

I'm struggling to figure out what vec3 error value will work in all cases, even if there is a large angle between the desired and actual heading. Any ideas?

Could you use a dot product in the forward (relative to the ship) and the desired heading, and if it's less than 0, flip the sign?

You are wrong, the cross product between headings changes sign at 180 degrees (i.e. it correctly chooses the shortest option between turning clockwise or counterclockwise). What happens at 90 degrees is that the cross product magnitude reaches its maximum; and correction for really wrong headings is smaller than for moderately wrong ones. There are two obvious solutions:

1. You can use atan2 instead of the cross product to obtain a real difference of angles rather than something that only has the right sign.
2. If you also control position (chasing the target, not only turning towards it) the resulting thrust (to turn back) would compensate the reduced torque when the heading is away from the target.