I implemented a simple AABB vs Sphere collision using this method from gamedev. It works fine when the sphere collides with the box from all sides.

The problem is that the sphere will not slip of the edge of the box. Meaning when the sphere falling in -y direction collides on the edge of the box, it will just go in +y direction again. See the picture below (taken after collision).

I think the problem is that the contactnormal is in this case always [0, 1, 0]. How do I achieve a more realistic behavior? For vector calculations I am using glMatrix.

var delta = vec3.create();
vec3.subtract(delta, centerRelSphereAABB, closestPoint);
distance = vec3.length(delta);

if(distance < C.radius){

   penetrationDepth = C.radius - distance;

   contactNormal = vec3.create();
   vec3.subtract(contactNormal, delta, A.centerOfMass);
   vec3.normalize(contactNormal, delta);

   //contact point
   contactPoint = vec3.create();
   vec3.scale(contactPoint, contactNormal, C.radius - penetrationDepth);
   vec3.subtract(contactPoint, model_1.centerOfMass, contactPoint);
}

The data is then forwarded to my collision response.

[attachment=23468:Clipboard02.jpg]

Unfortunately, it's not clear exactly what your variables represent. For instance, is "closestPoint" a position in world coordinates, or a position relative to the box or sphere ...? Is "centerRelSphereAABB" relative to the box or sphere, rather than in world coordinates?

It appears you have some of the pieces of the calc, but, without knowing whether they are in relative coordinates or world coordinates, I can't help fix the code you posted.

However, the principle (pseudo-code) to determine the force on the sphere, assuming the box is rigid and the sphere flexible (bold variables are vectors, non-bold are scalars):

1. determine the collision point in world coordinates - e.g., P (IF your "closestPoint" is in world coordinates, that's P)

2. create a vector from the collision point P to the sphere's center of mass in world coordinates (which is in the direction of the force on the sphere) - e.g., F_D = S_cm - P
3. calculate the penetration depth D = SphereRadius - F_D.Length()

4. the force on the sphere is D * SphereResiliency * F_D.Normalize()

With that force determined, the response of the sphere depends on your collision response routine.

EDIT: If you need the box normal at the point of collision to calculate the torque on the sphere, you'll need to calculate that in the sphere-box collision routine.

Ok, sorry for that.

(bold variables are vectors, non-bold are scalars):

centerRelSphereAABB - center of the sphere relative to the center of the box

closestPoint is the closest point on the box to the center of the sphere

---> So I can do a simple distance check for the collision

I translated my code into pseudo code, I hope it is much more readable now

€: bold is not possible in code format styling

delta , contactNormal , contactPoint

delta = centerRelSphereAABB - closestPoint;
distance = length(delta);

if(distance < Sphere.radius){

   penetrationDepth = Sphere.radius - distance;

   contactNormal = delta - Sphere.center;
   contactNormal = normalize(contactNormal);

   //contact point
   contactPoint = Sphere.center - contactNormal * (Sphere.radius - penetrationDepth);
}

It appears you're mixing local-to-box coordinates and vectors with world coordinates and vectors. Apples and oranges. It's kind of a mess.

First: be mindful that there's a difference between coordinates and vectors. Yeah, they're normally represented with the same structure (such as Vector3). However, vectors are a direction relative to some origin. Coordinates are positions relative to some origin. Note the emphasis on "relative to some origin."

Repeating a question not answered: "... is 'closestPoint' a position in world coordinates, or a position relative to the box ... ?"

IF you're using local-space coordinates and vectors, you must convert them to world-space correctly, assuming you want the sphere's response to be relative to it's position in world-space.

It might help if you take pencil and paper and draw the vectors relative to their origins. For instance:

IF closestPoint is relative to the box center, for instance, your delta is a vector relative to the center of the box (box origin.) You then subtract ?? (what I assume are world coordinates relative to another origin) Sphere.Center to get contactNormal. In words, contactNormal points from the center of the box to some point in the opposite direction of the center of the sphere, with a little offset (delta).

It appears your calculation to determine if the sphere intersects the box is okay. I would then suggest you work in world-space.

EDIT: Sorry for the late edit. I'm still working on my first pot of coffee and making mistakes.

Looking at your equation for contactPoint,

You already have delta which points in the direction of the closest point to the sphere center. The distance is, indeed, the length of that vector.

So, perhaps, contactPoint (world coordinates) = Sphere.Center - delta. That's in world coordinates, and is some position inside the sphere

Sorry for the late reply.

You were right. There was a small bug mixing up local and world coordinates.

Also I made a bug in calculating my bounding boxes which lead to an offset of the bounding box in x direction :D

Now everything works fine :)

in case of box colliding with sphere I just changed the contactNormal and contactPoint to this

negate(delta);
contactNormal = normalize(delta);
contactPoint = Box.centerOfMass - delta;

The behaviour looks right to me, but I am not 100% sure.

Re: contactNormal - depends on what you're trying to simulate and what you're going to use it for with regard to collision response. What you've calculated is the surface normal of the sphere at the point of contact. More commonly in a collision engine, the contact normal reported for the collision of (e.g.) object1 is the surface normal for the "other" object, object2, and can be used for friction and torque calcs.

As mentioned, it depends on what you're trying to simulate. If your intent is to provide a relatively close approximation of the situation, you're probably fine. Your system appears to be time-based, i.e., you check for collisions at intervals, which is a common approach. Just be aware that, as such, there is some degree of inter-penetration (which you recognize) and there is actually an "area" of contact, rather than a "point" of contact. If you use small increments of time, and the velocities of the objects are small, your method will likely be okay, particularly for spheres and boxes.

Note: as mentioned, you need to be mindful of the origins for relative coordinates. You mix center-of-mass with coordinate-origin which, for uniform boxes and spheres, will work. If you get into more complicated situations, such as centers-of-mass not coinciding with coordinate origins or different shapes, you'll get results which are not quite so good.