The first two videos where you are applying a force look like the force is too strong. I see similar behavior in my physics engine if the gravity is too high. Consider what happens when you apply a large force to an object. What the physics engine is probably doing each frame is:
- Set object force vectors to 0
- Add forces from gravity and user
- Integrate force to get change in velocity over delta time
- Detect collisions (with CCD or not)
- Solve collisions/penetration constraints (update velocity)
- Integrate velocity to get the new position.
The issue occurs because the (non-CCD) collision detection uses the previous frame's position to detect collisions. This means that if the box is almost touching (but not colliding yet), that there will be no collision and thus no collision resolution. This allows the box to penetrate into the wall/floor an amount proportional to F*dt/mass. On the next frame, the collision is detected and the solver tries to correct for the penetration. It might push the object out a little too far. If that happens, then you can get the same situation on the next frame where the over-penetration occurs. This is what causes the jitter.
I don't know what controls PhysX exposes, but in my engine I would reduce jitter by adjusting the speed at which penetrations are resolved. This is usually controlled by a parameter in the range [0,1] which represents what fraction of the penetration to resolve on the current frame. A value of around 0.2 to 0.3 is typically stable, more than that can be unstable and produce jitter. Maybe PhysX exposes a similar parameter. The term for this penetration resolution approach is called “Baumgarte stabilization.” In Unity which uses PhysX there is this parameter, though it hints that the penetration resolution is linear and affects velocity, which is not a good thing for stability. The method I described earlier is exponential (correction rate decreases over time proportional to the penetration).
You can also try turning on CCD and see if that helps, though it may not help depending on the implementation (e.g. if CCD is just between swept bounding spheres). To help it would have to be full CCD which accurately determines the time of contact (conservative advancement), and this is going to be slow.
As for the last video, it's hard to say exactly what is going on there. It might be a variation of the same issue - the top obstacle pushes the box down when it collides, causing overcorrection, and then on the next frame the bottom obstacle pushes back too far. At some point the push is enough to push the box center below the floor plane, which causes the box to pass through the floor. The result is that the box only collides with either the upper or lower obstacle, but never both at the same time. This prevents the box's motion from being stopped fully as it should.
However, it still seems a little strange, as if there are a (too) limited number of contact points that are solved per object (e.g. 2 instead of at least 4, or maybe the contacts are in the wrong place). One likely possibility is that PhysX collapses the contact point pairs to a single point for solving (instead of having a separate point on object A and B), this can produce incorrect motion if the penetration is large (point on A is far from point on B), if it's using the midpoint (p_A+p_B)/2 for solving collisions. I don't know why physics engines commonly make that simplification which makes no physical sense. It works much better to use separate points and doesn't cost that much extra compute to do so. Unity uses PhysX, and they only return a single point per contact.
Another common trick is to set all masses 1 to avoid some stability issues (e.g. stacking heavy object on top of light object), but I don't think that would help here because you only have 1 dynamic object.
Yet another possibility is that you have set up the center of mass and inertia tensor of the object incorrectly. If these are not realistic, then you can get a situation where objects don't rotate realistically when perturbed by collision.