Here's my dilemma..... I would like to use a physics engine but I'm not sure if it's practical for my project.   What I need currently is fairly simple. I need mesh collision and response with a pill shaped object (i.e. a character).  The thing is, I build my geometry at run time, and it goes straight into an octree.  It's actually built after I figure out where the character is going in kind of a "Just In Time" fashion. Also, it's my own custom mesh format.  I'd rather not take my mesh and put it in some 3rd party format because basically everything I need is already exists, i.e. faces, edges, vertexes, face normals and the octree.  So I'm wondering if there is an engine that will somehow let me use my own octree and for instance let me register callbacks to pass in the mesh data as needed.

The Bullet physics engine can be used as a collision detection library directly. Unless you're making a strategy game, I wouldn't worry about the performance cost of using the whole engine, but there are many benefits to writing custom physics for each game. It's what makes the gameplay feel truly unique compared to generic games.

A simple start for learning the basics of game physics would be a non-rotating capsule actor on a height map with voxel buildings. Get the mid-point position for collision detection by adding half a fixed step of velocity movement on the current position. This will represent the average position during the journey from start to end of the step to remain stable. Heightmaps can get away with just a few height interpolations along the feet and an obstacle map telling where it's impossible to climb. A capsule against a box without rotation can be simplified as a sphere colliding with a higher box. A sphere against a box finds the contact point by clamping the center of the sphere to within the box and comparing the distance with the sphere radius. Then you combine the contact points from multiple boxes and reduce the ones that are too close in angle. Then apply some of the impact to the velocity and some directly to the position as spill energy. Apply the physics in fixed steps for determinism and interpolate linearly using position and velocity to get the displayed location.

When walking with legs, make sure to adjust the relative velocity within the friction to adjust for walking motion. You don't want to feel like you're dragged against the floor pulled by a rocket.

More advanced collision detection can use convex decomposition into convex polyhedrons. A convex polyhedron is defined as a set of infinite planes cutting away matter from an infinite initial set. The direction of each plane can be represented using exact integers by not forcing it to length one.

24 minutes ago, Dawoodoz said:

The Bullet physics engine can be used as a collision detection library directly. Unless you're making a strategy game, I wouldn't worry about the performance cost of using the whole engine, but there are many benefits to writing custom physics for each game. It's what makes the gameplay feel truly unique compared to generic games.

A simple start for learning the basics of game physics would be a non-rotating capsule actor on a height map with voxel buildings. Get the mid-point position for collision detection by adding half a fixed step of velocity movement on the current position. This will represent the average position during the journey from start to end of the step to remain stable. Heightmaps can get away with just a few height interpolations along the feet and an obstacle map telling where it's impossible to climb. A capsule against a box without rotation can be simplified as a sphere colliding with a higher box. A sphere against a box finds the contact point by clamping the center of the sphere to within the box and comparing the distance with the sphere radius. Then you combine the contact points from multiple boxes and reduce the ones that are too close in angle. Then apply some of the impact to the velocity and some directly to the position as spill energy. Apply the physics in fixed steps for determinism and interpolate linearly using position and velocity to get the displayed location.

More advanced collision detection can use convex decomposition into convex polyhedrons. A convex polyhedron is defined as a set of infinite planes cutting away matter from an infinite initial set. The direction of each plane can be represented using exact integers by not forcing it to length one.

Actually several years ago, I implemented a sphere to mesh collision and response that worked pretty well.  But I'm guessing my math was kind of a hack job compared to a real physics engine.  I want to use something that's fast because I need the CPU cycles for run time terrain generation.

If I understand you correctly I can just call bullet with something like a movement vector, a pill shape, and the relevant mesh faces and it should do what I need?

Newton also has support for this with user collision callbacks (i remember somebody used this for minecraft voxel world).

But if all you want is capsule vs. terrain, i guess it's better to do it yourself. If you want generic shapes, vehicles, joints etc. later, i would use physics engine.

39 minutes ago, Gnollrunner said:

If I understand you correctly I can just call bullet with something like a movement vector, a pill shape, and the relevant mesh faces and it should do what I need?

You'll usually not use faces, because they have no volume, but that's essentially the idea. Just a set of functions responding to how shapes interact. If it doesn't fit your exact needs, you can always copy the source code and modify using class inheritance.

10 minutes ago, Dawoodoz said:

You'll usually not use faces, because they have no volume, but that's essentially the idea.

I really only have faces, edges and vertexes though, but I can guarantee I can pass in all the relevant geometry for a given movement vector. I guess I should pass the edges and vertexes in separately to keep those collisions from being calculated more than once.  If remember correctly you need to do collisions on all three.

The best is to convert your level geometry into a set of convex polyhedrons using a convex decomposition algorithm (for items) or construct solid geometry from boolean geometry operations (typical level design). Let each plane shrink along it's normal to compensate for the tolerance radius being added to convex hulls in Bullet.

Thin faces will cause small items under pressure to penetrate the surface or get stuck inside. Height maps in Bullet are using thin surfaces, but in that case you can force the center of each tiny item to be kept above the ground level by sampling the height yourself.

12 minutes ago, Dawoodoz said:

The best is to convert your level geometry into a set of convex polyhedrons using a convex decomposition algorithm

... would be much too slow for him i guess.

This is the Newton interface:

	    //  ***********************************************************************************************************
    //
    //    User Static mesh collision interface
    //
    // ***********************************************************************************************************
    NEWTON_API NewtonCollision* NewtonCreateUserMeshCollision (const NewtonWorld* const newtonWorld, const dFloat* const minBox, 
        const dFloat* const maxBox, void* const userData, NewtonUserMeshCollisionCollideCallback collideCallback, 
        NewtonUserMeshCollisionRayHitCallback rayHitCallback, NewtonUserMeshCollisionDestroyCallback destroyCallback,
        NewtonUserMeshCollisionGetCollisionInfo getInfoCallback, NewtonUserMeshCollisionAABBTest getLocalAABBCallback, 
        NewtonUserMeshCollisionGetFacesInAABB facesInAABBCallback, NewtonOnUserCollisionSerializationCallback serializeCallback, int shapeID);
    NEWTON_API int NewtonUserMeshCollisionContinuousOverlapTest (const NewtonUserMeshCollisionCollideDesc* const collideDescData, const void* const continueCollisionHandle, const dFloat* const minAabb, const dFloat* const maxAabb);

Seems it could be used as intended but i'm not sure. I assume Bullet has a similar interface too.

Hmmmm? You guys are convincing me not to use Bullet.   I guess I'll revisit my old code.  I had this working without all the issues, at least after I debugged it. For instance, I know you can sweep a sphere and do a pretty much perfect collision and response against a mesh. The good news is this thread was useful for me. ? Thanks for your input guys.

3 hours ago, Gnollrunner said:

need mesh collision and response with a pill shaped object

Pill shapes is second after spheres by both efficiecy of computation and ease of implementation (It just use  a two sphere tests and one of infinite length cylinder test under hood), but have much better fill factor than spheres. So any phisical engines around suppport it shape ever with GPU acceleration in case it uses PhysX under hood. It collider shape is widely used  for both approximate collider that consist whole character body inside as well for each character body parts like leg, arms, foots and ever fingers.  Also for body parts ellipsoide collider can be a little bit better option becouse it have better fill factor than capsule, but it  little bit more triky to implement  collision computation (require to scale objects to convert ellipsoide to sphere, perform sphere test with scaled objects, and then scale results back), so usualy it used into real-world robotics only (at least i never seen a game-intendent physic engines that support ellipsoids).