The main problem is that Bullet debug drawer has never meant to be used for massive drawing, but rather to draw simplified debug tests cases.
Consider drawLine. This is an extremely fine-grained control.
The issue was recently pointed out on Bullet forum as well. It is basically the same of using immediate mode VS buffer-based rendering.
Except it's actually worse because even if you have empty implementations doing NOP thus saving the drawcalls themselves, you still end up calling drawLine thousands of times to do something which could be pulled out in a single shot!
Compared to standard function calls, indirected function calls are not much slower. Problem is function calls are damn slow by themselves compared to optimized code. In my experience, breaking a piece of optimized code with a single function call can make it easily 20x slower. If this is a high-frequency, hot piece of code, such as the case of a renderer, we're going to have quite some issues with it!
So what do you do to fix it?
Nothing, you cannot. This is expected behavior.
Personally I wish bullet could just pass us information about what to draw and where (draw this collision shape here), just as the case for DrawSphere, DrawCylinder, DrawCapsule... Looks like heightmap terrains don't have this right to exist which is a bit of a shame but understandable to a certain degree.