How do you make "brush doors" like in Quake 3? I have my own convex hull map editor.

Usually, you'd tie the set of brushes to an entity such as a func_door -- an entity is created and the brush data is attached to the entity. See Valve's MAP format for more information. It's not Quake, but if I recall correctly, the map formats from Quake games and Half-life games are very similar.

There's two positions for func_door as I understand it. How does it work though? Would the planes of the convex hull be rotated around the door axis?

Typically, a func_door has two properties: An origin property, where you can specify the point at which the geometry rotates (the "origin"); and the axis property, the axis the door rotates around. I'd assume the process compiles the brushes into a mesh (or meshes), with the origin point as the local origin. Sure, you could rotate the planes, but the you should really be working with the mesh directly...

But then you can't test for collisions against the convex hull

But if memory serves, func_door only translates in ID games. I haven't been mapping in a while but if I recall correctly, they move along an axis by their own size, minus an offset specified in the editor.

How do you implement it in your engine?

First this brush is non-static. In physics terms, it is a kinematic object. You have to figure out if this non-static brush can live in the world representation together with the others (which will often be static). In my case, the answer was NO, because the whole world collision geometry has a single name in my system (although it can have multiple collision ids), making it inherently non-scriptable.

So, it lives in the "scriptable geometry pool" in my system - you could do different things.

Now, I don't have a func_door functionality myself but I have a gameplay-independent scripting system instead. This is a fairly advanced solution and I'm not sure you want to go in that direction. Just looking for a string ID might be preferable in your case. When you find this string, you spawn a collision sensor around the original door brushes. Doing that might require some care. No, you don't use the mesh directly: the door must start to open as you approach, not after you collided with it.