Why do NVIDIA drivers force one to use X? What if one, for example, wanted to 1) have only a fullscreen app and not lose any performance because there was a windowing system running, or 2) build an alternative to X on top of OpenGL?

Well, both reasons are kind of marginal cases. Nvidia isn't in the business of making nine individual nerds happy. They're in the business of shifting silicon at minimal additional cost.

It took long enough for them to properly support Mac & Linux -- they originally used to have the position that 10% of computer users didn't represent a big enough market for them to be bothered about.

You're asking for entire new driver stack (with all the associated maintenance overhead) but without a large enough demand to make that worthwhile.

They're X-drivers after all. If you want OpenGL without X, see if you can hack the Nouveau drivers. Or have a look at Wayland, an alternative to X.org.

Nvidia's and Ati's closed-source drivers are built around X11, no way to work around that. Without a WM, the X server installs in a few MB and starts up very quickly. I've been using this approach for an emulator-centric distro and it works great!

The open-source drivers (Nouveau/Radeon) now work without X11. You can use EGL to create the OpenGL context but I *think* that right now you are limited to the OpenGL ES subset of the specs. (Desktop OpenGL should become available later on.)