I've read the DRI WIKI page at http://dri.sourceforge.net/cgi-bin/moin.cgi/Documentation and it seems that building the drivers also require a rebuild of the entire X server. Is it possible to build just the DRI drivers (the files to be installed in lib/modules/dri and lib/modules/drivers) without having to rebuild the whole damn thing?

AFAIK, you need the whole damn thing. The only way around this is to use the "bleeding edge" prepackaged stuff which used to work quite well back when I last used it.

DAMN