Oh, and before you do anything else, ">watch this video. The Eastern Bloc has clearly been hiding some amazing music from us.
Building Documentation for SlimDX
One of the major pieces of work on SlimDX for the last release cycle has been documentation. We've written more of it than is humane, and I've been working hard on tweaking and perfecting the presentation. Like most things dealing with SlimDX, this turned out to be a hell of a lot harder than it should have been. I really wish I knew what Microsoft was doing with MSDN, because they can't possibly be struggling with the same problems I am and still be getting anything done.
Like many codebases nowadays, and nearly all managed ones, SlimDX docs start out as specially formatted doc comments embedded in the source code. The compiler reads off all these comments and produces an XML file alongside the output binary that encodes all of the documentation information. This part is a little obnoxious in C++/CLI, because the compiler has to look up and resolve all of the names referenced in the doc comments for cross linking. If it can't see the relevant identifier, it can't link it up correctly, and there are other problems with referencing specific members as well. These are all basically artifacts of applying the managed doc compile system to C++ code, and they make a mess of things. In most places we hack around it with forward declares or extra includes. Other than that insanity, though, this step basically works just fine.
The next step in producing documentation is to convert this XML file and the SlimDX DLL to actual readable documentation. That's done using a process of reflecting on the binary to find all of the types, members, etc and applying the correct bits of XML and XSL transforms to produce an output document. The format of the output document can be basically anything you want; just use XSL transforms that get you where you want to go. Microsoft provides a build engine for doing all of the reflection and transforms and everything called Sandcastle. Sandcastle is, for the most part, a pretty slick tool. Combined with a GUI to save you the annoyance of dealing with bat files and config files and junk, it produces some pretty neat looking documentation in HTML 1.x (CHM), HTML 2.x (VS), and online web format. I even patched the transforms to generate an extra little bit on pages which identifies the native equivalents to SlimDX types. Look at the SlimDX documentation for March 2008 as an example of the online documentation.
So far so good. Except for one detail -- this isn't good at all. The page loads slowly, because the index.html file is about 1400 KB. It uses frames, so you can't link to specific topics inside it, and the navigation frame doesn't update as you navigate. And as a final kicker, it took close to five minutes to build the docs each time, on my fast as hell work machine. I'm not willing to accept this level of quality, except temporarily while I make something better. So I've spent some time trying to make things better. My first idea was to try the other themes available out of the box with Sandcastle. There are three -- vs2005, Prototype, and hana. The one used currently in the SlimDX docs is vs2005. I tried hana, but it looks crappy and is even bigger. Prototype was much more promising; it compiled in around 3 minutes, index.html dropped by 300 KB, and the total size of all the HTML pages dropped to one third of the size of the vs2005 version. That was pretty good, with one caveat: the home page for Prototype was not displaying, either online in a browser or in the CHM file. It doesn't look good when the front page of your documentation is freaking blank.
After some web searching and a massive dose of good luck, I discovered Sandcastle styles. I initially though that it was a set of new styles for Sandcastle, rather than the ones provided by MS. That is not the case. It's actually a set of patches for the MS styles that fix their problems...including problems in the last version, released on May 30, 2008. Yes, that's right. Sandcastle's styles are so broken that people started another project just to make them usable. This is about where I began to realize that something is very wrong with the Sandcastle team and their product. (It's an amusing coincidence that Sandcastle's CodePlex site vanished while I was writing this.)
The patches did work, though, and allowed me to generate a fully working, non blank Prototype based set of documentation. Here it is. Except this hasn't solved anything. Thanks to expansion in SlimDX, the index.html file is back up to 1330 KB, and still loads slow. The layout is still frame based and you still can't link to specific topics. I've given up on the built in Sandcastle stuff at this point, because it's clear that it doesn't scale. Instead, I went looking for something that would handle all this more robustly.
I found DocProject. This thing is also Sandcastle driven, but does a lot of the heavy lifting by itself. It also integrates with VS 2005 or 2008. It also provides an external UI in case you're using the Express editions. One thing in particular caught my eye: Easily build an AJAX-enabled ASP.NET documentation website. Finally! AJAX based dynamic loading of topics, instead of a gigantic static list. This was clearly worth spending time with.
Right off the bat, VS 08 support doesn't work. It just crashes the IDE every time I try to build, on step 7. Fine, whatever; we just finished re-introducing VS 2005 support in SlimDX anyway. DocProject works with VS 2005, and creates a DocSite project (builds both online and CHM docs) using a project reference to SlimDX to do its work. I run a build, which churns away for a while, finishes building everything...and then freezes. It does this pretty much any time I do anything, just after finishing its work. It looks like the delay is in adding the HTML pages for the individual topics to the project, of which there are about 6500 currently. This process takes somewhere in the vicinity of an hour or two; I haven't had the patience to wait for it to finish instead of simply leaving for a while. Oh, and the external UI doesn't work, because it uses MSBuild and MSBuild can't understand project references to C++ projects. Fantastic.
Anyway, I can actually see the docs being produced by this thing, even if my iteration times are two hours or worse. This is all ASP.NET and AJAX hotness. Not bad. The HTML is even more compact. The navigation tree stays in sync with what you're looking at (doesn't autoscroll though). It allows you to save, print, bookmark, or email links to specific topics (but doesn't let you easily copy the link). Visually it's pretty snappy in default setup, and it can be reskinned like any other ASP.NET website. There's an admin page, index (which doesn't qualify names and so everything looks identical), and search that works well (but operates fully in memory and so mucks up the server somewhat). It extracts keywords (although it apparently picks up everything, including numbers). Okay, so it's not ideal. Still, we're getting much closer to the mark. Hey, it even has a space to put a logo in.
Now then, about those iteration times. I don't like waiting two hours between doing stuff. Like I said before, though, I think it's because all of the output HTML files are being added to the project file, which I just realized is optional. After turning it off...success! Things are fast! Thank god. So yeah, don't allow DocProject to add the Html directory to your project and everything will be fine. Builds are pretty snappy overall, and changing web page settings doesn't cause it to rebuild all of the docs. Unfortunately, its deployment option (which is supposed to copy just the files you need to upload out) doesn't actually work properly, so I'm building it partially by hand after the build finishes. I only need to do that when uploading though, which is not too bad.
Right now, I'm simply working on tweaking and tuning the current DocProject based documentation. It's an ASP.NET site after all, so there's a lot of configurability available. (Quite unlike the original Sandcastle stuff, which basically just offered a few dozen bool options.) These new docs aren't online yet due to some ASP.NET configuration issues on the server, but they should be visible soon, and I expect that they'll be the ones used for the June release. Here's a screenshot in the meantime.
However, the Sandcastle Help File Builder program you used a long time back ended up being pretty good for my needs (I'm only using the compiled help). I guess it didn't work out for you because of the Ajax thing?