I Believe in Static Checking II: Tests and Bugs
First, I'll address a comment on the last entry:
Quote: It's not that simple. You need a static type system or some other form of syntactic analysis if you want to verify your code is free of deadlock, for example. Dynamic typing can't do that.The trick is that static analysis can be run dynamically any time you want (barring limitations like needing source code). And a lot of these static analysis tools don't and can't make any assertions about the code, anyway. They merely attempt to detect obvious or common mistakes in code. They can't make an assertion as strong as "this will not deadlock" except for extremely trivial situations. They're still very useful tools, of course.
When it comes to application bugs, there are three basic points in time that a mistake can be detected:
- Build time
- Test time
- Runtime
When it comes to bugs, we don't want to find them at runtime if we can help it. If a bug pops up at runtime, it's far too late, and in many cases, may be quite difficult or tedious to reproduce and debug. Additionally, these bugs need to be triggered by a user; they will not make themselves readily known (except for severe mistakes). Worst of all, these are the bugs that are likely to slip out to customers. Finding bugs during the build or test stages is far safer and quicker. There's far less danger of an old bug silently reappearing unnoticed, and the problems are more readily reproduced. And when they're connected to the build system, you can pinpoint with considerable accuracy what it was that caused a test to break.
Ok, so now it's fairly obvious that post-build testing is a great idea. Why is it relevant though? As I mentioned in my last entry, testing is necessary for reliably findings places where constraints break, regardless of whether you have some static checking. Tests aren't perfect, though. They only check the code that they actually hit, and they only check that code for the values they're programmed to. Getting 100% coverage from your tests is near impossible, let alone running through all of the possible states that can occur for the code. For complex situations (for example, checking that your DAG based scene graph is correctly collecting and sorting all objects to be rendered, where correctly sorted is loosely defined), simply writing the code carefully and then doing QA and debugging is likely to be cheaper and easier than any testing method. (Unfortunately, this seems to come up in games a lot.)
Despite all those problems, testing gives us a lot of opportunities to sanity check code and prevent old bugs from reappearing. The truth is, tests are for all practical purposes nothing more than customized static checks. Whether or not our code is being run (in the case of tests) or simply read and processed (in the case of static analysis tools) is just an implementation detail. Suppose Valgrind could sandbox compile and run pieces of your code in order to check thread safety. Would that really change anything? Of course not.
So with this altered definition of static checking, it'd be pretty hard not to believe in it; not doing static checking would amount to developing software by accident. Don't worry, though -- I'm not about to leave you on such a pathetic cop out. I do believe in proper static checking and static typing, regardless of what kind of test framework you've got running post-build. In other words, I prefer to catch constraint mistakes at build time rather than test time. The reasons for that preference will be the subject of the next entry.
