And once you have a streambuf you can tie the input into the rest of the standard library pretty easily, especially with stream/streambuf iterators. Or even just using the insertion opreator to dump entire streambufs into a stream. This is just one of the many examples I've found where using the standard library interfaces as a common ground makes for very productive coding.
The current versions of the OStreamBuf and IStreamBuf classes are templated. I might go and create a non-templated version for the common > case. That should cut down on compile times. And after that I'll probably work on my sixth or seventh binary stream class revision.