OpenSSL is a heavyweight solution for many coders. It's definitely worth looking at, if only for learning approaches on how certain algorithms are implemented. I found it a little complex and preferred to provide an abstract interface to cryptography for my library, with plug-in implementations, separating each algorithm and implementing using my own base code for math, etc.

As far as books are concerned, 'Applied Cryptography, 2nd Ed.' by Bruce Schneier (ISBN 0-471-11709-9) is my current 'light bedtime reading'. An excellent grounding in cryptography, and each algorithm is easily separable.

Have fun!

[Edit: I am drunk. I did not read Gosper's post re the Schneier book]