The math is indeed linear algebra, vectors to be more specific.
You might find these lessons useful:
https://www.khanacademy.org/math/linear-algebra/vectors_and_spaces
The issue you describing seems to be a float to int conversion issue. Unfortunately you still haven't told us anything about the programming language you're using or what environment your working in. For instance, I can write for you all the C code I like, if you're not using C, it won't be of use.