Linear algebra tends to come in handy with both 2D and 3D games, as does trigonometry and geometry (and honestly, some basic physics stuff really helps as well). Though, your needs will really vary wildly from game to game spanning lots of math and physics disciplines. Depending on how poor your knowledge of mathematics really is, it may not be necessary to actually start from scratch and relearn everything (though that certainly never hurts, and I'm not saying you shouldn't do it). You could always simply brush up on whatever specifics you need to solve a certain problem. For most basic game-math needs, you're typically not the first person to have to solve it, and there are multiple online resources that can help you through them.
I find the "Math for Game Developers" youtube channel incredibly handy for some of the little problems. It's geared toward game related problems, however, you may find yourself a bit lost occasionally, and those math books you have will come in handy to fill some of the gaps. So, i guess I'm kind of recommending something along these lines (as an alternative to relearning everything) tackling your math problems one at a time, as you come across them.
I only recommend this sort of route because starting at the beginning and learning algebra, linear algebra, trigonometry and geometry can be a rather huge undertaking, and one you may lose momentum on long before you've learned everything you'd need. So, if you have a basic understanding, and are just really weak at it or have forgotten most of it (easy to do if you're not using it on a regular basis), simply brushing up on the specific solutions to certain problems can be a, if not good substitute, at least a much less time consuming one. There tends to be a handful of mechanisms that you'll use over and over with games. I'm personally rather weak with math as well, so have kind of just taken this latter strategy. Whlie it would be great to have a vast knowledge of mathematics, I honestly don't see myself simply reading math books for months on end without getting burned out on it.