As a practical matter (that is: because you want to write your own complex rendering engine) you probably shouldn't learn either. Nowadays there are plenty of very capable, very inexpensive (and mostly free-to-start) game engines such as Unity or Unreal Engine 4. If your goal is to get products out the door, whether you're an experienced team or lone newbie, one of these options (or the many others) is your most expedient and cost-effective route. Many major studios today (perhaps most) license their engine technologies from others because they don't have the experience on staff, can't afford the effort, or can better use the resources they do have elsewhere; or because they can't take on the risk of an unproven engine, or because even if they succeed they can't just hire new people who are experienced in their proprietary technology -- in short, even game studios have a hard time making the economics of it work out.
Especially if your math/graphics background is shallow, as you say, then what you might benefit from going directly to OpenGL or DirectX3D (regardless of version) is beyond you, and will likely remain so for years. OpenGL or Direct3D are not the language you want to be speaking when what you really want are recipes for getting things done -- they're the wrong level of abstraction for reaching that end. They're the language you speak to write rendering engines, they're not the language you speak to write games.
If you're doing it to educate yourself or you're doing it to satisfy a curiosity, then by all means go right ahead. Just be aware that by all practical and tangible measures, rolling your own complex engine is almost certainly foolhardy. For my money, rolling your own engine is really only viable when you know that existing solutions don't meet your needs, when either A) your needs are actually so straight-forward that the complexity or philosophy of existing solutions would work against you, or B) your needs are actually so complex/novel/stringent that no existing solution meets (or could reasonably be made to meet) your needs.
I say all this somewhat begrudgingly as someone who takes pride in building my own solutions, and having extremely high standards for solutions I'd simply adopt.