Keep in mind that game development is sort of a two layer process.
If you're a programmer then it's like learning to paint, whether you go to school or teach yourself you are essentially teaching yourselves about the process of painting, you have to learn the way the paint functions, what tools and different brushes you have, how you might remove spilled paint or what have you. That's what they teach you in school if you go for a degree, you end up with useful knowledge about how the thing works on a fundamental level and also get practice in being forced to paint in different ways you might be uncomfortable with.
Game development on the other hand is a job, it's like taking that painting knowledge and now going out and painting murals for schools or a church or making art to hang on a wall, it could even be something like painting a house. All that knowledge about using the tools will come in handy but you also now need to learn specifics about your profession, tips and tricks and what the standard way of doing things might be in your field.
Arguably learning with a library can teach you more about the low level of how the things work but it makes it very difficult to learn higher level techniques used in modern development without you advancing to the point where you are making very difficult projects that require you to look up and teach yourself the relevant techniques. On the other hand a game engine like unreal or something tends to shove these techniques on you more upfront because they are built into it. It's much easier to learn what materials are and what their uses are from unreal or unity than it is to learn by making things from scratch in opengl or direct3d, even if the latter will give you a better understanding of how the thing is actually implemented in code. That applies to other things too: 3d animation, level building, AI path finding and scripting, event triggers. You get a more "industry standard" look at some of the tools people use.
Boiling all that into a few words, I personally found that it helped a lot to learn both things. Try a few languages, try a few libraries, try a few game engines, you'll learn different things from each of them, even just experimenting a bit.