I would like to explore a career possibility in game development and I am more interested in the design of the game engines. What would be a good start, (books, online courses etc) to get an idea if this suits for me? I have a strong background in math and physics (PhD - level).

When you say "design of game engines," that sounds to me like you want to be writing code, is that correct? Do you know how to program already? Have you tried to write a simple game?

37 minutes ago, jpetrie said:

When you say "design of game engines," that sounds to me like you want to be writing code, is that correct? Do you know how to program already? Have you tried to write a simple game?

By design of game engines I mean writing the code to describe the physics and the game environment. I know how to program, but I have not yet been involved in writing code for games. I found the book "Game Engine architectures" by Jason Gregory. Would this be a good book to get an idea of what steps are involved in game engine design?

It might be a reasonable book (I've never looked at it), but the most important thing is going to be actually trying your hand at writing some games that leverage the kinds of technology you want to write or work with. You can start doing that without reading any books, really. Think of a simple game concept and start building it using your preferred language and toolset.

I have a copy of Game Engine Architecture, but I have not yet had an opportunity to read it.  I bought this book on a solid recommendation from a friend.  This book is also a commonly recommended by poster on blogs and forums.  I would say that it is a good starting point for your ambition.  It may also benefit you to look into UDK.  The Unreal Engine is popular and has been used to make several successful AAA game titles.  I know there are several engines of which this statement could be made, but the Unreal Engine has one unique distinction.  The developers make the source code available.  This could be a great supplementary learning tool. You can get the source code for UDK here.  

https://docs.unrealengine.com/latest/INT/GettingStarted/DownloadingUnrealEngine/

Lucas, he cannot read the unreal engine as a means to try to start writing a game engine?  or at least not if he plans to make a commercial game engine for money ... read the license, you will see they give you no rights to use the code to compete with them selling any form of engine derived from their work - only games using and tied to their work.

He can only look at academic items and projects which do not have such restrictive commercial licenses.  But yes, he should look at existing engines and the few books that exist about them.

Also, you can read "books" and tutorials on unreal engine, unity engine, etc ... without caring about their copy write, but you cannot reference their actual code in the process of creating your own engine.

OP - What type of game engines do you want to make? physical simulations (realistic), space/flight sims, shooter style games, 2d arcade games, RTS games, etc.  The engines of these games actual differs quite a bit (although much less than it did 10 years ago thanks to more general engine advances).

I would however kind of agree with Lucas in that you should use existing engines before thinking to write your own (first of all because you may find you'd rather make games than engines anyway - secondly because you can't really know how to provide tools to a carpenter if you don't understand carpentry ... so you can't write game engines if you don't understand game writing at some level - just like you can't compose music if you don't understand how it may be performed - that doesn't mean you need to be good at making games, just some level of basic experience to inform your direction).

Also, when i say "use" a game engine, I mean as a consumer of the engine and its tools ... not its source code.

Advising to look into Unreal Engine source without any knowledge about game engine architecture is a bad advice. Even as experienced programmer there are more reliable ways for learning to walk without starting to drive a 250 HP car.

Game Engine architecture is a good starting point for a general overview to the tasks a game engine needs to solve and I also used GrandMaster's Playground Tutorial as a cross reference when starting to write my game engine from scratch.

This are almost good starting points, you also should read into the topics of data management, render pipeline (with modern graphics APIs), networking and general programming technics of your prefered language (especially to write well performing code) because this are the common tasks a game engine needs to solve without getting to specific.

Another point is the kind of engine you want to write. An RPG or FPS engine might be a simpler task as an all-in-one solution that Unreal/Unity and other AAA title engines benefit to be because you would need a completely different tools and editor base (and of course management code on the engine side) where products like Unreal also intend to add certain level of security e.g. on encrypting assets to rptoect games made.

There are also plenty of differences on the architectural approach to sales models. Unity itself is closed source and single threaded on the user level what makes its management to user code more specific than Unreals almost open source model. The second aspect is platform compatibility, there are several options to platforms a game engine could support, especially in those all-in-one solutions what leads to a higher level of abstraction to offer a general interface but under the hood use highly platform dependent APIs.

Just as some basic thoughts

If you still want to look at source code then you should instead tale a look at some community driven projects for example Urho 3D or Crown

I haven't used it, but you can start with Godot:
https://godotengine.org/

It's a free and most importantly open source game engine.

You can pick it apart, and make a small project to test things. Since it's open source, if you make some amazing physics feature, you can inform the team and they may put it into the engine for everyone to use.
Then you get kudos for that. On your resume or website or whatnot.

I wouldn't start with building your own engine just yet since you need to figure out the beast that you're dealing with first. And that means looking at what's out there already.

And other game engines: https://www.slant.co/topics/1495/~3d-game-engines