It is encouraging having a newcomer ask a question that DOES understand the amount of work and manpower needed to create anything close to AAA games!
But coming to your questions:
When it come to engines, personally I would pick either Unity or Unreal Engine 4. Both of them are quite capable of doing anything up to AAA games, both of them have large communities and attractive Indie options.
There are differences, but technically they have come pretty close. It will take until the release of Unity 5 for Unity to teally close the gap to UE4 (due out this spring or summer), by then it would be pretty hard to find anything you can do in Unreal Engine 4 you cannot do in Unity 5. Unity has come a long way from its umble beginnings....
Now, if I had to name the differences and strengths of both engines, they would be:
- Performance: IDK about UE4 performance really, but given epics track record and Unreal usually being picked even for AAA games, I guess they should be on par with some of the best engines out there. Unity made a lot of progress there in the last years, and with Unity 5 a lot of old baggage in the engine core will finally be replaced and updated (lighting engine (true deferred renderer, yay), 64bit editor, and so on)... still, if you need to squeeze out that last bit of performance, UE4 most probably will keep a slight edge.
That is not to say you as a beginner will need all this performance...
- Pricing: UE4 can be had for as little as 20$, plus 20$ every time you need an update (its a monthly sub, so you just pay for a month and then cancel the sub). Unity has a free version.
However, Unity free has some features cut (again, nothing that should be of any concern as long as you work on project appropriate to a beginner), while UE4 comes featuere complete and WITH SOURCE for 20$ (not that you could do much with the source as a beginner anyway). The Unity Pro version, which is feature complete (but without source) costs 1500$ per seat.
And the last thing is: UE4 asks for 5% royalities for anything you earn over 50k$... Unity is free of any royalities.
So which pricing model is better depends on what you need. Want to create a smaller, simpler game and expecting to make some bucks with it? Unity will be the better bet. Want to play around with AAA quality stuff without intending to make any serious money with it? Pick UE4.
- Community and Stock Assets: Here Unity has the clear edge. While UE4 comes with a suprising amount of stock systems and assets that semm to be really good, the amount of thirdparty stuff available for Unity through its asset store is staggering. Some of it rivals systems created for big AAA games both in look and performance. Most of it is not for free, but as long as you can spend some bucks and only need one or two plugins, systems or assets to kickstart your project, the prices are quite amazing for the quality you get. Most stuff is below 100$, and some quite amazing stuff can be had for 5-10$!
Also, the Unity community is large, and lots of tutorials are available, as well as books, youtube vids and all. Unreal has catched up a lot since entering the Indie world with UDK, but I still think Unity has an edge here. After all, Unity was already taking over the Indie space as engine of choice while Unreal still was experimenting to get the formula right (UDK was amazing in its own right, but 25% roylities were a major let down for any professional Indie)
Both Engines seem to attract large communities, but Unity clearly had a head start. Especially their Asset store is amazing.
- Editor: If there is one thing Unity DID get right from the start (the engine core was up to Unity 3 not that cutting edge), it was recognizing that the Editor should be a central place where devs work in. You will find lots of programers that disagree with me here (but then again, lots of programer follow the "the code is obvious" philosophy just because they a) to lazy to comment or b) want to make it hard for newcomers to understand their code ), but that was a revelation in a time when lots of engines still had quite wonky editors, with these "editors" really consisting of multiple tools. Unity had from the beginning one slick "tool to rule them all". It is what enabled nonprogrammers to get quite far before they had to either learn to program or hire a programer.
Now, the UE4 editor also looks quite good, and seeing that it brings some systems out of the box that need to be added as third party plugins in Unity, and me not having handled the Editor GUI myself, do take my opinion with a grain of salt. In the end, both Editors follow a similar strategy, and seem to be quite comparable.
The editor of both engines are most probably tied for usability and features. But lots of other engines cannot compare to these editors in the Indie space.
Now, let me tell you this:
1) Unity is a very capable engine. Do not listen to people that trash talk about it, or try to interpret the lack of AAA games as a sign that it lacks features or performance. Most people trash talking haven't tried it or just briefly, and don't really know what they are talking about.
And the reason why AAA compnaies do not pick Unity for AAA games is simple: if you can spend 50 millions on a game, 500k$ for a full license to a toptier commercial engine is nothing. If you buy that, you will most probably get an unlimited amount of seats. To my knowlegde, Unity has no such offer at the moment, still trying to find its niche with Indies and smaller companies.
Now, if you need for example 200 seats, Unity Pro will already cost you 300k$.... so the pricing is no longer looking so attractive anymore.
Then there is the simple fact that Unity hasn't really been an alternative to toptier engines long enough (it only started being competitive with Unity 4, and will reach almost-parity with 5) to make a name for itself... Unreal on the other hand has been around for 10+ years, Cryengine had some strong games to make a name for itself (all the Cry-something titles), the same can be said about Frostbite and others.
Just look at Unigine... it seemed extremly capable for a while, and seemed an offer to good to be true. But between the DX11 Bubble never really taking off, and other better known engines catching up, Unigine seem to fallen to the wayside even though its price was like a tenth of the bigger commercial engines.
If you want to get proof that the AAA companies are slowly recognizing Unitys worth, look no further than Blizzard. Hearthstone was made in Unity... yes, it is one of their smaller projects, and Unity was most probably picked because a) not so many devs were working on the project, and b) lots of youngsters nowaday learn Game dev with Unity, so if you start a project with newcomers, Unity can be good choice... but that is again just... like... my opinio, man
2) Never let anyone tell you "you don't need to know any coding". It is simply not true (sorry, Tangletail, but it has to be said).
The Blueprint Visual scripting tool integrated into UE4 sounds amazing, and to get similar tools in unity you have to spend 40-50$ in the asset store.
But none of these tools will do much when you have zero programing knowledge. You don't need to know the syntax of the language the engine uses for scripting anymore (C++ for UE4, C# and JavaScript for Unity), and true, a leveldesigner or artist might be able to drag together a usable small script to get small things in the level moving (like opening a door with a trigger or somesuch)...
In the end, somebody needs to program the heavy lifting game logic, and this person will need some very good insight into programming concepts, no matter if you write C++ code or drag the stuff together with a Visual Scripting tool.
In response to Tangetails post above:
The visual scripting part, which looked a little bit too positive to me, I already touched on.
The other points are quite valid from my point of view, but:
1) true, the Asset Store CAN be a dangerous place. That is, if you buy stuff without thinking or reading the reviews. I buy lots from the store because, you know, I happen to have a well paid job and game dev is my expensive hobby
I have been burned exactly twice from all the dozens of assets I bought over the years. One was an asset that was actually quite helpful, got my battle and shooting system to where it is today... but it was horribly coded, and the price, in hindsight was ridicolous. It was valuable for me when starting the project as I wasn't sure how to tackle the projectile physics. But I soon took the general concept, and rewrote all from scratch in C#, while improving all the details the original dev did so horribly wrong!
Did I pay too much? Yes, even though it helped me in the end, I had to recode everything as the quality was so poor. Would I have been able to avoid it? I though it was a new asset, so the lack of positive review seemed to be just because of that at the time... fast forward a year, it still has a single review that is not that positive. That means one of two things: a) nobody is buying it or b) people don't like it but do not want to give bad reviews.
In the end, if you look at the review part of the asset, and it has an incredible amount of happy customers that say the dev is really helpful and responsive, just buy it. If there are not that many reviews, or lots of negative ones, be wary. Also, most devs have an open forum topic for support requests. Visit it, ans see how active the dev is in answering questions and how many people are aksing questions. If its a lively place and the dev is very active, you most probably have nothing to fear.
The second time I got burned I bought a really good asset. But a year later, when I needed support, I had to find out the dev has abandoned the asset, and dissappeared.
On the other hand, some of the assets I got are really good, and their devs are giving amazing support. They are really professional, even if its an asset you only payed 20$ for.
So yes, I do have to object to describing the asset store as a snakes nest. As everywhere, there are black sheeps, and you need to do some sanity checking before buying anything. But there are many hard working professionals there providing good solutions at an amazing price, so its unfair to them to trash talk the whole asset store.
2) The quirks of Unity... ah, yes, how I come to HATE them! A lot of my work also goes into working around the limitations of the engine.
But to be honest, apart from the ancient lighting and shadow system (which will hopefully be replaced by something better in Unity 5), my biggest enemy currently is PhysX. Now, it might be that the integration into Unity is just not that great (after all, WheelColliders to my knowledge were introduced by Unity... and while they are a good starting point, boy, do they need work!)... but generally, AFAIK, Unity uses stock PhysX. And as far as I can tell from what I heard from other Physics engines, even the pricey Havok, they seem to be not much better when it comes do create realistic physical behaviour out of the box, or even worse, balance this realistic behaviour against the arcade needs of your game!
In the end, a lot of systems in an existing engine will be designed in a way that does not fit with your use cases. That means either get the source and fiddle with it (which would be too much for a beginner, and would kind of negate part of the reason why you went with an existing engine in the first place), or you work around these things.
Most of the time the latter works quite fine, though the solution looks ugly from the devs point of view, and might other limitations, even if the player in the end sees none of these shortcomings.
I will believe anyone who tells me things are better in UE4, because I haven't tried it thouroughly, so I couldn't judge it. If somebody tells me these limitations and problems are competly inexistent in UE4, I will believe it when I personally see it.
Then there is the fact that a lot of the quirks wouldn't be a problem if I wouldn't aim for the stars. For a simpler concept, simpler physics and graphics requirements, these limitations and quirks would never be a problem at all.
3) I completly agree on the point with creating his own engine... that is, if he wants to create an engine and not a game.
But true, it will teach much when it comes to becoming a better engineer... or even working more efficiently with an existing engine in the future. Programing for example, so you do not have to rely on visual scripting tools for everything