I use a combination of planning and just plain cowboy coding.

When I start a project it starts with an idea. This idea usually stays in my head for a few weeks, during wich i refine it and come up with some strategies. After that i decide its time to start coding some. While coding you will soon run into problems, so i start coding something else
That's when the planning starts. I plan out the solution for the existing problems and then implement those. After that I will probably have some ideas for additions to the current project and the process starts over again.

I do not like to think about everything before i start working on a project, because that leaves me little room for expension.

I do however always write down small ideas i come up with during the coding.

I don't know if this is a good way to work it since i've not finished a game yet (still working my first mayor project)

Basically, do you plan everything in advance or just kinda wing it?[/quote]
I think this style is called cowboy coding... You should plan things in advance, otherwise you'll waste too much effort on refactoring the code.

Also, writing an engine just to try a game concept is a huge overkill. I'd recommend you use one of the freely available SDKs (Unity, UDK et al) or if you want to work on an engine - try working on one of the open source engines out there.

This might be more information then you need but i am going to briefly break down the entire development in the game industry.


Steps to create a game

The basic steps in the game industry

Pre-Productions: Essential you create your idea and document it out in a Game Design Document. Their are many other documents like a TDD and an Art bible but i think you really need a GDD as that is the core of the games design. After this you would normally pitch it to a publisher if you were a company, but in your case you would just start to build based on the designs that you have laid out.

This is very important as it allows you to in many way test the game before it is built and allows for a holistic plan of the game. This can seem really daunting especially when you don't always have the answers but try your best to figure it out before you build it because it will save you time. Many people build and uses wiki's for this. I use a word template.

There are many more steps in pre-production such as pitching and prototyping, etc however i am going to leave them out.

Production: This is where you would be creating and building the majority of the game. Production is general broken up into milestones to give goals and structure to it. If you are a single individual trying to produce a game. I would recommend trying to break it down into smaller chucks called sprints. Here is a link to a sprint template i use. (Sprints can be used to break down large goals into single deliverable and allow for an individual to plan out the how long a project or goal will take. I rarely plan more then 2 weeks in advance as things change. For more info see Scrum)

This is where a lot of iteration on the design you have built will happen. I find that about 30% of the original design remains. This is a result of both cuts due to time/resource and change of direction in design.

Some of the major milestones in industry are: Alpha == feature complete, Beta == content completely, at this point bugs are the major focus, Gold Master == Ship to the presses for mass distribution.

Post Production: This would be doing your post mortem and review what went well and how you could improve in the future.

I hope this was useful. If you have any questions please feel free to contact me.

After failing many projects, I can tell you what works and what doesn't work. (btw, I think this is more of a project management problem)
Here's how the typical project starts:

1. Someone has an idea. They start thinking about it and it generates a bunch of new ideas. They get super excited about it along the way. Then, they want to make the idea come to life. Either they can bring the idea to life by themselves or they have to find other people who can do it for them. The initial enthusiasm is the jolt of energy required to get people to start investing their time and money into the idea. A lot of game ideas fail to launch because they never get past this phase.

2. Either the idea dies on the vine or a more formalized process of development begins to take place. The idea is still nebulous, so lots of thought needs to be invested on nailing down the specifics for how the idea works. In terms of a game, this means defining the game mechanics, the units, the story, the player experience, interfaces, etc. This is typically done with a game design document or rapid prototyping.

3. The project eventually moves into the construction phase (if it's gotten this far). This is where the going gets tough and the initial spark of enthusiasm fades. Once the enthusiasm fades, people who aren't getting something from the project will quit (be it money, experience, satisfaction, promises, etc). If your project loses key people, it will most likely fail and/or the remaining team will suffer from lower morale. When the going gets tough, morale is everything because it alone is what keeps the team going forward with the project. The construction phase of the project is also where new ideas for features come up. Some of those features may really enhance the game. Other features may be cool "glitter" but will only waste time (but, consider the morale cost and chilling effect on creativity of excluding it). The most common case is the notorious 'feature creep' which is a feature which changes the size and scope of your project, which then translates into requiring more work and time, eventually leading to a project which grows beyond the capabilities of the existing team. If it seems that a project grows too big, it may never be completed and this leads to a sense of futility and despair with the team members, which also has a negative consequence to team morale and leads to project failure. You as a designer, are the gatekeeper for deciding which features get added and excluded. You must hold the reigns tight because the project success depends on it. Having a clearly written and well defined game design document will be like the light house of clarity in a sea of ideas.

4. If you've miraculously gotten past the construction phase, you'll move on into the testing phase of development. Here, you hunt for bugs, design flaws, and balance issues. Most likely, you'll have been doing lots of iterative game play testing during your construction phase, so the testing phase is a quality assurance and wrap up phase. I may only write a few sentences on it, but this phase may be one of the most important phases for guaranteeing project success. If you ship a non-working or buggy product, it will sell poorly and damage the reputation of your game and your company/team/organization and anyone affiliated with you. Lots of incomplete or buggy games have been shipped prematurely with the "oh, we'll just submit a patch a few weeks after release" mentality. Don't do this or overlook the importance of testing.

5. The next phase is deployment/release. This is where you get your game out in front of your users, turn on the registration servers, the login servers, the game servers, etc. and get ready for business. This phase of the project is a responsibility jointly shared by the business side and the technical side of the house. You can flub this up and all the hard work you and your team have put into the project leading up to this moment may have been for nothing, regardless of how great your product may be. Did you do marketing for the game prior to release? How are you distributing the game? (digital? retail? free download?) What technical work needs to be done? Can your hardware infrastructure handle the network load? etc etc. Doing well in this phase is an exponential multiplier. Based on some of the games I've seen, you could make wheelbarrows of money by selling polished turds if you execute this well (*cough* farmville *cough*).

6. The last phase is the training and maintenance phase of the project. It's often overlooked but also quite important because it extends the lifetime of your product (notice how the verbage has changed from 'project' to 'product'.) I've had the experience of developing a software tool and executed all the previous phases perfectly but seen the project fail because I couldn't maintain it or train end users on how to use the software. A lot of my life energy was wasted because the product couldn't be maintained (I wrote some software while I was in Iraq and I left the country). Your team may lose some members. With those members goes their technical expertise in your product. They may be fired or laid off by the company, may have some family emergency, get hit by a bus (or randomly blown up by a mortar/rocket), get hired by someone else, get bored and move on to bigger and better things, etc. How do you fill in the vacuume of knowledge left in their wake? Documentation. Cross training. Commented code. If your game has a critical bug fix that needs to be applied, who will do it and how will you get it out to your users? Take a lesson on the importance of this step in the project lifecycle from a company who did this well: Valve. Half-life came out in the late 1990's and had a good lifespan. Valve also encouraged and supported the modding community which extended the lifespan even further (via Counter-strike and other HL mods). The popularity of CS entrenched Valve in the minds of the customers. When HL2 came out, it was a sure bet that it would sell well. It then became the launching platform for Valves digital distribution hub "Steam", which was able to gain a majority of the market share for DD, further cementing their position as a major player in the game industry (and actually revolutionizing the distribution model).

All in all, when you're designing your game, take stock of your resources before deciding how complex it's going to be. It's better to create an overly simple game which gets finished and polished than to create an overly complicated game which may or may not get completed to a high standard of quality (hardest lesson to learn and remember). For a lesson on the bounties of simplicity, just look at the front page of Google. As one of my Operating Systems professors once said, "Make it simple for now, we can always add complexity later" (he's the windows kernel architect at Microsoft).