Refactoring "for the sake of refactoring" isn't a great thing, but that is very much different from taking time to step back and review how you're going about things, and that very much can advance you farther along toward your end goals even if you spend time 'not advancing' while you hammer things out.
I try to get it hammered out before I code in the first place. Only as designs evolve dramatically do I usually see a need for re-factoring. Such as actions having entire lists of parts, tools, and skills required, and skills boosted, instead of just one or two parts, tools, and skills. Occasionally, i'll see that a design has serious flaws, in which case, yes - I do re-factor - or re-design from scratch to be more precise. By and large I don't sit down to do code entry until the code is fully formed and written in my head (or on paper if exceedingly complex). You might say that an ounce of research and design before code entry saves a pound of refactoring later. In essence, i try to figure out how to write it first, then enter the code, as opposed to writing the code while trying to figure out what to write, then re-writing the code once i do figure out exactly what to do. If I need to write code while figuring it out, I do it in pseudo-code in the design notes section of the todo list or in comments in the source, figure out out, then enter the code into the editor. Could the resulting code stand further re-factoring? Most likely. Hardly any piece of code can't be code-smithed and massaged a bit more. Does it really need it? Probably not. Especially since I'm the only one who will ever have to work with it. There are sometimes advantages to being an Army of One.