well, there are enough examples of quite high profile game, or at least successfull Indie titles that seemed to have been created in their core as a result of prototyping and experimentation...
For example Goat Simulator. Not saying that this game is the pinnacle of a well designed game (it isn't, intentionally), but it just shows that sometimes, some very successfull games can come out of just playing around with prototypes long enough.
And then there are the many, many games that went through a long design phase.... only to be pivoted into a competly different direction during development. I read somewhere that a very successfull RPG was originally planned as RTS (Can't remember which though)... a more recent example might be this:
When Wargaming worked on World of Tanks, they originally planned to create an MMORPG with Elves and Orcs.... ingame shots of the early prototype even show (ugly looking) fantasy scenery and orcs and elves. Then somebody had the idea to instead do something else (not surprising seeing how year after year another WoW clone tried and failed from 2004 onwards)... and as tanks are really popular in russia, they went with that. The rest is history, and WG is a rich company now.
So... I guess both can work. It depends on how you work best, and most probably finding a good balance between going into prototyping without a plan and planning forever without ever giving it a try.
In the end, as murphys laws say, no plan survives the first contact with the enemy, so the sooner you can achieve that, the sooner you can either improve your plan or come up with a new one.
Or to put it in the words of "lean startup": start experimenting early, fail fast, learn, and pivot if necessary.