Orchestral pieces require many different instruments all working together simultaneously. Even the most skilled musician can't play an entire orchestra by himself, he needs some help.
No, a musician may not be able to play every instrument in the orchestra by himself. However, the composer of that orchestral piece had to learn how to write music for each instrument. This requires learning about what each instrument can do, what they sound like, and the technical limitations of each instrument. As a composer, there are few things worse than the violist coming to you after the first rehearsal of your new piece, with the sheet music in hand and sternly saying to you "this note is not on my instrument".
The thing is, what you are describing has already been tested. It's how pretty much all software was developed, until people began to realise it was a horrible way to develop software and switched to more iterative approaches.
Then perhaps the reason why this approach does not work is simply due to the current state of knowledge about game design. The solution could be that we need game designers who are more knowledgeable about design - as well as a general increase in knowledge about design by all people in game development. This will happen, as we try more and more to approach game design in a scientific and analytical manner. This will lead to the identification of common patterns in design as well as naming standardization for those patterns. As this happens, game designers as well as all the other developers will be able to grasp and put together high level designs that work - and they'll be able to do so very quickly and efficiently, as well as being able to communicate these designs among each other - all because these patterns of design will be a standard part of their vocabulary.
This is precisely what happened in music with the development of music theory. By learning about music theory (which is a standard part of music school) one learns about all of the guidelines about how to compose music that sounds good. One learns about all of the patterns in harmony, voice leading, chord progressions, structures, and the like. By using a set of rules which consist of patterns that have been time tested, this takes away a huge amount of guess work in the process of composing a piece. With music theory, it's possible to compose a piece that sounds good without ever needing to sit down at an instrument to hear it. When the same thing happens to game design, it will take away the need for all of that guess work, trial and error, and iteration that happens in the studio. It will be possible to put together a detailed design document of a game that works great before even needing to step foot in the production studio. This process of developing game design theory is already happening, with the help of people such as those at Extra Credits, and Errant Signal.