if I were just to offer the player lots of spells to learn and have them create sort of a hybrid class based on what they choose to work with.
The Elder's scroll series do that. There is no class, but skills that you chose to up or not to create your own class, in a way. So you might look at it for more informations :)
Or would that be too difficult to implement?
It depends on how you implement it :D (more seriously, I don't think it's more difficult than going with a class system)
Can anyone thinks of a similar system
You can think that house = race. Being in a house (race) give you bonus in some skills and "racial" advantages. For exemple, if you are in the Griffindor house you have a bonus in Charisma, and a bonus when you learn fire based spells. When in the Slytherin, you have a bonus in Cunning, and when learning poison based spells.
If you pass a course, you get a new spell/ability/stat boost, but there's a chance that you fail.
Nice idea, I liket it :D
that would be easy to implement?
Same answer that before :)
but I do think there should be some kind of competition amongst "students"
It would be more works, but having different quest for each "house" would be nice. And each quest you win, you gain new equipment for your quest, equipment that your enemies' house won't have.
increases your bro network
Ahahah, I'm not partying I'm increasing my bro potentiality and networking!