One way to think about it is that honest negative feedback from a stranger could be better for your game than a friend saying "that's nice", when they don't really like it but rather because they know how much work you put in.
Not that this makes it any easier to hear, especially when not phrased constructively!
When you release your game in the wild, people will make a split second decision as to whether they might like it, and unfortunately the looks can be an important factor.
That said, do think about whether improving the graphics is the most important thing for your game right now. How sure are you that the gameplay is solid? Have you play tested it? Is there an interesting progression system? You might spend time polishing some graphics that might end up needing tweaking (or cutting) once you've honed the core game.
Don't let a throwaway comment derail your priorities!