I am wondering about when is a good time to say, “yeah, this project is not shaping up at all. Time to start over."

I went through this feeling after going into from productions to alpha for an visual novel, the visuals looked okay as I was drawing over some 3D models I created for layouts. It looked bland, not terrible, not bad, not good, just oatmeal of visuals, plus the story is not quiet working out so that's going to have to be redone.

So, basically the question is when is a good time to step back, and ether call it quits or restart from scratch?

Perhaps some of the project can be reused? Your skills have improved and that will help you the next time around for sure.

How about making the entire visual novel with stick figures now, and get the rest of the game working? If the rest is good, you can always look into perfecting the visual novel afterwards. You'll feel more invested then, I'm sure.

You're asking when it's time to quit, and for that I have the same answer no matter what the obstacle may be in life.
You NEVER quit when it's something you're passionate about, something you dream about. Things will get tough, but you grow, learn and overcome the challenges, and not only will you reach your goals, you will become a better person in doing so.

You SHOULD quit if whatever you're doing is done for the wrong reasons (to get approval, to get money, to feel significant etc).

So what should you do about this project?
Well, if you're passionate about THIS project, step back and find out how you can improve one piece at a time.
It will be tough, but it will be worth it.
If you're passionate about making games, but not this project in particular, I would say find another project that sparks your interest, and use what you've learned so far to avoid ending up in the same pitfalls again.

Every project will have phases where it all feels hard and discouraging. Don't let that stop you.
You know in your heart, and in every bone in your body what you truly want, so go for it. ?

I hope this will help you decide.
And best of luck, no matter what!

I'll also have to add that even the project you're most passionate about - the one with all the sparks - can for periods feel like work. Why is that? Well, it is work, some parts are just more fun than others. Like @andypett says, you stick with this one if you really want it to be done.

As early as possible. A game that is abandoned before it is 10% finished can still count as a successful prototype that showed me what not to do. A game that is abandoned after being 90% finished cannot be considered anything other than a failure. I have a long list of abandoned prototypes, but only two real failures.

What's the purpose of the project?

If it is for learning, then learn what you intended and move on. When you are no longer learning from it put it away.

If it was to be a commercial success then asking the question shows you're not on a viable path anyway, and should start over by doing market research.

To me it was always about learning as much as I can in as little time as possible. Usually that meant I threw the project away after a day or a week. Those projects taught me a lot about design, what kind of mechanics I like, how I can make them feel good, how they feel bad and so on. Basically I learned what works and doesn't. When I did longer projects (6 - 12 months) then I only learned how to do bigger projects. Sure some design stuff here and there, but that was second hand.

Basically in 20% of the time I learn 80%. To learn the other 20% I need to invest 80% of the time. And those 20% usually are more about polishing and project management. So depending on what you want to learn you can throw away projects or keep them for long.

And if you are not sure if this should be the longterm project, then it probably isn't. The games I worked on for longterm always had something that I found interesting and kept me working hard on it. Motivation loss is normal during long projects and the game being fun is something that helps a lot to get you invested.

I Hope that helped.

Do you think the project is worth saving?

If No, then go ahead and drop it.

If Yes or Unsure, look around at related artworks and figure out what art style you want to emulate (I highly recommend against realistic art if you want to avoid blandness). As for the story, ask yourself; Who's trying to accomplish what; How are they going about it; What is the result?