What is the best way of calculating normals given that you have a list vertices which are shared beween some polygons, typically used by vertex array and indexing? At the moment, I just count the number of times a vertex is used in a polygon somewhere and accumulate the normals then average them out at the end. However this does not deal well with sharp corners and surfaces like that of a cube. Is there ways of differentiating automatically when you should use same normal of the same vertex or when you should use separate normals depending the polygon the vertex is in?

Why do you average the normals? What exactly are they "normal" to then ;;. The polygon class tutorial on this website has a space for adding in a vector for normals into the private data and it could be done when building the vertex list (assuming all the vertices are static). I may just not understand your question, though

My fellow Americans I have just signed legislation that outlaws Russia forever. Bombing will commence in five minutes.

yes there is. use the average stuff on a single surface. a cube should be declared as 6 surfaces. it is up to you how you implement it. The other way is generating the normals when creating the models. if you use a 3d editor you should not bother about this, it takes care of normal generation. If you generate geometry on-the-fly (eg. terrain), than usually you dont create boxes, or such non-continual surfaces.

"Knowledge is no more expensive than ignorance, and at least as satisfying." -Barrin

I know that some of the more popular graphics formats create "smoothing groups", or something like it, in which they group polygons or vertices that should have smoothed or averaged normals. Alternatively, you could set a threshold like 45 degrees so that any normal that is more than 45 degrees different from your base normal are not added to the average. You can find the angle between two vectors by (I think)

cos(angle in radians) = U dot V / |U||V|

Thats the dot product of the two vecotors divided by the product of the absolute value of their lengths.

I am currently using ac3d which does not generate normal for its own file format, at least I don''t think it does, It might very well uses the threshold method and generate the normals on loading, given the models.

So far the surface notion sounds most promising, although that would mean switching to some other file format that distinguish the underlying triangles from the concept of surface, ac3d kind of uses them synonmously, a triangle is just a type of surface.