So I have a heightmap for my terrain. The heighmap is a texture, and I have tessellation control and evaulation shaders in glsl (using OpenGL 4.0 atm) that tessellate them into smaller squares. I've tried to write code that looks at differences in neighboring pixels in the heightmap to determine the normals at a given point. The result of this is the attached image. It looks like it more or less works, but there's undesirable artifacts. I've drawn a black circle around one such artifact in the close up screenshot to the right. 

Here's my tessellation evaluation shader that computes vertex positions and normals:

	#version 400 core
	uniform sampler2D texture0;
uniform sampler2D texture1;
	uniform mat4 cameraProjectionMatrix;
uniform mat4 cameraMatrix;
uniform vec3 cameraPosition;
uniform vec3 cameraRight;
uniform vec3 cameraUp;
uniform vec3 cameraDirection; 
	layout(quads, equal_spacing, ccw) in;
	in vec3 ES_Position[];
	out vec2 UV;
out vec3 FS_Position;
out vec3 lightingNormal;
	vec3 getNormal(float cellSize,float totalPixels,vec2 uvCoords)
{
    float heightL = texture(texture0,uvCoords - vec2(1.0/totalPixels,0)).r*10.0;
    float heightR = texture(texture0,uvCoords + vec2(1.0/totalPixels,0)).r*10.0;
    float heightD = texture(texture0,uvCoords - vec2(0,1.0/totalPixels)).r*10.0;
    float heightU = texture(texture0,uvCoords + vec2(0,1.0/totalPixels)).r*10.0;
	    vec3 tangent = normalize(vec3(cellSize,heightL - heightR,0));
    vec3 bitangent = normalize(vec3(0,heightD - heightU,cellSize));
    return normalize(cross(tangent,bitangent)) * vec3(1,-1,1);    
}
	void main()
{
    float totalMapSize = 400.0;
    float totalPixels = 2048.0;
    float cellSize = totalMapSize/totalPixels;
	    vec3 highXPos = mix(ES_Position[3],ES_Position[2],gl_TessCoord.y);
    vec3 lowXPos = mix(ES_Position[0],ES_Position[1],gl_TessCoord.y);
    FS_Position = mix(lowXPos,highXPos,gl_TessCoord.x);
    vec2 uvCoords = FS_Position.xz/totalMapSize;
    float actualHeight = texture(texture0,uvCoords).r*10.0;
    FS_Position.y = actualHeight;
    gl_Position = cameraProjectionMatrix * cameraMatrix * vec4(FS_Position,1.0);
	    UV = gl_TessCoord.xy * vec2(0.33,1);
	    vec3 norm = getNormal(cellSize,totalPixels,uvCoords);
	    //vec3 normL = getNormal(cellSize,totalPixels,uvCoords - vec2(1.0/totalPixels,0));    
    //vec3 normR = getNormal(cellSize,totalPixels,uvCoords + vec2(1.0/totalPixels,0));    
    //vec3 normD = getNormal(cellSize,totalPixels,uvCoords - vec2(0,1.0/totalPixels));    
    //vec3 normU = getNormal(cellSize,totalPixels,uvCoords + vec2(0,1.0/totalPixels));
    //vec3 normX = mix(normL,normR,0.5);
    //vec3 normY = mix(normD,normU,0.5);
    //vec3 normAround = mix(normX,normY,0.5);
    //lightingNormal = mix(norm,normAround,0.5);
    lightingNormal = norm;
}

Trying to approximate what pixel it is on and finding the supposedly neighboring pixels using floating point math seems hacky, I feel like there's a better way to do this that will eliminate those artifacts and make it look a lot more smooth. 

Not sure what I am looking at. I see the shadow wrong but what is the giant beam going up. Looks like a strange height map or something.

Also odd, you are calculating normals without using the actual current pixel, just the surrounding ones. I would take the current pixel and sample the right pixel and upper pixel only and use that to calculate the normal and tangents. I think if you cross the right sample with the upper sample, you will get a more proper normal.

Sorry that is (right - current) = vector, (upper - current) = vector. Cross the two.

Is the second picture a "spike" in the terrain height map? Like the surrounding terrain is all at ~0.2 and there is one pixel at 1.0? If that is the case, then I'm going to agree with dpsdam450 that determining the local behavior of the surface without directly using the height sample at uvCoords might be the culprit.

Note: I'm assuming a right-handed coordinate system where Z points up, X and Y describe a flat plane and that the center of a pixel in the height map corresponds to a vertex in the terrain mesh.

If the tangent of the surface lies in the same general direction as X and the bitangent in the same general direction as Y, and d is the distance away from uvCoord you sample the height map, then the tangent should be something like:

normalize(  vec3( d,0, heightAtUV - texture(uvCoord - vec2(d,0)) )
          + vec3( d,0, texture(uvCoord + vec2(d,0)) - heightAtUV ) )

And the bitangent:

normalize(  vec3( 0,d, heightAtUV - texture(uvCoord - vec2(0,d)) )
          + vec3( 0,d, texture(uvCoord + vec2(0,d)) - heightAtUV ) )

Note that I omitted the reference to the texture being sampled. Then your normal is normalize(cross( tangent, bitangent)). That should smooth out any big spikes and correctly generate a normal that points straight up at the pinnacle of a spike in the texture. You can adjust the tightness of the shading by manipulating d, which you've wisely incorporated into the parameters of getNormal().

If you just sample the points up and to the right of the current point, you'll get other inconsistencies like the tip of a spike pointing out towards XY instead of straight up. That won't be too much of a problem if you've got mostly rolling terrain or a very high triangle density, but you don't know when you need a cliff or a shear face.

ETA: The method I've described will probably result in spikes not having much of a "dark side". Artifacts around such discontinuities in discrete data is unavoidable because of the Nyquist-Shannon Theorem. You need a height map slightly larger than twice the triangle density to avoid it. For that reason spikes in the terrain should be limited to having a pinnacle no less than 2x2 vertices, which is probably what a level designer would do anyway. You might get some shadow "bleeding" over sharp edges, too, but it will at least be smooth. You might avoid this by turning the above equations inside out and normalizing the vectors before adding them, then normalizing again x_x.

Quote

" If you just sample the points up and to the right of the current point, you'll get other inconsistencies like the tip of a spike pointing out towards XY instead of straight up. That won't be too much of a problem if you've got mostly rolling terrain or a very high triangle density, but you don't know when you need a cliff or a shear face. "

Since a point such as a one-pixel peak is not a surface, but a point: You should perform the surface normal calculation I said. If you want the accurate peak normal to point at the sky, it would be averaging the surrounding pixles. That's how per-vertex normals are calculated for a mesh, same thing would apply here.

1 minute ago, dpadam450 said:

Since a point such as a one-pixel peak is not a surface, but a point: You should perform the surface normal calculation I said. If you want the accurate peak normal to point at the sky, it would be averaging the surrounding pixles. That's how per-vertex normals are calculated for a mesh, same thing would apply here.

Fair. It's probably silly to worry about edge cases like spikes, which no height map terrain should have anyway.