I'm having an issue where my normals, after applying my TBN matrix, seem to have these weird splodges everywhere on them and I cannot seem to figure out what they or what's causing it.

I assume the issue is likely with my TBN matrix construction, though I am unsure where I might be going wrong in this.

I think the issue might be to do with my mesh normals which are calculated by sampling the values around the current pixel and estimating their differences. This normal is then transformed to view-space and used in the calculation for TBN. Though I am not entirely sure if this is the issue.

// n in this calculation is a view-space normal
// https://stackoverflow.com/a/13983431 
// Collect samples around current point
float t = HeightValue(vertexUV + vec2(0.0, texelSize.y));
float b = HeightValue(vertexUV - vec2(0.0, texelSize.y));
float r = HeightValue(vertexUV + vec2(texelSize.x, 0.0));
float l = HeightValue(vertexUV - vec2(texelSize.x, 0.0));
float topLeft = HeightValue(vec2(vertexUV.x - texelSize.x, vertexUV.y + texelSize.y));
float bottomLeft = HeightValue(vec2(vertexUV.x - texelSize.x, vertexUV.y - texelSize.y));
float topRight = HeightValue(vec2(vertexUV.x + texelSize.x, vertexUV.y + texelSize.y));
float bottomRight = HeightValue(vec2(vertexUV.x + texelSize.x, vertexUV.y - texelSize.y));
// 2.0 * is used to place more weight ont the neighbour pixels than diaganol
float x = topLeft - topRight + 2.0 * (l - r) + bottomLeft - bottomRight;
float z = topLeft + 2.0 * t + topRight - bottomLeft - 2.0 * b - bottomRight;

// Normal
vec3 n = normalize(vec3(x, 2.0, z));
n = .... // convert normal to view-space

vec3 baseTangent = vec3(1,0,0);
vec3 baseBitangent = vec3(0,0,1);

vec3 tangent  = normalize(baseTangent - dot(baseTangent, n) / dot(n, n) * n);

vec3 bitangent = normalize( 
	(baseBitangent - dot(baseBitangent, n) / dot(n, n) * n) -
	dot(baseBitangent, tangent) / dot(tangent, tangent) * tangent);
	
TBN = mat3(tangent, bitangent, n);	

This is an image of what the normals look like when the TBN matrix is applied:

Try this instead, it's copied from my terrain normal calculation and is known to be correct using the conventions of my engine. I note that your code flips sign of the “dx” value, and the tangent/bitangent calculation is more complex than needed.

// Calculate gradient (Sobel filter)
float dx = 2.0f*(r - l) + (topRight - topLeft) + (bottomRight - bottomLeft);
float dy = 2.0f*(t - b) + (topLeft - bottomLeft) + (topRight - bottomRight);
dx *= (1.0/8.0);
dy *= (1.0/8.0);

vec3 t = vec3( 1.0, 0.0, dy );
vec3 b = vec3( 0.0, 1.0, dx );
vec3 n = normalize( cross( t, b ) );

vec3 tangent = normalize( cross( b, n ) );
vec3 bitangent = normalize( cross( n, tangent ) );

TBN = mat3(tangent, bitangent, n);	

@undefined Thanks for the suggestion. I tried this method and I'm still getting those black patches 

Are you visualizing the normals correctly? It looks like the black is just negative values. You should do something like this to debug normals so that they are in [0,1] range:

gl_FragColor.rgb = n*0.5 + 0.5;

The colors are also much too saturated. If rendered with correct gamma, it should be lighter in color and more desaturated.