I'm working on implementing normal mapping for my renderer and I am using rapidobj to load the model. While loading the model data for each mesh, I am trying to calculate the tangent and bitangent for the mesh following the method presented by LearnOpengl.

However, some of the tangents and bitangents are resulting in NaN. When this occurs ‘f’ which we calculate is has the value of ‘inf’. These vectors then produce black spots on the resulting mesh when outputting the normals from the normal map. I'm not sure what I am doing wrong in my calculations for this to happen and was wondering if anyone could guide me in the right direction or spot my mistake. Thanks in advance.

// Load mesh data: position, tex and normals
for( std::size_t i = 0; i < shape.mesh.indices.size(); ++i )
{
	auto const faceId = i/3; 
	auto const faceMat = std::size_t(shape.mesh.material_ids[faceId]);

	if( faceMat != matId )
		continue;

	const auto& idx = shape.mesh.indices[i];

	ret.positions.emplace_back( glm::vec3{
		result.attributes.positions[idx.position_index*3+0],
		result.attributes.positions[idx.position_index*3+1],
		result.attributes.positions[idx.position_index*3+2]
	} );

	ret.texcoords.emplace_back( glm::vec2{
		result.attributes.texcoords[idx.texcoord_index*2+0],
		result.attributes.texcoords[idx.texcoord_index*2+1]
	} );

	ret.normals.emplace_back( glm::vec3{
		result.attributes.normals[idx.normal_index*3+0],
		result.attributes.normals[idx.normal_index*3+1],
		result.attributes.normals[idx.normal_index*3+2]
	} );

;			}

// Calculate the bitangent and tangent vectors
for (std::size_t i = 0; i < shape.mesh.indices.size(); i += 3)
{
	auto const faceId = i / 3; 
	const auto& idx0 = shape.mesh.indices[i];
	const auto& idx1 = shape.mesh.indices[i + 1]; 
	const auto& idx2 = shape.mesh.indices[i + 2]; 

	glm::vec3 edge1 = ret.positions[idx1.position_index] - ret.positions[idx0.position_index];
	glm::vec3 edge2 = ret.positions[idx2.position_index] - ret.positions[idx0.position_index];

	glm::vec2 deltaUV1 = ret.texcoords[idx1.texcoord_index] - ret.texcoords[idx0.texcoord_index];
	glm::vec2 deltaUV2 = ret.texcoords[idx2.texcoord_index] - ret.texcoords[idx0.texcoord_index];

	// this is becoming inf when vectors are NaN
	float f = 1.0f / (deltaUV1.x * deltaUV2.y - deltaUV2.x * deltaUV1.y);

	glm::vec3 tangent = glm::vec3(0.0f, 0.0f, 0.0f);
	tangent.x = f * (deltaUV2.y * edge1.x - deltaUV1.y * edge2.x);
	tangent.y = f * (deltaUV2.y * edge1.y - deltaUV1.y * edge2.y);
	tangent.z = f * (deltaUV2.y * edge1.z - deltaUV1.y * edge2.z);

	glm::vec3 bitangent = glm::vec3(0.0f, 0.0f, 0.0f);
	bitangent.x = f * (-deltaUV2.x * edge1.x + deltaUV1.x * edge2.x);
	bitangent.y = f * (-deltaUV2.x * edge1.y + deltaUV1.x * edge2.y);
	bitangent.z = f * (-deltaUV2.x * edge1.z + deltaUV1.x * edge2.z);

	ret.tangent.emplace_back(tangent);
	ret.tangent.emplace_back(tangent);
	ret.tangent.emplace_back(tangent);

	ret.bitangent.emplace_back(bitangent);
	ret.bitangent.emplace_back(bitangent);
	ret.bitangent.emplace_back(bitangent);
}

Eventually the UVs of your test mesh are bad.
If the UVs of both vertices from an edge are equal, you get a division by zero i guess.

Though, the mesh does not look like a typical UV problem case.
But still worth to try some other meshes to see if the problem persists.

Thanks for the reply!

The issue persists over different models. Am I possibly indexing incorrectly into my data to calculate this? I'm using rapidobj to load models. While I feel it's correct the way I am doing it, possibly I am wrong?

Indexing bugs would be my next guess too.

But it looks like your positions and UVs are right, since don't see any texture seams.
You should compare with the code for vertex positions and UVs, or post it as well.

Btw, it's inefficient to use duplicated vertices for each triangle, wasting lots of memory and bandwidth.
You should use duplicates only where UVs differ.

Libraries like this should help to optimize the data, and pretty sure it would generate tangents as well:

https://github.com/zeux/meshoptimizer​

Edit: Another such library to import models is AssImp. I use it, and assume it can generate tangents, but not sure. It can merge duplicates, but won't optimize as well as the above.

https://github.com/assimp/assimp​