I'm looking for an algorithm that I can use to remove vertices that are close to each other within a margin of error in a triangular mesh. Pretty much similar to Blender's "Remove Doubles" feature, if anyone is familiar with it.

I think the issue isn't just removing the doubles, but also how would I handle the face indices once I remove "duplicate" vertices?

I had to do this recently but unfortunately I wasn't using a simple vertex/index format so the details where somewhat different.  Going from how I did it and from the top of my head, I would say you do something like as follows........ Find the  two (or possibly one triangle, if it's on the edge of the mesh) that contain the two vertexes you are going to merge (i.e. they share and edge). These two triangles are simply removed and the triangle list is compacted accordingly. Now you are gong to remove one of the two vertexes that are close to each other and assign any indexes in triangles that reference that vertex to the other vertex. In reality I guess you could just leave the vertex you are destroying in the list and just dereference it, or you could actually remove it and compact the vertex list.  Hope I'm steering you in the right direction.

The simplest way is to just reconstruct a new mesh:

for each triangle of original mesh

{

create new triangle in new mesh

for each original triangle vertex

{

if new mesh already has a similar vertex use its index; else create new vertex and use its index

set vertex index to the new triangle 

}

}

Edit: An entire new mesh is not necessary, but an old and a growing new vertex buffer have to exist while the algorithm is running.

This is how I do it in my Mesh class:

        /// <summary>
        /// Welds close vertices into one.
        /// </summary>
        private void weldVertices( int[] tvis_ )
        {
            const float sqrDistThreshold = DupedVertexMinDistance * DupedVertexMinDistance;
            var duplicatedVertices = new Dictionary<int, int>();
            for( var i = 0; i < vertexCount - 1; ++i )
                for( var j = i + 1; j < vertexCount; ++j )
                    if( !duplicatedVertices.ContainsKey( j ) )
                        if( (pointByIdx( i ) - pointByIdx( j )).sqrMagnitude <= sqrDistThreshold )
                            duplicatedVertices.Add( j, i );

            // adjust triangles
            for( var n = 0; n < indexCount; ++n )
                if( duplicatedVertices.ContainsKey( tvis_[ n ] ) )
                    tvis_[ n ] = duplicatedVertices[ tvis_[ n ] ];
        }

For performance reasons, this method just identifies duplicated vertices and corrects triangles. I later use a different method to remove orphan vertices and invalid triangles (triangles with 2 or 3 duplicated indices). 

7 hours ago, Waaayoff said:

I think the issue isn't just removing the doubles, but also how would I handle the face indices once I remove "duplicate" vertices?

By same way. Really you have to keep in mind that you work with triangles not with vertices. Put to output buffer vertices of first triangle and record by the way new indices - it is a size of new buffer before adding a each vertex. Then get a second triangle. Search output buffer for its vertices. By search results you will know new indices of vertices that have to be merged. Then add vertices of triangle that not exists on output buffer yet and remember it indexes. Now you have full set of indices for second triangle so can just add it to output index buffer. And so on for next triangles. Also keep in mind that hardware and as result graphics API like D3D or OpenGL do not support separate attribute indexing. It mean that vertices that have same coords but belong to different poligons as result have a different normals, UV coords or other atributes is a different vertices, so have to be doubled to keep mesh integrity by other attributes.