.

There are multiple cases of potential zero vectors, which then can't be normalized. Maybe one of these causes the problem.

For example:

else {
			auto dist = player->position - m_camera->GetPosition(); // if both points are the same,
			dist = dist.unit(); // we get a division by zero here, casuing infinite numbers and potential problems

			auto pos = player->position - vec3(300, 0.0f, 300.0f);
			pos.y = pos.y + 2000.0f;
			m_camera->SetPosition(pos);
			m_camera->SetDirection(dist);
		}

Or here:

        right = up.cross(forward); // if right and forward are the same or form a line, we get a zero vector
        right.unit();

In most cases you need to handle those cases, if you can't be sure the problem is guaranteed to never happen.

Often all we can do is to ensure our numbers do not break assumptions of the code processing them. Meaning, our results may be still bad, but at least there is no NaNs or a crash in the application.
In this cause the only assumption is that right is perpendicular to forward. So any vector that is perpendicular will work.
I use code like this to generate such perpendicular direction:

const sVec3 ArbitraryPerpDir (const bool normalize = true) const
	{
		const sVec3 &v = *this;
		float x = fabsf(v[0]);
		float y = fabsf(v[1]);
		float z = fabsf(v[2]);
		sVec3 temp;
		if (x<=y && x<=z)		temp = sVec3(1,0,0);
		else if (y<=x && y<=z)	temp = sVec3(0,1,0);
		else					temp = sVec3(0,0,1);
		temp = v.Cross(temp);
		return (normalize ? temp.Unit() : temp);
	}

Using this to apply a fix:

        right = up.cross(forward); // if right and forward are the same or form a line, we get a zero vector
        float mag = right.length();
        if (mag > 1.0e-6f)
        	right /= mag; // valid, so normilze
        else
        	right = forward.ArbitraryPerpDir(); // close to zero, create arbitrary dir for a fallback

That's annoying extra work, but in doubt always do it, otherwise NaNs start to propagate through all your data os things interact with each other, and it does not work robustly.
However, if forward is already zero, this fix won't help of coarse.

Another minor issue:

else {
			auto dist = player->position - m_camera->GetPosition(); // this uses the old camera position, which can be anything
			dist = dist.unit();

			auto pos = player->position - vec3(300, 0.0f, 300.0f);
			pos.y = pos.y + 2000.0f; // now we know the new position, which is guaranteed to be different from player pos
			m_camera->SetPosition(pos);
			m_camera->SetDirection(dist);
		}

So i would calculate the new position before you calculate the difference to player, to eliminate one probable failure case.

To find what's the real cause of the flicker, something like that might help:
Log suspicious values to a text file each frame. You can then see how values change over time, and where there are some unexpected jumps.
Do not view the scene from the faulty camera, but use another camera for debugging purpose. But visualize the faulty camera matrix, so you can observe it from a robust perspective, which might give some insight.

@undefined Thanks! this fixed my issue! The camera now correctly looks down at it. The other issue I am having now is that my object seems clipped from the edges but it has nothing to do with the camera far plane since if I set it to a really high value, the issue persists