Are matrices with a positive forward vector considered left-handed, and are inverse (negative) vectors considered right-handed?

For example, are the following layouts correct?

// left-handed transform vector
[rx ux fx 0]
[ry uy fy 0]
[rz uz fz 0]
[tx ty tz 1]

// right-handed transform vector
[rx ux -fx 0]
[ry uy -fy 0]
[rz uz -fz 0]
[tx ty  tz 1]

These are meant to be model matrices to transform an object into world-space where transformations follow the model*view*projection (MVP) order of multiplication to distort vertices into their viewport coordinates (before viewport transformation). From what I've gathered this layout is a common convention for DirectX, and even though we all live in a programmable pipeline world, devs typically use a left-handed coordinate system as that was what the fixed-function pipeline provided.

The way I personally calculate the look-at matrix, for objects' model matrices, not for the camera's view matrix, is like so:

// right-handed lookAt matrix to transform objects in world-space
[rx ux -fx tx]
[ry uy -fy ty]
[rz uz -fz tz]
[ 0  0   0  1]
// NOTE: to get the view-matrix equivalent, just inverse-transpose

This is how I do it in OpenGL. I multiply my matrices in the order of projection*view*model (PVM) where the translation vector's in the 4th column of the matrix rather than the 4th row of my DirectX example above. I think this is the common layout convention in OpenGL, from what I've seen so far. It also appears to be right-handed as moving objects positive along the z-axis appear to be backwards, into the screen when the view matrix is at its identity.

EDIT: The odd thing, in research I've done since high school anyway, the right, up and forward vectors always appear to be stored in the 1st, 2nd and 3rd columns respectively regardless of whether the translation vector is stored in the right-most column, or the bottom row. This doesn't seem correct to me as matrices using PVM vs MVP should be transposed versions of each other to get the equivalent final matrices to transform objects into clip-space.

This relates to the cross product. There is a way to determine the direction of the cross product of two vectors, using the "right hand rule" - in short, to determine the direction of (A cross B), you orient your right hand such that you can sweep your fingers from A to B. Your thumb will point in the direction of the cross product. Google "cross product - right hand rule" for better pictures and explanations. But you could redefine the cross product by using your left hand - this will just reverse the direction of the resulting vector.

you can determine the Z vector of your coordinate system by taking the cross product of the X and Y vectors (or in other language, get the k vector by taking the cross product of the i and j vectors). If you find the z direction by using a right handed cross product of x and y, its a right handed coordinate system. Likewise for a left handed cross product, and left handed coordinate system.

Hope that makes sense. But in short, the z vector is reversed.

We say two (orthogonal for simplicity) coordinate frame have the same orientation if we can transform one in the other by applying a rigid transformation (rotation + translation). There are only two possible orientations which we call right-handed and left-handed because they correspond to the two constructions already described by mv348. If we have to transform something from a coordinate frame with a right-handed (resp. left-handed) coordinate frame to a left-handed (resp. right-handed) one, we have to use a reflection and thus use a matrix with negative determinant. However, If we want to transform something between two coordinate frames with the same orientation (this is basically always true in graphics APIs) we use a matrix with positive determinant. As long as you are consistent with your conventions, you can simply ignore these issues.

This relates to the cross product. There is a way to determine the direction of the cross product of two vectors, using the "right hand rule" - in short, to determine the direction of (A cross B), you orient your right hand such that you can sweep your fingers from A to B. Your thumb will point in the direction of the cross product. Google "cross product - right hand rule" for better pictures and explanations. But you could redefine the cross product by using your left hand - this will just reverse the direction of the resulting vector.

A gem of knowledge there. I knew that the order of the cross product is important (AxB = right | BxA = -right), but I didn't make a connection to right-handedness for lack of a better term. It does make sense to me. If this is the case, if I wanted to generate a left-handed forward vector, I'd use Up X Right, correct?

Hope that makes sense. But in short, the z vector is reversed.

That said, if I try to find the local right vector with a positive z vector, then it's a left-handed system, correct? Here's my example for a lookAt model matrix:

Vector3 lookAt; // assume this was setup ahead of time as a normalized "lookAt vector"
Vector3 forward(0.0f, 0.0f, 1.0f);
Vector3 right = Vector3::Cross(forward, lookAt);
Matrix4 trans;

// construct rotation/translation matrix
trans.m[ 0] = right.x; trans[ 4] = up.x;  trans[ 8] = forward.x; trans[12] = 0.0f;
trans.m[ 1] = right.y; trans[ 5] = up.y;  trans[ 9] = forward.y; trans[13] = 0.0f;
trans.m[ 2] = right.z; trans[ 6] = up.z;  trans[10] = forward.z; trans[14] = 0.0f;
trans.m[ 3] = pos.x;   trans[ 7] = pos.y; trans[11] = pos.z;     trans[15] = 1.0f;

Given that the global forward vector is positive, and the forward vector's components aren't inserted into the matrix as negative values, this would make it left-handed, right? Also, the translation vector is along the bottom row of the matrix instead of the right-most column. I realized now that my matrix in my OpenGL code wasn't constructed correctly (although it seems to yield correct results). I took a look at the gluLookAt code, and it looks like this:

void glhLookAtf2( float *matrix, float *eyePosition3D,
float *center3D, float *upVector3D )
{
float forward[3], side[3], up[3];
float matrix2[16], resultMatrix[16];
//------------------
forward[0] = center3D[0] - eyePosition3D[0];
forward[1] = center3D[1] - eyePosition3D[1];
forward[2] = center3D[2] - eyePosition3D[2];
NormalizeVector(forward);
//------------------
//Side = forward x up
ComputeNormalOfPlane(side, forward, upVector3D);
NormalizeVector(side);
//------------------
//Recompute up as: up = side x forward
ComputeNormalOfPlane(up, side, forward);
//------------------
matrix2[0] = side[0];
matrix2[4] = side[1];
matrix2[8] = side[2];
matrix2[12] = 0.0;
//------------------
matrix2[1] = up[0];
matrix2[5] = up[1];
matrix2[9] = up[2];
matrix2[13] = 0.0;
//------------------
matrix2[2] = -forward[0];
matrix2[6] = -forward[1];
matrix2[10] = -forward[2];
matrix2[14] = 0.0;
//------------------
matrix2[3] = matrix2[7] = matrix2[11] = 0.0;
matrix2[15] = 1.0;
//------------------
MultiplyMatrices4by4OpenGL_FLOAT(resultMatrix, matrix, matrix2);
glhTranslatef2(resultMatrix,
-eyePosition3D[0], -eyePosition3D[1], -eyePosition3D[2]);
//------------------
memcpy(matrix, resultMatrix, 16*sizeof(float));
}

gluLookAt() creates a view matrix, however. My matrix is a model matrix for getting objects to face certain objects. I'm also assuming that the matrices are stored by column (every 4 elements make up a column in gluLookAt()). That said, here's how I construct my model matrix version of gluLookAt():

Matrix4 Matrix4::LookAt(Vector3 position, Vector3 target, Vector3 up)
{
	Matrix4 resultMat;
	Vector3 forward = target - position; // "position" would generally be called "eyePosition" for a view matrix
	forward.Normalize();
	Vector3 right = Vector3::Cross(forward, up);
	right.Normalize();
	up = Vector3::Cross(right, forward);
	
	resultMat.Set(	  right.x,	up.x,	-forward.x,	position.x, 
			  right.y,	up.y,	-forward.y,	position.y,
			  right.z,	up.z,	-forward.z,	position.z,
			  0.0f,		0.0f,	0.0f,		1.0f);
	return resultMat;
}

Is my model matrix setup correctly using a right-handed coordinate system? My global forward vector is <0.0f, 0.0f, -1.0f>.

This discussion has more to do with interpretation than numbers. If we use the left-handed convention (LH), then cross(Forward, Up) = Left and not Right as in the right-handed convention (RH). The exact same formula for the cross product is used. You are however interpreting the same numbers in a different way. For example, you may decide your Forward vector is (0, 0, -1) in RH and (0, 0, 1) in LH. The Up and Right vectors are, for example, defined as (0, 1, 0) and (1, 0, 0) in both cases. If we compute the cross product of Forward and Up we then get (1, 0, 0) in RH and (-1, 0, 0) in LH. You thus get Right in one case and -Right (or Left) in the other, but you have done the same operation.

I do not understand in your code what are lookAt and forward. I have always considered the two terms to mean the same exact thing. I always used Forward, Up and Side as in the GLU code you have posted. Note that by calling the Right vector as Side, you are actually making the code independent on handness/orientation. If you are working in a RH coordinate system, then Side will be Right, otherwise it will be Left. But the code is the same in both cases.

The following pseudo-code describes the LookAt function from the old D3D9 documentation:

zaxis = normal(cameraTarget - cameraPosition)
xaxis = normal(cross(cameraUpVector, zaxis))
yaxis = cross(zaxis, xaxis)

 xaxis.x           yaxis.x           zaxis.x          0
 xaxis.y           yaxis.y           zaxis.y          0
 xaxis.z           yaxis.z           zaxis.z          0
-dot(xaxis, cameraPosition)  -dot(yaxis, cameraPosition)  -dot(zaxis, cameraPosition)  1

It is different from your code mainly because the projection transformations used in the two APIs are defined differently. In DX the camera is looking in the +Z direction while in OpenGL it is looking in the -Z direction. Note however that the other axes are defined slightly differently as well. The xaxis points to the right of the camera (note that cameraUpVector and zaxis are swapped but the cross product works differently so you get the right vector in the end). The yaxis points up (the discussion is similar to the xaxis vector). If you use this code with a RH convention, you get the xaxis pointing to the left, the yaxis pointing up and the zaxis pointing forward.

Hope this help,
Antonio