I have been working on cleaning up my old linear algebra code after getting some actual programming experience (in audio...), and I have the above question about best practices: if a matrix is not invertible, what should the function Mat3 inverse( Mat3 const& a ) do? I know that MOST matrices are invertible, but I don't want things to explode on the off chance things go sideways.

I see five possibilities (I'm using the Adjugate matrix divided by the determinant to form the inverse):

1. Rely on the calling context to check if a matrix is invertible. (Yeah right)
2. Don't check the determinant and let the floating point exception happen. (Eww)
3. Assert that the determinant isn't zero (though I haven't written assertions yet).
4. Throw an exception.
5. Return some matrix which isn't the inverse, such as the matrix 'a' itself or a null matrix of all zeroes, which can be checked; in the case of returning 'a', you could check it against itself for equality. (Bugs?)

I'm leaning Assertion for development, possibly backed by throwing an exception for 'production' code (who am I kidding, none of this will ever go live :P) so that it is recoverable or at least loggable.

Are you making a math library for games or for "serious" applications --stuff where rare bugs have serious physical/financial consequences?

In general, the theme for bug handling in games is to have lots of testing up front.  During production, error as early as possible to make it easy to identify the hopefully recent change that caused the bug.  When you ship, remove all error checks because you are confident in your testing, right? 

So, yeah.  Assert.  If a non-invertible matrix goes through the invert function in a game, something went irrecoverably wrong.  There is no exception/error-handling to be done.  There's a bug and it needs to be fixed quickly.

My goal is games (or just graphics); I'm not doing finite element structural simulation or anything. Not saying I'm good at it either

Thanks!

I pretty much agree with corysama, though I have a slightly different conclusion. Yes, assert is a great idea. However, when writing code, you should (almost) always handle errors. Who knows where this code is going to end up being used? And it is much easier to properly handle the error now when writing the code, instead of trying to bolt on error handling 2 years from now.

Almost certainly, this means, in addition to the assert, throw an exception. It's trivial for you to write the code, today's processors are fast, and non-thrown exceptions are cheap, so there's no reason to remove the exception for production code. Then, someday if someone wants to handle the case of the non-invertible matrix, all they need to do is catch the exception, instead of having to agonize over what to do.

Error codes are a legitimate error handling mechanism as well, but exceptions are preferred modern C++. Unless you are planning to use your code on some limited embedded platform or the like, I would go for exceptions.

In our math library we will assert if the matrix isn't invertible when the Invert() function is called. We'll generally work around it by either working with the transform in a different representation (Float3 Translation + Quat Orientation + Float3 scale), or by using special-case inverse functions that only work for certain matrix configurations. For instance we have one that only works for rotation + translation (camera/view matrix) and one that only works for scale + translation (projection matrix). 

22 hours ago, gdunbar said:

Almost certainly, this means, in addition to the assert, throw an exception...

Thanks gdunbar, I was thinking it'd be good to back up an assertion with an exception and you sold me pretty solidly.

16 hours ago, MJP said:

In our math library we will assert if the matrix isn't invertible when the Invert() function is called. We'll generally work around it by either working with the transform in a different representation (Float3 Translation + Quat Orientation + Float3 scale), or by using special-case inverse functions that only work for certain matrix configurations. For instance we have one that only works for rotation + translation (camera/view matrix) and one that only works for scale + translation (projection matrix). 

I hadn't considered a work around. By using an alternate representation are you exploiting:

1. the possibility that the transforms individually are invertible but their composition isn't*
2. possible differences in the accuracy of the arithmetic that lead to non-invertibility in one form but not the other
3. knowledge of the transformation and how to invert it ahead of time (which could include nudging things just a tad so the math works out)

* That doesn't sound like a mathematically sound assertion, but I figure I'd ask just in case.

Just FYI:

3 hours ago, Zorinthrox said:

• the possibility that the transforms individually are invertible but their composition isn't*

Such thing can't exist. Let us have 3 invertible matrices:

For all of these following applies:

• All matrices (A, B and C) are row-equivalent and column equivalent to each other - e.g. are NxN
• All matrices have non-zero determinant
• All matrices have full rank
• All columns of each matrix are linearly independent and form a basis of N-dimensional space
• There is no eigen number of 0

Now, the following applies:

And therefore also applies for:

Therefore to answer your note - if the individual transforms are invertible, their composition always has to be invertible.

Makes sense.

On 6/7/2018 at 6:54 AM, Zorinthrox said:

Thanks gdunbar, I was thinking it'd be good to back up an assertion with an exception and you sold me pretty solidly.

I hadn't considered a work around. By using an alternate representation are you exploiting:

1. the possibility that the transforms individually are invertible but their composition isn't*
2. possible differences in the accuracy of the arithmetic that lead to non-invertibility in one form but not the other
3. knowledge of the transformation and how to invert it ahead of time (which could include nudging things just a tad so the math works out)

* That doesn't sound like a mathematically sound assertion, but I figure I'd ask just in case.

I was referring to a combination of 2 and 3. Basically if you know you have a projection matrix and you want to invert it, then you use the appropriate scale/translation inverse instead of the generalized matrix inverse. We don't try to automatically detect when it would be more appropriate to use a different method of inverting, it has to be done manually by the programmer.