Here is what I made to track pixels between frames for the 5-point algorithm that I intended to use for scene reconstruction.

https://github.com/dmdware/srt

The first result I had with the tracking was strangely similar to a map of surface normals. I don't remember how it happened, but I think it was because I was tracking 2x2 pixels (so very inaccurate and prone to jumping to other likely matches) or because the checks I used were different.