So I've been working on a stylized game character with the purpose of learning as much as possible about the technical aspect of character development and I've reached a point where I'd start baking normal maps, namely the dead end.

First let me share brief details about the character. Max poly budget - 7k triangles ( I got about 6900 or so). 1 texture map for everything. Here is the actual high and low poly.

And this is where I fail somehow and I'd truly appreciate if anybody helps out:

My baker is Marmoset Toolbag, by the way.

So I tried to adjust the cage, added support geometry and yet I still get with that awful waviness. Nor to mention, the nails of the character are barely visible, considering that the map is 2k.

What do you guys think?

I suspect that at least some of the “waviness” might be caused by either the form of the low-poly mesh, or--more likely I suspect--the triangulation of the low-poly mesh. You might perhaps experiment with a single region that shows such waviness, explicitly triangulating first in one way and then in the opposite way.

That is, something like this:

-----       -----        -----
|   |       |\  |        |  /|
|   |  =>   | \ |   or   | / |
|   |       |  \|        |/  |
-----       -----        -----