GPU Skinning Problem

B. / · 2018-03-09T17:20:55

Hello, I have a Problem with GPU Skinning, I load from a COLLADA File my object with the vertices and the weights and bone indices for it and the bones with the matrices. For every vertex I choose 4 weights and 4 bone indices. For every non skin vertex i choose by the weights 1 0 0 0 and bone indices 0 0 0 0 (In the Bone Matrices Array is index 0 a Matrix Idetity) And i check up if all weights values together is always 1 or i calculate it to 1. So far so good, my Shader looks like this: bool HasBones; matrix BoneMatrices[256]; struct Vertex { float3 Position : POSITION; float3 Normal : NORMAL; float2 UV : TEXCOORD0; float3 Tangent : TANGENT; float4 Weights : WEIGHTS; int4 BoneIndices : BONEINDICES; }; float4 ApplyBoneTransform(Vertex input, float4 value) { if(HasBones) { float4x4 skinTransform = (float4x4)0; skinTransform += BoneMatrices[input.BoneIndices.x] * input.Weights.x; skinTransform += BoneMatrices[input.BoneIndices.y] * input.Weights.y; skinTransform += BoneMatrices[input.BoneIndices.z] * input.Weights.z; skinTransform += BoneMatrices[input.BoneIndices.w] * input.Weights.w; float4 position = mul(value, skinTransform); return position; } else return value; } Pixel vertexShader(Vertex input) { Pixel result = (Pixel) 0; float4 posWorld = mul(ApplyBoneTransform(input, float4(input.Position.xyz, 1.0f)), World); result.Position = mul(mul(posWorld, View), Projection); result.Normal = normalize(mul(ApplyBoneTransform(input, float4(input.Normal.xyz, 1.0f)), WorldIT)); result.UV = input.UV; result.View = ViewInverse[3] - mul(float4(input.Position.xyz, 1.0f), World); result.Tangent = normalize(mul(ApplyBoneTransform(input, float4(input.Tangent.xyz, 1.0f)), WorldIT).xyz); result.Binormal = normalize(cross(input.Normal, input.Tangent)); return result; } And if i set HasBones to true, my object will not draw right anymore, i only see two dark triangles I believe it depends on the bone matrices i load from the controller_lib of the COLLADA File and send it to the BoneMatrices in the shader + at the index 0 the Matrix Idetity. Has anyone an Idea what I make wrong and could help and explain me it? And i upload to this post the Collada file and images of the object draw in HasBones = false and HasBones = true Greets Benajmin Model.dae

Graphics and GPU Programming Programming DX11

Started by B. / February 03, 2018 06:41 PM

38 comments, last by B. / 6 years, 10 months ago

JoeJ

4,399

February 08, 2018 02:00 PM

You can set the target matrices to their according rest pose bone matrices. Then the model should look the same as in the modeling app without animation, and the 'combined' matrices should all end up being identity.

To test just if the vertices load correctly, you can ignore all matrices and render them without skinning. Again the model should look like in modeling app.

But there will be exceptions:

Often the model is already cut into pieces, and those pieces are parented under various bones or other nodes in the transform hierarchy.

This adds some additional complexity, as you now need the skeleton matrices to render them correctly even without animation or skinning. (You need to transform each piece by its parent hierarchy node.) This makes sense at least for props like guns or swords, usually parented by the hand bone so they animate properly when the hand is animated.

So for your initial test models you have it easier when the whole mesh is parented by the root.

Thinking of it, if you accidently parented your box model under one of the bones, this could explain the scaling you described eventually.

B. /

Author

February 08, 2018 02:30 PM

Hi Joe,

models will always draw correct in my engine and no model has a joint as parent, i am just a noob in gpu skinning, but not to transform a mesh right with a complex parent node hierarchy

So now i do CPU skinning, like you said, to test the code, to find faster bugs.

So I test my elbow cube model and animate (translate) the last bone, right side, to move down and again to the starpoint.

So i take only the first animation matrix that start form the startpoint and test the animated joint (the last joint) to see if my model keep his form, but unfortunately he tranlate the right side (the 4 edges) a little bit after left, so the mesh will get a little bit smaller

I think there is still a bug how i calculate the Local Space, that makes this translate effect!?

Here's what i have:


        public static Matrix CalculateLocalSpaceMatrix(Joint joint, Matrix result)
        {
            if (joint.Parent != null)
            {
                result *= CalculateLocalSpaceMatrix(joint.Parent, joint.Parent.Transform.ObjectSpaceMatrix);
                return result;
            }
            else
                return joint.Transform.ObjectSpaceMatrix;
        }

        private List<Vertex> originVertices = new List<Vertex>();

        private void DoSomethingToolStripMenuItem_Click(object sender, EventArgs e)
        {
            if (this.geometry != null)
            {
                Geometry g = this.geometry;

                if (this.originVertices.Count == 0)
                    foreach (Vertex vertex in g.Vertices)
                        this.originVertices.Add(vertex);

                List<Matrix> boneMatrices = new List<Matrix>();

                if (g.Bones.Count != 0)
                {
                    boneMatrices.Add(Matrix.Identity);

                    for (int iBoneIndex = 0; iBoneIndex < g.Bones.Count; iBoneIndex++)
                    {
                        Joint bone = g.Bones[iBoneIndex];

                        Matrix inverseBindMatrix = bone.InverseTransform.ObjectSpaceMatrix;
                        Matrix localSpaceMatrix = MatrixHelper.CalculateLocalSpaceMatrix(bone, Matrix.Identity);
                        Matrix animatedSpaceMatrix = this.AnimationList[0].TransformAnimations[0];

                        Matrix finalMatrix = localSpaceMatrix * inverseBindMatrix * animatedSpaceMatrix;

                        if (iBoneIndex == 2)
                            boneMatrices.Add(finalMatrix);
                        else
                            boneMatrices.Add(Matrix.Identity);
                    }
                }

                List<Vertex> skinVertices = new List<Vertex>();

                for (int i = 0; i < this.originVertices.Count; i++ )
                {
                    Vertex vertex = this.originVertices[i];

                    Matrix boneTransform = new Matrix(0);
                    boneTransform += boneMatrices[vertex.BoneIndices[0]] * vertex.Weights[0];
                    boneTransform += boneMatrices[vertex.BoneIndices[1]] * vertex.Weights[1];
                    boneTransform += boneMatrices[vertex.BoneIndices[2]] * vertex.Weights[2];
                    boneTransform += boneMatrices[vertex.BoneIndices[3]] * vertex.Weights[3];

                    vertex.Position = Vector4F.ToVector3(Vector3.Transform(vertex.Position, boneTransform));

                    skinVertices.Add(vertex);
                }

                List<float> result = new List<float>();

                foreach (Vertex item in skinVertices)
                    result.AddRange(item.ToFloatArray());

                this.scene.Graphics3D.VertexBuffer.Update(result.ToArray());
            }
        }

I also read the Collada Docu about skinning again: https://www.khronos.org/collada/wiki/Skinning

And the docu say that the bone matrices array are the invert bone matrices

Quote



This source defines the inverse bind matrix for each joint, these are used to bring 
  coordinates being skinned into the same space as each joint.  Note that in this case the
  joints begin at 0,0,0 and move up 30 units for each joint, so the inverse bind matrices
  are the opposite of that.

So i add in my Joint Class a second tranform class that store the Object Space Matrix of the bone, the first store the inverse matrix (load from file) and the second store the invert matrix from the file (to draw the skeleton preview right).

Any Ideas?

Greets

Benjamin

JoeJ

4,399

February 08, 2018 03:50 PM

I'm a bit confused you have 3 matrices there:


Matrix inverseBindMatrix = bone.InverseTransform.ObjectSpaceMatrix; 
Matrix localSpaceMatrix = MatrixHelper.CalculateLocalSpaceMatrix(bone, Matrix.Identity); 
Matrix animatedSpaceMatrix = this.AnimationList[0].TransformAnimations[0];

You could try to explain where each of those come from (but likely i won't get it... terminology is so confusing to me with this things.)

However, there must be the bone matrix in global space, once in restpose and once animated, and not inversed.

To proof this is right, you should now visualize it, with something like:

DrawLine (matrix[3], matrix[3] + matrix[0], red);

DrawLine (matrix[3], matrix[3] + matrix[1], green);

DrawLine (matrix[3], matrix[3] + matrix[2], blue);

If there is a bug about matrices, you should see it.

Similary you can this with vertices, just by rendering a point. Red for skinned, green for unmodified. (no need to upload buffers to GPU, so another potential source of error less.)

I do this kind of visual debugging all the time, and in combination with some trial and error many things can be fixed without thinking about it. Works also for non graphics problems. Eyes are the best debugger to me.

What seems unusual to me is the way CalculateLocalSpaceMatrix() goes from the current bone to the root. I would expect going from root to the bone, so you might need to inverse the final result (or inverse rotation order when using the result) to have the bone in worldspace. Visual debugging should clarify such things...

B. /

Author

February 08, 2018 05:00 PM

Quote



Matrix inverseBindMatrix = bone.InverseTransform.ObjectSpaceMatrix; 
Matrix localSpaceMatrix = MatrixHelper.CalculateLocalSpaceMatrix(bone, Matrix.Identity); 
Matrix animatedSpaceMatrix = this.AnimationList[0].TransformAnimations[0];

The inverseBindMatrix is the world/object space matrix of the bone i load from the file, remember, i wrote, that the docu say that each marix in the bone matrices array in the file is inverse. That why i had two transform classes in my bone class, one is original load from the file, the inverse Version and one that is the invert value of the inverse Version to draw my skeleton lines.

Quote



This source defines the inverse bind matrix for each joint, these are used to bring 
  coordinates being skinned into the same space as each joint.  Note that in this case the
  joints begin at 0,0,0 and move up 30 units for each joint, so the inverse bind matrices
  are the opposite of that.

The localSpaceMatrix, i thought to get in the end the final tranform of the bone relative to the bone hierarchy. I use this code if i group many meshes and transform only the group node/parent node.

The animatedSpaceMatrix should be the targetsource. I animated the last bone of 24 frames, so i have 24 targetsources/matrices.

So the result of TransformAnimations[0] or TransformAnimations[23] should be in the after multipy them with the other matrices give a matrix idetity.

Quote
However, there must be the bone matrix in global space, once in restpose and once animated, and not inversed.

Collada exportert for the bone matrices only the inverse world/object space matrices. So I invert these and store them in my second Transform Class in the bone class.

The animated matrices exported normal. That the TransformAnimations Array, i store them for the mesh.

So in the end i try it with what i wrote

Quote
Finally, let's combine both involved transforms to one (that's what i've missed in my very first post.)
	matrix combined = mS.Inversed() * mT; // or mT * mS.Inversed(), depending on the convention your math lib uses
vec animatedSkin = combined.Transform(skin);
	
Usually you can write just 'animatedSkin = combined * skin', but 'Transform' makes more sense eventually


inverseBindMatrix * animatedSpaceMatrix without success, so i thought i try it again with multiy it with the Local Space Matrix

B. /

Author

February 08, 2018 05:28 PM

ps. I check up the file to see, if the bone world/Object space matrix is the same as the first animation matrix, and its a little bit different

Bone:

1.000000 0.000000 0.000000 3.963553 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.011606 0.000000 0.000000 0.000000 1.000000

First Animation Matrix:
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000

That could explain, why i dont get the origin model shape back, after tranform these

In the file i read joint3-Interpolations-array (Animation Matrices Array)

Could it be that i first need to calculate the right animation matrices realtive from the bone object space, befor i can transform my vertices?

Here is the animation_lib of the file


<library_animations>
    <animation id="joint3-anim" name="joint3"><animation><source id="joint3-Matrix-animation-input"><float_array id="joint3-Matrix-animation-input-array" count="24">

0.041667 0.083333 0.125000 0.166667 0.208333 0.250000 0.291667 0.333333 0.375000 0.416667 0.458333 0.500000 0.541667 0.583333 0.625000 0.666667
0.708333 0.750000 0.791667 0.833333 0.875000 0.916667 0.958333 1.000000</float_array><technique_common><accessor source="#joint3-Matrix-animation-input-array" count="24"><param name="TIME" type="float"/></accessor></technique_common></source><source id="joint3-Matrix-animation-output-transform"><float_array id="joint3-Matrix-animation-output-transform-array" count="384">

1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -0.142293 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -0.532450 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -1.115392 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -1.836036 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -2.639301 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -3.470107 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -4.273373 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -4.994017 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -5.576958 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -5.967116 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -6.109408 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -5.989200 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -5.656859 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -5.154813 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -4.525488 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -3.811309 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -3.054704 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -2.298099 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -1.583921 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -0.954595 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -0.452549 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 -0.120208 0.000000 0.000000 0.000000 1.000000
1.000000 0.000000 0.000000 3.447073 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000</float_array><technique_common><accessor source="#joint3-Matrix-animation-output-transform-array" count="24" stride="16"><param type="float4x4"/></accessor></technique_common></source><source id="joint3-Interpolations"><Name_array id="joint3-Interpolations-array" count="24">
 LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR
LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR LINEAR
LINEAR</Name_array><technique_common><accessor source="#joint3-Interpolations-array" count="24"><param type="name"/></accessor></technique_common></source><sampler id="joint3-Matrix-animation-transform"><input semantic="INPUT" source="#joint3-Matrix-animation-input"/><input semantic="OUTPUT" source="#joint3-Matrix-animation-output-transform"/><input semantic="INTERPOLATION" source="#joint3-Interpolations"/></sampler><channel source="#joint3-Matrix-animation-transform" target="joint3/matrix"/></animation><animation><source id="joint3-visibility-animation-input"><float_array id="joint3-visibility-animation-input-array" count="3">

0.041667 0.500000 1.000000</float_array><technique_common><accessor source="#joint3-visibility-animation-input-array" count="3"><param name="TIME" type="float"/></accessor></technique_common></source><source id="joint3-visibility-animation-output"><float_array id="joint3-visibility-animation-output-array" count="3">

1.000000 1.000000 1.000000</float_array><technique_common><accessor source="#joint3-visibility-animation-output-array" count="3"><param type="float"/></accessor></technique_common></source><source id="joint3-visibility-animation-intan"><float_array id="joint3-visibility-animation-intan-array" count="3">

0.000000 0.000000 0.000000</float_array><technique_common><accessor source="#joint3-visibility-animation-intan-array" count="3"><param type="float"/></accessor></technique_common></source><source id="joint3-visibility-animation-outtan"><float_array id="joint3-visibility-animation-outtan-array" count="3">

0.000000 0.000000 0.000000</float_array><technique_common><accessor source="#joint3-visibility-animation-outtan-array" count="3"><param type="float"/></accessor></technique_common></source><source id="joint3-visibility-animation-interpolation"><Name_array id="joint3-visibility-animation-interpolation-array" count="3">
 STEP STEP STEP</Name_array><technique_common><accessor source="#joint3-visibility-animation-interpolation-array" count="3"><param type="name"/></accessor></technique_common></source><sampler id="joint3-visibility-animation"><input semantic="INPUT" source="#joint3-visibility-animation-input"/><input semantic="OUTPUT" source="#joint3-visibility-animation-output"/><input semantic="IN_TANGENT" source="#joint3-visibility-animation-intan"/><input semantic="OUT_TANGENT" source="#joint3-visibility-animation-outtan"/><input semantic="INTERPOLATION" source="#joint3-visibility-animation-interpolation"/></sampler><channel source="#joint3-visibility-animation" target="joint3/visibility"/></animation></animation>
  </library_animations>

JoeJ

4,399

February 08, 2018 05:49 PM

9 minutes ago, B. / said:
ps. I check up the file to see, if the bone world/Object space matrix is the same as the first animation matrix, and its a little bit different

Smells like a trace

i guess that's caused by the modeling app (for precision issue the difference would be too huge).

I remember when working with XSI there was an option to set the restpose for the skeleton or individual bones, which cased issues to me. Maybe it's something like that hidden somewhere in the modeling app you'd need to fix.

What could also help is to ignore the inverse bind matrix from file and calculate yourself instead. (i'd do this anyways because i don't trust those extremely complex applications so much)

B. /

Author

February 08, 2018 05:54 PM

Yeah, because in the visual_lib stay the same object space matix as in the animation array, but the collada docu say, i could ignore the visual matrices, because all imported stays in the matrices array in the controller_lib

I will check this carefully and answer than

JoeJ

4,399

February 09, 2018 07:57 PM

1 hour ago, B. / said:
i was right, the matrices i need was in the visual_lib, and all the object space matrices from the joints and the animations are all realtive to their parents.

Oh, you assumed them to be in global space? They are always local and you need to traverse the hierarchy to get final world space. This is good for mixing animations. E.g. you have a waving arm animation just from the sholder, and walk cycle animation. You can than lerp the waving animation with the walk and it will work as intended. If matrices would be in world space, the waving hand would stay in place while the rest of the character moves away Another advantage is that the same animation works for different characters with different proportions.

So, instead of saying 'object space matrices are relative to parents', you should better say 'local matrices' to be less confusing. For similar reasons some of your variable names confused me. (But take what i say with a piece of salt - i'm self taught!)

Further your code looks like this: For each bone you process its parenting hierarchy to get its transform, is this correct?

If so, note that this is very inefficient.

You should process the hierarchy like a tree from the root to the leaves (the other way around), so you touch every bone just once(!). While doing so you calculate the animated world space matrix for each bone so any relation to hierarchy is resolved. Then setting up skinning matrices is straight forward, e.g. like this:


	for each bone // can be in random order
	{
	Matrix worldSpaceRestBoneMatrix = bone.matRestWS; // this is in world space, so hierarchy does not matter anymore at this point, precalulated once when file is loaded
	
Matrix inverseBindMatrix = Inverse(worldSpaceRestBoneMatrix);
	
Matrix animatedSpaceMatrix = bone.matAnimWS; // also world space, calculated once per frame with hierarchy and animation data
	Matrix finalMatrix = inverseBindMatrix * animatedSpaceMatrix; // or vice versa
	}

Sorry i did not realize this earlier. I hope this resolves the still unknown bug...

GPU Skinning Problem

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

GPU Skinning Problem

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines