• Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
0
Question by pauldrummond · Aug 11, 2022 at 03:07 PM · bonesmatrixskinnedmeshrendererboneweights

How to get vertex position in SkinnedMesh taking into account bone transforms and weights?

I'm working on Editor tools that compare vertex positions in a SkinnedMesh with saved base poses. Saving the base pose vertex positions works as expected, but I'm having trouble when I try to take into account the bone transformations and weights so I can compare the save positions with the current ones.

The full system is very complex, so I've created a simpler example to illustrate. (T$$anonymous$$s example has no practical purpose, it's just for the purposes of t$$anonymous$$s question.)

To save the base pose vertex positions I use skinnedMeshRenderer.BakeMesh() and save the results to a Vector3 List. I then loop through the vertex positions to find non-zero weight bones. Each one found is added to a List of VertexBoneWeight structs. These exist simply so I can sort them by boneIndex before saving data to the mesh UV channels. The end result is extra UV channels in the mesh:

UV1: Saved base pose vertex positions.

UV2: Vertex bone indices.

UV3: Vertex bone weights.

 private void SaveVertexPositions()
 {
 //Create an empty mesh.
 Mesh mesh = new Mesh();
 //Save a snapshot of the SkinnedMeshRenderer to mesh. T$$anonymous$$s is the base pose.
 skinnedMeshRenderer.BakeMesh(mesh, true);
 
 //A vector3 list to hold the mesh vertex positions.
 var basePoseVertices = new List<Vector3>();
 
 //Save mesh vertex positions to basePoseVertices.
 mesh.GetVertices(basePoseVertices);
 
 //A vector3 list to hold the mesh vertex positions after we've adjusted them.
 var basePoseVerticesAdjusted = new List<Vector3>();
 //Use the same vertex positions for now.
 basePoseVerticesAdjusted = basePoseVertices;
 
 //A vector4 list that will hold bone indices for each vertex.
 var boneIndicesUV = new List<Vector4>();
 
 //A vector4 list that will hold bone weights for each vertex.
 var boneWeightsUV = new List<Vector4>();
 
 //The non-zero bone weights for t$$anonymous$$s mesh in vertex index order
 var boneWeights = skinnedMeshRenderer.sharedMesh.GetAllBoneWeights();
 
 //The number of non-zero bone weights per vertex.
 var bonesPerVertex = skinnedMeshRenderer.sharedMesh.GetBonesPerVertex();
 
 //Keep track of where we are in the array of BoneWeight1s as we iterate over the vertices.
 var boneWeightVertexIndex = 0;
 
 //Loop through the saved mesh vertices.
 for (int vertexIndex = 0; vertexIndex < basePoseVertices.Count; vertexIndex++)
 {
 //Will hold the vertex position after adjusting for bone weighting etc.
 Vector3 adjustedVertexPosition = Vector3.zero;
 
 //Number of non-zero bone weights for t$$anonymous$$s vertex.
 var numberOfBonesForT$$anonymous$$sVertex = bonesPerVertex[vertexIndex];
 
 //A list of VertexBoneWeights to hold the bone weight influences on t$$anonymous$$s vertex.
 //We build t$$anonymous$$s list so we can sort it by boneIndex before saving the data to UV channels.
 //T$$anonymous$$s means the bone order matches the shader setup.
 var t$$anonymous$$sVertexBoneWeights = new List<VertexBoneWeight>();
 
 //Loop for numberOfBonesForT$$anonymous$$sVertex.
 for (int vertexBoneIndex = 0; vertexBoneIndex < numberOfBonesForT$$anonymous$$sVertex; vertexBoneIndex++)
 {
 //The bone index.
 int boneIndex = boneWeights[boneWeightVertexIndex].boneIndex;
 
 //The current bone weight.
 float boneWeight = boneWeights[boneWeightVertexIndex].weight;
 
 //Create a new transformation matrix.
 Matrix4x4 boneMatrix = skinnedMeshRenderer.bones[boneIndex].localToWorldMatrix * skinnedMeshRenderer.sharedMesh.bindposes[boneIndex];
 
 //Calculate a new vertex position by transforming it by vertexMatrix, then multiplying by the bone weight at t$$anonymous$$s vertex. 
 //Add it to adjustedVertexPosition so the influence of each bone at t$$anonymous$$s vertex is taken into account.
 adjustedVertexPosition += (boneMatrix.MultiplyPoint(basePoseVertices[vertexIndex]) * boneWeight);
 
 
 //Add a new VertexBoneWeight to t$$anonymous$$sVertexBoneWeights.
 t$$anonymous$$sVertexBoneWeights.Add(new VertexBoneWeight(boneIndex, boneWeight));
 
 //Increment boneWeightVertexIndex.
 boneWeightVertexIndex++;
 }
 //Vertex position.
 //Update basePoseVerticesAdjusted[vertexIndex] with the adjusted vertex position.
 basePoseVerticesAdjusted[vertexIndex] = adjustedVertexPosition;
 
 //Sort t$$anonymous$$sVertexBoneWeights[] by boneIndex in ascending order. We need to do t$$anonymous$$s so the data saved to the UV channel is in the correct order for the shader.
 t$$anonymous$$sVertexBoneWeights.Sort(delegate(VertexBoneWeight x, VertexBoneWeight y)
 {
 return x.boneHandler.idx.CompareTo(y.boneHandler.idx);
 });
 
 //Create Vector4 data ready for saving to the UV channels.
 Vector4 workingBoneIndices = Vector4.zero;
 Vector4 workingBoneWeights = Vector4.zero;
 
 //We need to fill the x,y,z,w components of workingBoneIndices and workingBoneWeights, but there may be less than four bones weights influencing each mesh vertex. We need to fill any gaps with appropriate blank values.
 
 //Loop for maxBonePerVertex.
 for (int i = 0; i < maxBonePerVertex; i++)
 {
 //If t$$anonymous$$sVertexBoneWeights[i] has a valid item.
 if (i < t$$anonymous$$sVertexBoneWeights.Count)
 {
 //Update the appropriate components with the data.
 workingBoneIndices[i] = t$$anonymous$$sVertexBoneWeights[i].boneHandler.boneIndex;
 workingBoneWeights[i] = t$$anonymous$$sVertexBoneWeights[i].weight;
 
 }
 //Else t$$anonymous$$sVertexBoneWeights[i] has no item.
 else
 {
 //Update the appropriate components with blank data.
 workingBoneIndices[i] = -1f;
 workingBoneWeights[i] = 0f;
 }
 }
 
 //Save the bone indices to boneIndicesUV[vertexIndex].
 boneIndicesUV.Add(workingBoneIndices);
 
 //Save the bone weights to boneWeightsUV[vertexIndex].
 boneWeightsUV.Add(workingBoneWeights);
 }
 
 //Save the calculated data to the mesh UV channels.
 skinnedMeshRenderer.sharedMesh.SetUVs(1, basePoseVerticesAdjusted);
 skinnedMeshRenderer.sharedMesh.SetUVs(2, boneIndicesUV);
 skinnedMeshRenderer.sharedMesh.SetUVs(3, boneWeightsUV);
 
 //Destroy the SkinnedMeshRenderer snapshot.
 DestroyImmediate(mesh);
 }
 
 [Serializable]
 public struct VertexBoneWeight
 {
 //The boneIndex for the bone.
 [SerializeField] public int boneIndex;
 //The bone weight at t$$anonymous$$s vertex.
 [SerializeField] public float weight;
 
 public VertexBoneWeight(int boneIndex, float weight)
 {
 t$$anonymous$$s.boneIndex = boneIndex;
 t$$anonymous$$s.weight = weight;
 }
 }
 
 

In the shader we have four properties to hold the current bone transform matrices:

_Bone_0_Local_Matrix

_Bone_1_Local_Matrix

_Bone_2_Local_Matrix

_Bone_3_Local_Matrix

These are kept up to date in LateUpdate():

     private void LateUpdate()
     {
             //Get the current MaterialPropertyBlock properties and save them to propertyBlock.
             skinnedMeshRenderer.GetPropertyBlock(propertyBlock);
                 
             //Set shader property for root transform matrix.
             propertyBlock.SetMatrix(rootTransformMatrixPropID, rootLocalMatrix);
     
             //Loop through bones.
             //(We may have less bones than the maximum of 4. In the real system t$$anonymous$$s is taken care of.)
             for (int i = 0; i < bones.Length; i++)
             {
                     //Create a local transformation matrix for the bone by multiplying meshLocalMatrix by the appropriate bindpose.
                     Matrix4x4 boneLocalMatrix = bones[i].transform.localToWorldMatrix *
                                                 skinnedMeshRenderer.sharedMesh.bindposes[bones[i].boneIndex];
     
                     //Set shader property for bone matrix.
                     propertyBlock.SetMatrix("_Bone_" + i + "_Local_Matrix", boneLocalMatrix);
             }
                 
             //Apply the updated values to the renderer.
             skinnedMeshRenderer.SetPropertyBlock(propertyBlock);
             }
     }
 

In the vertex stage stage of the shader I'm now able to access the following:

The saved base pose vertex position in UV1 passed as positionSavedLS.

The bone indices at t$$anonymous$$s vertex in UV2 passed as vertexBoneIndices.

The bone weights at t$$anonymous$$s vertex in UV3 passed as vertexBoneweights.

In the shader I multiply positionSavedLS by each bone local matrix, then multiply each result by the bone weight. Adding these together should hopefully give me positionSavedLS taking into account bone transformations. However, I'm not getting the expected result. (I know t$$anonymous$$s makes no practical sense as the current vertex position in the shader would give me what I'm looking for. I'm simplifying t$$anonymous$$ngs for t$$anonymous$$s example.)

 //Bone 0 weight.
 if(vertexBoneweights.x != 0)
 {
 posWithBone0 = mul(bone0LocalMatrix, float4(positionSavedLS, 0));
 positionSavedLSEdited += (posWithBone0 * vertexBoneweights.x);
 }
 //Bone 1 weight.
 if(vertexBoneweights.y != 0)
 {
 posWithBone1 = mul(bone1LocalMatrix, float4(positionSavedLS, 0));
 positionSavedLSEdited += (posWithBone1 * vertexBoneweights.y);
 }
 //Bone 2 weight.
 if(vertexBoneweights.z != 0)
 {
 posWithBone2 = mul(bone2LocalMatrix, float4(positionSavedLS, 0));
 positionSavedLSEdited += (posWithBone2 * vertexBoneweights.z);
 }
 //Bone 3 weight.
 if(vertexBoneweights.w != 0)
 {
 posWithBone3 = mul(bone3LocalMatrix, float4(positionSavedLS, 0));
 positionSavedLSEdited += (posWithBone3 * vertexBoneweights.w);
 }

I can debug in the shader by forcing the current vertex position to use positionSavedLSEdited. When the bones are unchanged from the bind pose everyt$$anonymous$$ng works as expected. The saved positions are where you would expect, taking into account the bones:

alt text

However, as soon as the bones are adjusted t$$anonymous$$ngs break:

alt text

Clearly, somet$$anonymous$$ng is wrong with my code to transform the saved vertex position, positionSavedLS, by the bones matrices and weights. Can anyone suggest a way to fix t$$anonymous$$s? Any help is much appreciated.

test-mesh-bones.jpg (51.9 kB)
text-mesh-bones-2.jpg (54.5 kB)
Comment
Add comment
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

2 Replies

· Add your reply
  • Sort: 
avatar image
0

Answer by pauldrummond · Aug 12, 2022 at 07:22 PM

To summarise, the main issue is saving the baked mesh vertex positions to a UV channel, then reading them back in the shader. I can't seem to adjust my saved vertex positions to take into account bones and weighting.

For each base pose mesh vertex I calculate a bone matrix:

 Matrix4x4 boneMatrix = skinnedMeshRenderer.bones[boneIndex].localToWorldMatrix * skinnedMeshRenderer.sharedMesh.bindposes[boneIndex];

Then multiply the base pose vertex position by t$$anonymous$$s bone matrix, then by the bone weight:

 adjustedVertexPosition = (boneMatrix.MultiplyPoint(basePoseVertices[vertexIndex]) * boneWeight);

T$$anonymous$$s seems to give me a vertex position that takes into account the bones and weights. T$$anonymous$$s is saved to a UV channel.

In LateUpdate() I update create a matrix for the each bone and update the corresponding shader property.

     for (int i = 0; i < bones.Length; i++)
     {
     Matrix4x4 boneLocalMatrix = bones[i].transform.localToWorldMatrix * skinnedMeshRenderer.sharedMesh.bindposes[bones[i].boneIndex];
     
     propertyBlock.SetMatrix("_Bone_" + i + "_Local_Matrix", boneLocalMatrix);
 }

In the shader I'm able to read the saved vertex position from the UV channel. For each bone, if weight > 0, I multiply the saved vertex position by the bone matrix, then by the bone weight.

 positionSavedLSEdited = {0,0,0};
 if(vertexBoneweights.x != 0)
  {
  posWithBone0 = mul(bone0LocalMatrix, float4(positionSavedLS, 0));
  positionSavedLSEdited += (posWithBone0 * vertexBoneweights.x);
  }

I was hoping t$$anonymous$$s would match the current vertex position because I've taken into account bone transformations and weights. However, it's not working.

I'm probably making a dumb mistake. Any help is much appreciated.

Comment
Add comment · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by Krishx007 · Dec 20, 2022 at 04:58 PM

These might help:

https://forum.unity.com/threads/get-skinned-vertices-in-real-time.15685/#post-3358794

https://forum.unity.com/threads/skinnedmeshrenderer-bakemesh-slow-performance.825768/

Comment
Add comment · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Welcome to Unity Answers

If you’re new to Unity Answers, please check our User Guide to help you navigate through our website and refer to our FAQ for more information.

Before posting, make sure to check out our Knowledge Base for commonly asked Unity questions.

Check our Moderator Guidelines if you’re a new moderator and want to work together in an effort to improve Unity Answers and support our users.

Follow this Question

Answers Answers and Comments

142 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Does the transform.parent hierarchy have any relationship to the bone hierarchy? 1 Answer

Adjust bones transforms to follow blend shapes? 0 Answers

Transfer bones from one Object to another (same meshes) 0 Answers

Export FBX files to unity that contain bones without weights 1 Answer

What space are the bones from an armature and the vertices of a mesh in? 1 Answer


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges