• Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
1
Question by UDN_b9382f3b-c1f9-4e21-9f3d-5742b4363a79 · Jan 12, 2018 at 08:26 AM · unity 5controllerkinectrigged

How to Control a rigged body with KINECT

I want to control a rigged body like Robot Kyle by moving my body using KINECT. I am unable to find any tutorials covering this.

KINECT SDK1 had a controller for rigged body but SDK2.0 does't.

I want the rigged body to move as the player.

I have examples of KinectView and other few. I think I may have missed something. Please help me achive this.

Thank you

Comment
Add comment
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

2 Replies

· Add your reply
  • Sort: 
avatar image
1
Best Answer

Answer by rightdroid · Mar 07, 2018 at 02:48 PM

The short answer is to use Inverse kinematics (IK) where you can - which is hands, feet and head - and you'll have to manually code the rest. I'm in the same boat, did a lot of googling, but couldn't really find any easy solution.

So what I'm doing is I am building upon the BodySourceView and BodySourceManager scripts that were in the examples (direct link for the SDK v2 download. And here's a tutorial I used for initial setup).

Next you'll need a properly rigged mesh and correctly configured Avatar in Unity, which I assume you have. This step took me the most time, since there's no tutorial I've seen that covers newer versions of Unity/Mecanim and rigging for Kinect v2. Well, turns out the simplest was to create my own rig in Blender, bone by bone, skin it and export to Unity. It got the rig recognized and thus allowed me to use the IK system. Then I had to tick IK pass in animator window for the animator controller:

alt text

So now I have a rigged mesh with IK enabled. Next I'm setting my rigged body mesh to be centered around SpineMid object that KinectView dynamically generates when Kinect detects a skeleton. alt text

Manual adjustment is most likely necessary, depending on the height of the mesh and bone placement. This is basically splitting SpineMid gameobject's position Vector3 into x,y,z and adding an offset value to y:

 _player.transform.position = new Vector3(_spineKinectPos[0], _spineKinectPos[1]+spineAdjustment, _spineKinectPos[2]);



Then add IK action to a script that has access to the Kinect generated gameobjects. I had a playerController script that did all the positioning of the player.

 protected Animator animator;
 private GameObject _player;
 Start(){
    _player = gameObject;
    animator = _player.GetComponent<Animator>();
 }
     //a callback for calculating IK
         void OnAnimatorIK()
         {
             if(animator)
             {
                 // check if player is active
                 if(_playerData.IsActive)
                 {
                     if(_handRightKinectPos != Vector3.zero)
                     {
                         animator.SetIKPositionWeight(AvatarIKGoal.LeftHand,1);
                         animator.SetIKPosition(AvatarIKGoal.LeftHand, _handRightKinectPos);
                     }
                     if(_handLeftKinectPos != Vector3.zero)
                     {
                         animator.SetIKPositionWeight(AvatarIKGoal.RightHand,1);
                         animator.SetIKPosition(AvatarIKGoal.RightHand, _handLeftKinectPos );
                     }
                 }
                 else
                 {
                     animator.SetIKPositionWeight(AvatarIKGoal.LeftHand,0);
                     animator.SetIKRotationWeight(AvatarIKGoal.RightHand,0);
                 }
             }
         }



_handRightKinectPos and _handLeftKinectPos are Vector3 positions of HandRight and HandLeft Kinect gameobjects respectively. Now, all I needed was hands, but you can do it for feet and head's looking direction too.


This got me the basics working, but it doesn't look too great. So I added body rotation depending on the position of shoulders:

 // calculate body angle depending on the position of shoulders
                 if (_shoulderLeftKinectPos != Vector3.zero && _shoulderRightKinectPos != Vector3.zero)
                 {
                     float _leftShoulderX = _shoulderLeftKinectPos[0];
                     float _leftShoulderY = _shoulderLeftKinectPos[1];
                     float _leftShoulderZ = _shoulderLeftKinectPos[2];
                     float _rightShoulderX = _shoulderRightKinectPos[0];
                     float _rightShoulderY = _shoulderRightKinectPos[1];
                     float _rightShoulderZ = _shoulderRightKinectPos[2];
 
                     float _absoluteX = Mathf.Abs(_leftShoulderX) + Mathf.Abs(_rightShoulderX);
                     float _absoluteZ = Mathf.Abs(_leftShoulderZ - _rightShoulderZ);
 
                     float _division = _absoluteX / _absoluteZ;
                     float _spineAngleRadians = Mathf.Atan(_division);
 
                     if(_leftShoulderZ > _rightShoulderZ)
                     {
                         _spineAngle = 90 + (_spineAngleRadians * (180.0f / Mathf.PI));
                     }
                     else{
                         _spineAngle = 270 - (_spineAngleRadians * (180.0f / Mathf.PI));
                     }
                 }
 _player.transform.rotation = Quaternion.AngleAxis(-_spineAngle, Vector3.up);



Now you most likely need to adjust the angle values. I reckon you'd need to do similar math for other kinds of rotation calculations as well, neck tilting according to to angle to neck bone etc.

To be honest this is hardly optimized way of doing it, but it's working for me and gives me decent fps.

Comment
Add comment · Show 1 · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image UDN_b9382f3b-c1f9-4e21-9f3d-5742b4363a79 · Jun 25, 2018 at 10:24 AM 0
Share

Wow, Thanks for your time.

avatar image
0

Answer by TarunL · Jun 12 at 01:34 PM

Wow. this is awesome. I was struggling with it. And today i got your answer as the most satisfying one. Thanks a ton for putting this much effort

Comment
Add comment · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Welcome to Unity Answers

The best place to ask and answer questions about development with Unity.

To help users navigate the site we have posted a site navigation guide.

If you are a new user to Unity Answers, check out our FAQ for more information.

Make sure to check out our Knowledge Base for commonly asked Unity questions.

If you are a moderator, see our Moderator Guidelines page.

We are making improvements to UA, see the list of changes.



Follow this Question

Answers Answers and Comments

158 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

aldonaletto 0 Answers

The name 'Joystick' does not denote a valid type ('not found') 2 Answers

Converting Kinect depth coordinates to work with Unity 0 Answers

Kinect data retrieved on Update() call not available on LateUpdate() call. 0 Answers

Overlapping avatar controlled with OpenNI 0 Answers

  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges