Rigidbody Add Torque with Mouse Drag

I want to spin a rigidbody around an axis when dragging the mouse. When the mouse is released the rigidbody should slowly come to a stop from drag. I can do this already with a modified version of the DragRgidbody script in C#, but I'm wondering if there is a more efficient way to do it. I only want to add torque on a single axis (z) to one rigidbody in a locked position, so a lot of that code seems unnecessary for my purpose. I've tried a ton of different techniques from all over the community, but drag rigidbody with spring joint is the only one that seems to work the way I want it to.

The code below was based on this Question and is the best thing I've come up with without a spring joint, but it's still not working the way I want it to. This script will spin the rigidbody with the mouse, but it feels too loose. That is to say it doesn't move with the mouse exactly. Is it possible to make it work that way, or is the spring joint my best option here? Is there much of a performance difference anyway?

 public class MouseTorque : MonoBehaviour 
 {
     public float holdDrag = 10f;
     public float freeDrag = 1f;
     Vector3 lastMousePos;
     
     void FixedUpdate ()    
     {
         RaycastHit hit;
         
         if (!Physics.Raycast (Camera.main.ScreenPointToRay (Input.mousePosition), out hit, 100f) )
             return;
 
         Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
         Vector3 mousePos = ray.GetPoint (hit.distance);
 
         Vector3 cross = Vector3.Cross (lastMousePos, mousePos).normalized;
         float theta = Mathf.Asin (cross.magnitude);
         Vector3 w = cross * theta / Time.fixedDeltaTime;
         
         Quaternion q = transform.rotation * rigidbody.inertiaTensorRotation;
         Vector3 torqueForce = q * Vector3.Scale (rigidbody.inertiaTensor, (Quaternion.Inverse(q) * w));
         
         if (Input.GetMouseButton (0))
         {
             rigidbody.angularDrag = holdDrag;
             rigidbody.AddTorque (torqueForce.normalized, ForceMode.VelocityChange);
         }
         else
         {
             rigidbody.angularDrag = freeDrag;
         }
         
         lastMousePos = mousePos;
     }
 }

The example code in this Question looks helpful too, but I'm not sure how I would modify it to follow a mouse drag.