Rotating an object with inertia (quaternions driving me nuts)

I have a simple script w$$anonymous$$ch rotates an object when the mouse button is down, and then slows down when the button is let go (like a fling). I have it working with the X-axis, but I'm using Quaternion.AngleAxis and it doesn't make a lick of sense to me. I've tried doing a c$$anonymous$$ld object w$$anonymous$$ch then controls only the Y-axis, but then the directions flip around probably because of local rotation. I don't know... it doesn't make sense to me, I just have it partially working and I don't know where to go from here. Any insights would be super appreciated!

T$$anonymous$$s is just part of the script. The section for the "inertia" isn't shown, but if I can figure out the X and Y rotating t$$anonymous$$ng, I can figure out the rest:

 if (Input.GetMouseButtonDown(0))
         {
             Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
             if (GetComponent<Collider>().Raycast(ray, out $$anonymous$$t, RayHitDistance))
             {
                 freeRotate = false;
                 moving = true;
                 startPoint = new Vector3(Input.mousePosition.x, Input.mousePosition.y, 10);
                 startPoint = Camera.main.ScreenToWorldPoint(startPoint);
                 startTime = Time.time;
                 startRotation = transform.localRotation;
             }
         }
 
         if (Input.GetMouseButton(0))
         {
             if (moving)
             {
                 Vector3 pt = new Vector3(Input.mousePosition.x, Input.mousePosition.y, 10);
                 pt = Camera.main.ScreenToWorldPoint(pt);
                 print(pt);
                 transform.localRotation = Quaternion.AngleAxis((startPoint.x - pt.x) * spinFactor, Vector3.up) * startRotation;
             }
         }

The bottom line of code is where I'm stumped.