Why Is Object Rotation Resetting After 90 or -90 Degrees

If

you attach this to an object, clicking and dragging up and down will rotate accordingly.

The problem is holding the rotation after you finish dragging (and start again). If the object's X rotation is between -90 and -90, it will rotation again without issue. But if the rotation is outside of that range, unity sets the X rotation back to a number within that range once you click the object again. Why is this, and how do I stop this? The continuous rotation during the drag works.

The rotation also bounces between 180's if you drag past it. I'm guessing this is gimbal lock, but does this cause the above issue as well?

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class DragToSpin : MonoBehaviour
 {
     Vector3 mouseStartPos;
     Vector3 mousePos;
     Vector3 startRot;
     float dragDifference;
     bool dragging;
 
     // Start is called before the first frame update
     void Start()
     {
         dragging = false;
     }
 
     // Update is called once per frame
     void Update()
     {
         mousePos = Input.mousePosition;
     }
 
     private void OnMouseDrag()
     {
         if (!dragging)
         {
             mouseStartPos = mousePos;
             dragging = true;
             startRot = transform.rotation.eulerAngles;
         }
 
         dragDifference = mousePos.y - mouseStartPos.y;
         transform.rotation = Quaternion.Euler(startRot.x + dragDifference, 0, 0);
         
     }
 
     private void OnMouseUp()
     {
         dragging = false;
     }
 }