Why is my rigidbody getting stuck in the air when using mouse movement?

I am working on a mobile game that allows a player to move around boxes using touch. I found that using mouse down/drag/up works like touch input on mobile.

My problem is that these boxes have a rigidbody on them, with gravity turned on, and while they can be moved around fine, occasionally, every 30 movements or so, the box, and whatever other box is next to it, gets stuck in the air. I am not sure whats going on here but would appreciate the help. Below is the code that I am using for moving my boxes.

private void OnMouseDown()
    {
            distance = Camera.main.WorldToScreenPoint(transform.position);
            positionX = Input.mousePosition.x - distance.x;
            positionY = Input.mousePosition.y - distance.y;
    }

    private void OnMouseDrag()
    {
            Vector3 CurrentPosition = new Vector3(Input.mousePosition.x - 
            positionX, Input.mousePosition.y - positionY, distance.z);
            Vector3 WorldPosition = 
            Camera.main.ScreenToWorldPoint(CurrentPosition);
    }

    private void OnMouseUp()
    {
         rb.useGravity = true;
    }

I have looked at many different posts about the potential problem and I can say that the following are not the problems:

  • Time scale is set to 1.
  • The size of the box is 1, 1, 1 (Primitive box)
  • Gravity scale is set to -9.81

try to put this OnMouseDrag() or in Update()

if (Input.touchCount == 0)
     rb.useGravity = true;