Objects not Detecting Mouse Clicks Reliably (OnMouseDown())

I have set up the following script to let the player drag objects around my physics-based game. Sometimes it works flawlessly but half of the time it doesn't detect the mouse at all.

The same script is applied to multiple objects and they all have a rigidbody 2d and a box collider 2d, some instances of a preface might work while the other doesn't in the same scene.

I'm using Cinemachine with one virtual orthographic 2d camera.

 public float mouseSpeed = 35f;
 public float movementSmoothing = 2f;
 Vector2 mousePos;
 Vector2 mouseStartOffset;

 Rigidbody2D rb;

 void Start()
 {
     rb = GetComponent<Rigidbody2D>();
 }

 void OnMouseDown()
 {
     mousePos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
     mouseStartOffset = new Vector2(mousePos.x - transform.position.x, mousePos.y - transform.position.y);
     Debug.Log("The Mouse is Down!");
 }

 void OnMouseDrag()
 {
     mousePos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
     rb.MovePosition(Vector2.Lerp(rb.position, Vector2.MoveTowards(rb.position, mousePos - mouseStartOffset, mouseSpeed * Time.deltaTime), movementSmoothing));
     Debug.Log("Dragging!");
 }