Raycast inconsistent

Hi, I have a teleportation script that will teleport a player to where the mouse is or to first wall hit, like this:

 private void Leap()
   {
     Vector2 mousePos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
 
     RaycastHit2D hit = Physics2D.Raycast(player.transform.position, mousePos, Mathf.Infinity, mask);
 
     if (hit.collider != null)
     {
       player.transform.position = new Vector3(hit.transform.position.x, hit.transform.position.y, 0);
     }
     else
     {
       player.transform.position = new Vector3(mousePos.x, mousePos.y, 0);
     }
   }

If I stand too far from the wall that should stop the Raycast, it fails to hit more often then if I stand far away from it. That means I easily can teleport past that wall if I am standing far enough from it.

The red walls should stop me from teleporting past them: https://dl.dropboxusercontent.com/s/41bk74r7q1kjpxb/raycast_bug.gif