Top down look rotation snaps when looking left, right, and back

Sure enough, these are the 3 directions the camera sees. The camera is moved back and up and angled down at 60 degrees. so it makes sense that only left right and forward are snapping. However, I have no idea why they are snapping in the first place. Heres my code:

Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
			RaycastHit hit;
			if (Physics.Raycast (ray, out hit, 50, rayMask))
			{
				Vector3 point = new Vector3 (hit.point.x, transform.position.y, hit.point.z);
				transform.LookAt (point);
			}

So basically when I rotate a certain amount, it rotates another 20 or so degrees in a “snappy” fashion. its only when I look left right and at the camera. I am sure only the plane is in the cast mask.

I did a similar thing with an xbox controller turning the player.

Get the LookAt position you want, in your case: point.
Now save your current rotation: var rot = transform.rotation.eulerAngles.
Give the transform the new lookAt point like you have.
Now Lerp the rotation to smooth out the turning. eg. transform.rotation = Quaternion.Lerp(target rotation, current rotation, speed/smoothness).