Camera can't keep up with mouse when the FPS is low

So I am trying to make a first person controller, and everything is working fine untill the FPS drops. After the fps drops camera just can't keep up with the mouse. I move my mouse and there's a short pause before the camera rotates, and when it does it goes extremely slowly.

This is the code I'm using:

 public float mouseSensitivity = 3f;
 public Transform playerBody;

 float xRotation = 0f;

 // Start is called before the first frame update
 void Start()
 {
     Cursor.lockState = CursorLockMode.Locked;
 }

 // Update is called once per frame
 void LateUpdate()
 {
     float mouseX = Input.GetAxis("Mouse X") * mouseSensitivity; 
     float mouseY = Input.GetAxis("Mouse Y") * mouseSensitivity;

     xRotation -= mouseY;
     xRotation = Mathf.Clamp(xRotation, -90f, 90f); 

     transform.localRotation = Quaternion.Euler(xRotation, 0f, 0f); 

     playerBody.Rotate(Vector3.up * mouseX);
 }

Also, there's a same problem with standard asset first person controller, just not as bad as it is with mine.