Issue with wave-like motion when rotating with mouse input

I have encountered a rather odd issue with my program.

(To the moderators, I'm not sure if this is in the right space, if it isn't, please do move it)

The setup:

In my scene, I have a GameObject which I have set as my player/character for this game. The player/character has a child GameObject, which is a cube, which represents the head, and is a child via the hierarchy. It also has a body for the character(both of these are just children of the main children/character).

When I move the mouse, rotation of the head should follow where the mouse is(the mouse is hidden, so it relies on mouse input). If the movement around the y-axis is larger than a set amount, the body of the character should turn as well. I have this all working fine.

The problem occurs with the actual movement. In certain circumstances, it will create a wave-like movement, while still following the mouse

Ie. if I move the mouse vertically, then back to being "flat", then move it horizontally(without any y-movement), it will create a wave movement horizontally. The same occurs with the other rotation, where if I move the mouse horizontally first, then move it up and down, it will create a wave, but only in the up/down movement.

Gyazo for GIF demonstrating issue

In the GIF above, I move the mouse slightly upwards, then left and right. The "wave-like" movement can clearly be seen, as it moves to the left, and to the right.

This is my code:

 void moveView()
 {
     //This method is called every Update(), after a method that locks the cursor using
     //Cursor.lockState = CursorLockMode.Locked, then sets it to Cursor.visible = false;
     //Here are the raw inputs, the sensitivities are ySen and xSen, and are public floats.
     toRotOnY = Input.GetAxisRaw("Mouse X") * xSen;
     toRotOnX = Input.GetAxisRaw("Mouse Y") * ySen;
     //I think its here, but I'm not sure exactly what the *= operator does to Quaternions. Also, the init values are set in Start()
     targetRotHead *= Quaternion.Euler(-toRotOnX, toRotOnY, 0f);
     //The 0f in the Z feild is to lock rotation to x/y only.
     //Should body rotate to match head?
     if (this.transform.eulerAngles.y >= targetRotHead.eulerAngles.y + 10 || this.transform.eulerAngles.y <= targetRotHead.eulerAngles.y - 10)
     {
     //If the body should rotate, do it here
         this.transform.rotation = Quaternion.Slerp(this.transform.rotation, Quaternion.Euler(0f, targetRotHead.eulerAngles.y, 0f), moveSpeed);
        
     }
     //Rotate to the wanted rotation
     //moveSpeed is a float which dictates the smoothing.
     headWithCamera.transform.rotation = Quaternion.Lerp(headWithCamera.transform.rotation, Quaternion.Euler(targetRotHead.eulerAngles.x, targetRotHead.eulerAngles.y, 0f), moveSpeed);
  

 }