How to Change rotation while preserving local horizontal rotation

Ok, so I've made some sort of third person thing where you can run around small spherical planets. The player is rotated so their down is always toward the centre of the planet and they are pulled towards it. The camera is child of an empty gameObject called cameraPivot which has the same x and y as the player. Currently I have it so that the cameraPivot object has the same rotation as the player, then offset along the x and y axis according to the mouse input. Here is the Update script attached to cameraPivot:

 void Update ()
     {
         //Rotate Camera according to mouse
         transform.rotation = player.transform.rotation; // Set camera rotation to player rotation
 
         horizontalRotation -= Input.GetAxis("Mouse X") * mouseSensitivityX; // Find horizontal rotation from mouse input
 
         verticalRotation -= Input.GetAxis("Mouse Y") * mouseSensitivityY; // Find vertical rotation from mouse input
         verticalRotation = Mathf.Clamp(verticalRotation, -60, 60); // Dont let camera look up or down too far
 
         transform.Rotate(verticalRotation, horizontalRotation, 0); //Rotate camera by verticalRotation and horizontalRotation
     }

This all works fine, except for one thing - I want the horizontal rotation of the camera to be independent of the player's horizontal rotation. This would be easy enough on a flat world, but since the direction 'up' is for the player is constantly changing, it makes it a lot more difficult. Obviously using transform.Rotate like I am now won't work, as that creates a rotation relative to the players rotation. Basically, the end result I want is that the camera's w and y roation change relative to the player, but the x rotation stays the same, only being affected by mouse movement. I mean x, y and z along the axis of the camera, not the scene, of course.

Hope that made sense :P help would be greatly appreciated, I've searched but can't seem to find a way that works. I'm still trying to learn how to use all the rotation tools in unity properly, so possibly there's some simple solution I've overlooked.