Keep Camera from player at a certain distance while orbiting around player

So far I have everyt$$anonymous$$ng working perfectly with the following code. The problem is, using transform.forward w$$anonymous$$le keeping distance sets the camera's Y-axis to the same height as the player's Y-axis. I want the camera to be a bit $$anonymous$$gher than the player rather than equal to the player.

The reason I am not simply attac$$anonymous$$ng the camera to the player is because the player is a sphere that is constantly rotating. The method I am using keep the camera separate from the player w$$anonymous$$le still keeping its distance. The player's movement is based on the direction the camera is facing, if that matters at all.

using UnityEngine; using System.Collections;

 public class RotateCamera : MonoBehaviour
 {
     public float rotateSpeed = 300.0f;
     public float distance = 3.0f;
 
     public GameObject player;
 
     // Use this for initialization
     void Start ()
     {
 
     }
     
     // Update is called once per frame
     void Update () 
     {
         // If A key is pressed, rotate camera to the left
         // If D key is pressed, rotate camera to the right
         // if S key is pressed, rotate the camera 180 degrees around player
         if (Input.GetKey(KeyCode.A))
         {
             transform.RotateAround(player.transform.position, Vector3.down, rotateSpeed * Time.deltaTime);
         }
         else if (Input.GetKey(KeyCode.D))
         {
             transform.RotateAround(player.transform.position, Vector3.up, rotateSpeed * Time.deltaTime);
         }
         else if (Input.GetKeyDown(KeyCode.S))
         {
             transform.eulerAngles = Vector3.Lerp(transform.eulerAngles, transform.eulerAngles + 180f * Vector3.up, 100 * Time.deltaTime);
         }
     }
 
     void LateUpdate ()
     {
         // Keep camera a specified distance away from player
         transform.position = player.transform.position - transform.forward * distance;
     }
 }

EDIT: basically I want to combine these two lines of code:

transform.position = player.transform.position - transform.forward * distance;

Combined with

 transform.position = player.transform.position - transform.up * distance;