Camera Lerp behind player

I have a camera which follows my player around, as well rotate 360 degrees around him. The issue I'm having is the camera movement is rigid and follows directly behind the player. I would like to use a Lerp function to allow a small bit of delay to catch up to the player but I'm having trouble implementing it.

The Lerp function I have commented out works if I have none of the other stuff, but I need to get them working together, anyone have any ideas? I am using the following code:

 void Start()
     {
         //Starts the camera 5 units away
         distance = 3.5f;
 
         Vector3 angles = transform.eulerAngles;
         x = angles.y;
         y = angles.x;
     }
 
     void Update()
     {
         if (target)
         {
             x += Input.GetAxis("Mouse X") * xSpeed * 0.02f;
             y -= Input.GetAxis("Mouse Y") * ySpeed * 0.02f;
 
             y = ClampAngle(y, yMinLimit, yMaxLimit);
 
             Quaternion rotation = Quaternion.Euler(y, x, 0.0f);
             Vector3 position = rotation * new Vector3(0.0f, 1.0f, -distance) + target.position;
 
             transform.rotation = rotation;
             transform.position = position;
 
             //transform.position = Vector3.Lerp(transform.position, target.transform.position + new Vector3(0.0f, 1.0f, -3.0f), Time.deltaTime * speed);
 
         }
     }