Quaternion.Slerp

Slerp / lerp or any other kind of "linear interpolation" was never ment to be used as damped movement. It's actually just a trick for a very simple damped movement. Fact is that it will never reach the destination because the remaining distance get's divided each step, so it get's smaller and smaller but it never reaches it's destination. Usually you decide when it's close enough so you actually don't see any visual difference.

Unity's math lib Mathf has some quite useful functions for damping a value change. You still can use Slerp, but the way it's intended. the t parameter should be "animated" from 0.0 to 1.0. If you increate t linearly the rotation will be at a constant speed. If you damp the t value at the end you get a slow down at the end.

Keep in mind that slerp or lerp need both the start and the end position / rotation as constanc values. Don't use the current rotation since it changes during the rotation.