How to apply dampening to hit normal based rotation?

I'm using a single raycast to detect the hit normal of objects in front and rotating a cube gameobject accordingly to avoid collision with obstacles in front of it. It works very well, but during the raycast the rotation is very rough and jitters the cube slightly. How could I apply a dampening or smoothing factor to make the cube rotate smoothly like it does when it is not raycasting?

My simple script (C#):

 using UnityEngine;
 using System.Collections;
 
 public class SimpleAvoidCS : MonoBehaviour {
 
     public Transform target; //Target we are moving towards
 
 void FixedUpdate(){
 
     //Directional vector towards our target
     Vector3 dir = (target.position - transform.position).normalized;
     RaycastHit hit = new RaycastHit();
         
     //Front Raycast
     if(Physics.Raycast(transform.position, transform.forward, out hit, 5)){
         if(hit.transform != transform){
             Debug.DrawLine(transform.position, hit.point, Color.blue);
             dir += hit.normal * 50;
         }
     }
 
     Quaternion rot = Quaternion.LookRotation(dir);
     transform.rotation = Quaternion.Slerp(transform.rotation, rot, Time.smoothDeltaTime);
     transform.position += transform.forward * 1 * Time.deltaTime;
     
     }
 }