Bullet reflect not working properly

I'm trying to have a bullet projectile using raycast as collider to bounce off certain reflective objects. From what I can tell, it doesn't seem to mainly reflect based on where the bullet hits the object, but based on the reflective object's rotation. If the reflective object's rotation is 0, then no matter where the bullet hits it from, it will reverse its speed like it's hitting the object straight on.

I'm assuming the code I have does not take the rotation of both bullet and object into account. What can I change to make this work? Thanks in advance.

Code below here:

using UnityEngine; public class BulletBounce : MonoBehaviour { public float time; public float speed;

 private Vector3 direction;

 void Awake()
 {
     direction = Vector3.forward;
 }

 void FixedUpdate()
 {
     RaycastHit hit;
     if (Physics.Raycast(transform.position, transform.TransformDirection(direction), out hit, speed))
     {
         if (hit.transform.tag != "Player" && hit.transform.tag != "IgnoreBullet")
         {

             if (hit.transform.tag == "Reflect")
             {
                 direction = Vector3.Reflect(direction, hit.normal);
                 transform.position = hit.point;
                 transform.rotation.SetFromToRotation(Vector3.forward, direction);
                 return;
             }
         }
     }
     transform.Translate(direction * speed);
 }

}