Is there a way to use Physics.Raycast accurately while time.timescale is set to zero?

For a test I attached the following script to a cube at origin on a scene:

 using UnityEngine;
 
 public class Test : MonoBehaviour
 {
     void Start()
     {
         Time.timeScale = 0;
     }
 
     void Update()
     {
         gameObject.transform.position = new Vector3(0, -1, 0);
 
         if (Physics.Raycast(Camera.main.ScreenPointToRay(Input.mousePosition)))
         {
             print("Hit!");
         } else
         {
             print("No Hit!");
         }
     }
 }

While hovering over the location that the cube was before the update of the scene, the raycast is true. While hovering over the location where the cube actually appears the raycast is false. I don't know if this behavior is to be expected (Maybe the physics doesn't update the collider positions while time is stopped?) However, what have is a drag and drop kind of system of containers with 3D item representations that follow along with it in a menu. I want to be able to gather raycastHit information on the item clicked. Any help is appreciated.