TimeScaling & Rigidbody problem

In this code, you can use the button to slow down the time. Everything works fine, except for one not very pleasant thing, probably related to physics processing.

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 using UnityEngine.UI;
 
 public class TimeDelay : MonoBehaviour {
 
     #region Variables
 
     [SerializeField] private Button button;
     [SerializeField] private Sprite[] buttonImages = new Sprite[2];
 
     private float startFixedDeltaTime;
     private bool buttonTap = false;
     public float slowdownFactor = 0.05f;
 
     #endregion
 
     private void Start()
     {
         startFixedDeltaTime = Time.fixedDeltaTime;
     }
 
 
 
 
     public void doSlowmotion()
     {
         buttonTap = !buttonTap;
 
         if (!buttonTap)
         {
             button.GetComponent<Image>().sprite = buttonImages[0];
 
             Time.timeScale = 1f;
             Time.fixedDeltaTime = startFixedDeltaTime;
         }
 
         if (buttonTap)
         {
             button.GetComponent<Image>().sprite = buttonImages[1];
             
             Time.timeScale = slowdownFactor;
             Time.fixedDeltaTime = Time.timeScale * .02f;
         }
 
     }
 }

In my game, you need to destroy "buildings" made from simple primitives, launching in them spheres that play the role of a kind of projectile. Problems begin when we strongly slow down time, i.e. make a small value of Time.timeScale - in this case all Rigidbody-objects (that is, the primitives from which the buildings are built) are easily "thrown" into the air, as if they were given force exactly up. This is less pronounced on a small deceleration (1/2) and very much interferes with large ones (as in the example, a deceleration of 20 times). Below you can see the gameplay of the game, where this moment is expressed. What is the problem?

Video