Physics behave weirdly after timeScale change

So I have this game where the player robot is able to fire a mortar gun (cannon balls). My code for firing mortars is this:

 GameObject m = Instantiate(Mortar, turret.transform.position + Vector3.up, Quaternion.identity) as GameObject;
 Rigidbody body = m.GetComponent<Rigidbody>();                   
 var direction = (turret.forward + turret.up * 1f) * 500;
 body.AddForce(direction);

And this works really well, this results in a cannon ball being fired about 21 meters away. However, I am doing some operations in my game that takes place as fast as possible (about Time.timeScale = 20). When I then set the timeScale back to 1 after completing these operations, the mortar will now fire with much greater force, and the cannon balls will travel over 100 meters with the same code.

It seems that the Physics system somehow hasn't registered that the timeScale is 1 instead of 20, so it will confuse the timings for the physics.

If I switch to another scene and back to the scene, the physics works fine again. Is there anything I can do to make the Physics behave properly when changing timeScales?