Slow motion all but one - yet another round, hopefully the last

Hey guys, I know this is like, the third or fourth time something like this has been asked, but surprisingly, none of them gave a 100%-working answer. Hopefully this will be the last question, on the subject.

So here's what the docs says:

When timeScale is set to zero the game is basically paused if all your functions are frame rate independent.

This means that (correct me if I'm wrong) changing Time.timeScale will affect everything that depends on game time. So if I had something moving in Update like:

 transform.position += new Vector(x, y, z) * speed * Time.deltaTime;

This will get affected if we change the time scale. But:

 transform.position += new Vector(x, y, z) * speed;

This shouldn't get affected, if we were doing it in Update, since it runs every frame (time independent) right?

Well, if you make a simple test you'll see that changing the time scale will also affect this movement.

Try this in FixedUpdate, it will also get affected - in fact you'll get a very jerky movement.

So here's a couple of things I tried (in Update):

1. transform.position += new Vector(x, y, z) * speed / Time.timeScale;

This actually works pretty good, if you change the time scale to something like 0.1f, your movement won't get affected - But the problem is, this is frame-dependent, meaning performance will vary upon the hardware - Something very basic I learned when I started with Unity, is to multiply by Time.deltaTime - But if I do that, now it's now gametime-dependent, meaning changing the time scale will affect the movement. So, we got our smooth movement, but we lost our slowmo invincibility.

1. transform.position += new Vector(x, y, z) * speed / Time.timeScale * Time.fixedDeltaTime;

Now this seems to be the winning card I came up with, it works just like my first point in that changing the time scale won't affect it, but I'm not sure, if it gives smooth movement like the one we get when we multiply by Time.deltaTime - My guess is, since it actually makes changing the time scale has no effect, it doesn't make our movement time-dependent, it's still frame-dependent, so again, performance will vary upon hardware.

Now let me show you my test environment - I have a Player script attached to my player, which is just a sphere, and an Enemy script attached to my enemy spheres, which makes then move randomly using coroutines:

     public class Player : MonoBehaviour
     {
         public float speed = 25f;
         bool slow = false;
 
         void Update()
         {
             float h = Input.GetAxis("Horizontal");
             float v = Input.GetAxis("Vertical");
 
             transform.localPosition += new Vector3(h, 0, v)
                                         * Time.fixedDeltaTime
                                         * speed
                                         * 1 / Time.timeScale;
 
             if (Input.GetKeyDown(KeyCode.Space)) {
                 slow = !slow;
                 Time.timeScale = slow ? .1f : 1f;
             }
         }
     }

     public class Enemy : MonoBehaviour
     {
         public float timeToTake = 1f;
         public Transform[] locations;
 
         bool hasReachedTarget;
         Transform mTransform;
 
         void Awake()
         {
             mTransform = transform;
             hasReachedTarget = false;
         }
 
         IEnumerator Start()
         {
             while (true) {
                 Vector3 randPosition = locations[Random.Range(0, locations.Length)].position;
                 StartCoroutine(MoveTo(mTransform, randPosition));
                 while (!hasReachedTarget)
                     yield return null;
             }
         }
 
         // This is a modified version of the original `MoveTo` from UnityGems - Credits go there - Specifically the Coroutines++ tutorial.
         IEnumerator MoveTo(Transform objectToMove, Vector3 targetPosition)
         {
             float t = 0;
             Vector3 originalPosition = objectToMove.position;
             hasReachedTarget = false;
             while (t < 1) {
                 t += Time.deltaTime / timeToTake;
                 objectToMove.position = Vector3.Lerp(originalPosition, targetPosition, t);
                 yield return null;
             }
             hasReachedTarget = true;
         }
     }

I also made a video :) - To test out the previous case - I mess around changing different values.

You'll notice 2 things:

1. If I slow down time, it's true that I won't get affected, but collision start to mess up - I can go through walls! (~0:20)

2. If you were moving in normal time, and slow down time while you're moving, you'll get an incredible boost in speed! (~0:30)

My last attempt (not included in the video), was not to mess with time scale, but actually with the speed factor of the movement, I simply changed a few things:

My Player.Update:

 void Update()
 {
     float h = Input.GetAxis("Horizontal");
     float v = Input.GetAxis("Vertical");

     transform.localPosition += new Vector3(h, 0, v)
                                 * Time.deltaTime
                                 * speed;

     if (Input.GetKeyDown(KeyCode.Space)) {
         slow = !slow;
         var enemies = FindObjectsOfType(typeof(Enemy)) as Enemy[];
         foreach (var enemy in enemies)
             enemy.Slow(slow);
     }
 }

In my Enemy, I added a Slow method, and a static shared slowMoFactor:

 static float slowMoFactor = .1f;
 public void Slow(bool slow)
 {
     timeToTake = slow ? timeToTake / slowMoFactor : 1f;
 }

Now this works very well - Frame-independent movement, with player invincibility to time scale changing!

But I'm pretty sure, that real games aren't as simple as spheres moving around - So, slowing down each moving element individually, might not be very efficient, especially if there was so many, they'll have animations, rigidbodies, etc.

So, how should we go about slowing down time for all objects in the world except our player, in a robust, clean and efficient way? - A way that also, handles physics-related movement for objects that has rigidbodies.

Thanks a lot for any help in advance.