Slow motion for Physics game

I recently decided to add slow motion to my game, which can be activated at will by the player. I have no problem getting the desired effect, but my problem is either the physics or performance... let me explain:

The Physics It seems that the most common way to do slow motion is to adjust Time.timeScale, Time.fixedDeltaTime and Time.maximumDeltaTime, as I do in the snippet below, where .02f and .15f are the default values for fixedDeltaTime and maximumDeltaTime respectively.

 Time.timeScale = enableSlowMo ? .2f : 1;
 Time.fixedDeltaTime = Time.timeScale * .02f;
 Time.maximumDeltaTime = Time.timeScale * .15f;

This looks really good, but the downside is that because I make changes to fixedDeltaTime, the physics calculations are wrong, and the objects more differently in slow motion than at normal speed.

The Performance Issues My second solution was to just change Time.timeScale, this gives very jagged movement of most objects while in slow motion though. As such, I decided to enable interpolation on all rigidbodies, and voila, I have my desired result. The problem is though that in some of my levels I have several hundreds of rigidbodies (It's a physics game after all) which drastically affects performance when interpolation is enabled on all rigidbodies, especially in cases where a lot of collisions occur.

My own thoughts are that there may be some way to "calculate" my way out of the issues generated by the first solution, without cutting too much into performance, but I'm still coming up short.

In short, my question is: Are there better ways to do this, than the two I've mentioned?

Any input or hints would be greatly appreciated - Thanks for your time.