Player is Speeding up in collisions

I think your problem is two fold here. You don't have enough code provided to give a full answer so I will request you post how you activate when the player hits debris, you might be accidentally applying it multiple times.

As for slowing down the Player, you're going to want a speed modifier that can adjust the final result of force before it is assigned. I'm working on a similar game but it doesn't use force, just a simple translation. So what you'll want is to do is modify the speed based on the type of object it is colliding with.

 float step = (m_$$anonymous$$axSpeed * Time.deltaTime) * m_Speed$$anonymous$$odifier;

This is code from my game. So as you can see, the m_Speed$$anonymous$$odifier will effectively slow down the object when it walks on the debris. So you might have the m_Speed$$anonymous$$odifier = 0.25 or something when walking in debris, making the character walk at 1/4th the speed.

I am using rigid body forces and dont want to use a custom "speed modifier".

I think the Player is "speeding up" because his collision with the cube causes him to leave the ground, and then in the air he has no ground friction and moves quicker. I thiiink thats what's happening.

Ultimately I made it so the debris applies no force to the Player's rigid body, and the player just pushes them around with impunity. It's funner this way.