Physics goes crazy when rigidbody collides with moving character controller

I have a cube with a rigidbody that is flying through the air. I also have a player with a player controller. I am using CharacterController.Move to move the player when it is walking. For some reason when the player is walking and hits the cube in the air, the cube seems like it clips into the character controller a bit, and the cube goes flying at insane speeds. If the cube collides with the player and the player isn't moving, everything acts perfectly normal and the cube stops and falls when it hits the player. Is there a way to fix this?