Why is a force exerted on my on my Game Object?

Why is a force exerted on my on my Game Object when it interacts with another one of my gameobject. One gameobject is a ball that has a bouncy physics2d material on the circle collider 2D and a rigidbody 2D. The main character game object has a box colluder 2D and a rigidbody 2D. When the main character interacts with the ball and tries to push it away, the ball does its usual things and gets bounced off, but the main character moves to the other direction making it look like some wind is pushing it. How do I fix this?
I can provide code if necessary.

Tick the X constraint in the rigidbody2D.