Bounce physics is not working as expected in zero gravity. Is this right?

Bounce physics is not working as expected in zero gravity. I'm trying to make a variation of a simple pong type game. I don't want the rigidbody to have gravity applied to it. All the objects have a bounce material on them. If I leave gravity on, all the rigid bodies bounce as expected. If I turn gravity off. The ball hits the walls and slides across them rather than bouncing as expected. Anyone know how to do this is?