How to make rigidbody not effect movement while still using it for collisions?

I have a object with rigidbody and my movement and stuff works fine but when I'm stationary my rigidbody moves the object upwards slowly no matter what is selected only thing that effects it is the freeze constrains but that causes the object to clip through surfaces if you push it against them (probably due to the object not having full collision capabilities if I freeze an axis) so any help or alternatives would be appreciated.