Why does my player object falling slowly through the floor?

I'm making a little FPS game just so I can recall some of my knowledge because I didn't do game development in a while. I have the character controller on my player body and I removed rigidbody, because I realized it is causing some issues, like falling slowly through the floor for some reason. I'm not sure why. I was following a Brackeys tutorial on this link: https://www.youtube.com/watch?v=_QajrabyTJc&t=455s

This is my current setup. 'plyr' is the player body and on it was the rigidbody, when I removed it, I could move normally, but with the rigidbody on, I was slowly falling through the floor even with the colliders, there is a capsule which is a child of plyr and I removed a capsule collider from it.

Please help, thanks in forward.