Door hinge joints

Hello, I am using physics to open and close the door, so if something is in its way it stops instead of being animated and just going through the objects.

The problem is when the door is opening/closing I can use my player to push the door beyond its set limits and it starts glitching out.

How do I make sure that the player can't apply force more/lower than the limits set?

Thank you.