Using a capsule collider is giving me some trouble with collisions (pics inside)

When the player is running into the colliders of a square corner some force is transferred to the player which will yield a velocity that dies off very slowly. It can't be seen on the 2nd picture but the player gets a velocity in the y-plane too.

In the first pic I'm running into the colliders and then just letting go, and the gravity is enabled while running and for an extra frame (when standing still) before it's disabled.

If I'm using a box collider I don't run into these issues at all but I'm afraid that changing to one will bring other unforeseen issues I'd have to deal with. I'm currently "solving" this by having to set velocity to 0 for 4 frames from when I stop running but it's an ugly solution and I would like to know the reason this occurs.