Strange Rigidbody Behavior

Hello everyone, I've noticed a strange behavior from a rigidbody movement script I have created. Essentially, when the rigidbody is moving at certain angles, the rigidbody will begin to drift away from its intended line of travel (even when floating with no gravity. The example picture shows the approximate line of travel, beginning along the rectangle's angle (160 degrees) and then drifting along a sloped path away from the wall as time goes on. Note that this does not occur when moving North, South, East, or West (or seemingly even nw,ne,sw,se), and that the rigidbody always continues to move forward (it just begins to drift).

I've tracked the line of code that makes this happen down to:

             if (speedClamp) //player can move in x and z axis only, 
             {
                 playerVector.z = Mathf.Clamp (playerVector.z, -playerMaxForwardSpeed, playerMaxForwardSpeed);
                 Debug.Log ("playerVector.z is: " + playerVector.z);
 
                 playerVector.x = Mathf.Clamp (playerVector.x, -playerMaxForwardSpeed, playerMaxForwardSpeed);
                 Debug.Log ("playerVector.x is: " + playerVector.x);
             }

If: playerVector.x = Mathf.Clamp (playerVector.x, -playerMaxForwardSpeed, playerMaxForwardSpeed); is commented out, the rigidbody moves correctly (never drifting away from its intended path), while the line for playerVector.z does not prevent the rigidbody from moving properly.

I'm quite confused as to why this is occurring, so if anyone has an idea, I would be grateful to know about it.