Remove Force from Moving Object

I have a rolling ball that the player can move around, which I'm doing using rigidbody.AddForce. I want to include a button (Left Shift, in my current code) that stops the ball completely, but am running into an issue. The ball stops still the moment the button is pressed, but afterward will continue rolling just as it was before. I've tried adjusting the ball's physics material friction and the AddForce Force Mode, neither seem to make any difference.

Can anyone help me to get the ball to halt completely, and not move again until the player's next input?

 function FixedUpdate ()
 {
     if(Input.GetKeyDown(KeyCode.LeftShift))
         rigidbody.velocity = Vector3.zero; 
     
     if(Input.GetKey("up"))
         rigidbody.AddForce(0,0,speed); 
         
     if(Input.GetKey("down"))
         rigidbody.AddForce(0,0,-speed); 
     
     if(Input.GetKey("right"))
         rigidbody.AddForce(speed,0,0);
         
     if(Input.GetKey("left"))
         rigidbody.AddForce(-speed,0,0);
 }