How to remove recoil when not shooting?

Two days ago, I added a simple recoil to my camera when I shoot my guns. I added this to my MouseLook script(I'm using the standard assets' FPSController) :

     float yRot = sideRecoil + CrossPlatformInputManager.GetAxis("Mouse X") * XSensitivity;
                 float xRot = upRecoil + CrossPlatformInputManager.GetAxis("Mouse Y") * YSensitivity;
                 sideRecoil -= recoilSpeed * Time.deltaTime;
                 upRecoil -= recoilSpeed * Time.deltaTime;
                 if(sideRecoil < 0)
                 {
                     sideRecoil = 0;
                 }
                 if (upRecoil < 0)
                 {
                     upRecoil = 0;
                 }

and this function at the bottom:

 public void AddRecoil(float up, float side)
         {
             sideRecoil += side;
             upRecoil += up;
         }

The recoil works. Smoothly. But how can I make it go back to the original position? Like a recoil recovery? I can't find anything on the internet.

P.S I'm calling the AddRecoil Function from my Gun script.