Changing CharacterController's height causes jittering issue in Unity

As title said changing height of Character Controller causes jittering issue.

For example when I reduced the height of CharacterController, it seems CharacterController was floating at each frame and it keeps falling to the ground. Opposite situation has same issue, different is collider is actually already passing the ground so it just push up the object by physics engine.

(I saw some similar issue with Terrain, in this case it's worse, it just falling through the terrain)

Anyway someone told that I have to change center of CharacterController too, with foot offset(I don't know what exactly that is), so I tried to change the center of CharacterController when resizing height but it still have same problem, nothing different.

This is the code that I'm using:

 void Update() {
     if(Input.GetKeyDown(KeyCode.LeftControl)) {
         isCrouched = !isCrouched;
     }

     float newHeight = isCrouched ? crouchingHeight : originalHeight;
     controller.height = Mathf.Lerp(controller.height, newHeight, Time.deltaTime * smooth);

     Vector3 newCenter = isCrouched ? new Vector3(0, 0.2f, 0) : Vector3.zero;
     controller.center = Vector3.Lerp(controller.center, newCenter, Time.deltaTime * smooth);

     float newCamPos = isCrouched ? origCamPos.y - 0.2f : origCamPos.y;
     Vector3 newPos = new Vector3(cam.localPosition.x, newCamPos, cam.localPosition.z);

     cam.localPosition = Vector3.Lerp(cam.localPosition, newPos, Time.deltaTime * smooth);
 }

Is there a way to rid off that jittering issue while changing height of Character Controller? Any advice will very appreciate it.