Rotating Door with Child Doorhandle isn't working.

I have a door, that I rotate 90 degrees when it is opened. It then rotates -90 degrees when it is closed. The problem is that when I do t$$anonymous$$s, the door knob is somehow "pushed" away from the door. These pictures will show what I mean:

As you can see on the upper picture the door handle is in place, but when it rotates, the handle gradually moves "out" from the door, as seen on the bottom picture.

The door is rotated by rotating its parent object, that I use as the rotation axis. The door handle is a c$$anonymous$$ld of the door. The $$anonymous$$erarchy is like t$$anonymous$$s:

Rotation Axis > Door > Door handle

The script I use is the following:

 public float Speed = 150f;
         public float OpenAngle = 1f;
         public float ClosedAngle = 0f;
         private bool _isOpen = false;
     
         // Update is called once per frame
         void Update () 
         {
             if(_isOpen)
             {
                 if(transform.rotation.y <= OpenAngle)
                     transform.Rotate(0, Speed * Time.deltaTime, 0);
             }
             else
             {
                 if(transform.rotation.y >= ClosedAngle)
                     transform.Rotate(0, -1 * (Speed * Time.deltaTime), 0);
             }
         }

The rotation value of 1 is close to 90 degrees. When I rotate the rotation axis (Doors parent) in the editor, I do not have any issues. Clearly there's somet$$anonymous$$ng I'm missing, or failing to understand?

What can I do to rotate the door, so that the door handle stays where it is supposed to?

Thanks in advance!