Calculate offset to other object for aim down sight system?

Hey, so I'm creating an aim down sight system right now and I want the gun to move to the right position when aiming. The problem is that I'm working with different scopes as well so the correct position changes with every scope. I thought about creating an empty transform at the middle of every scope and then move this transform with Vector3.Lerp to the centre of the camera wich works perfectly fine except that the parent of this centre point of the scope wich is the model for the gun as well, of course, aren't moving with the centre point. I searched for a solution where the parent is treated as a child of the centre point for a short time, but it seems like that isn't working. Now I'm thinking about calculating an offset between the centre point and the gun transform, recalculate this offset when aiming and then lerping between the offset when not aiming and the one when you're aiming. I tried InverseTransformPoint but that didn't work as expected or I'm just to dumb. Any ideas on how to do this the right way? I'm thankfully for EVERY response.