First person camera snaps when switching cameras

Basically I'm trying to remake Five Nights at Freddy's for practice with a twist. I set up a first person controller with a camera and set up a camera system that turns off the first person camera and switched through the other cameras.

So what happens is the cursor is locked in first person but then once the cameras are on your cursor is free to click through the cameras. The problem is when you move your cursor to click on the cameras, when returning back to the first person camera, you are left in the same position you went in the cameras in (which is what I want) but then once you move your mouse the camera snaps to another place. (I figured out its basically the opposite of the position you moved your cursor in the cameras). This is really bugging me and I don't know how to fix this, anyone have a fix?