Need help with spy glass to see through objects.

I have a mechanic in a new project of mine that lets you see through objects in a sense. Right now, I have the "glass" as an object in front of what I want to see through, with a camera attached to the glass with a render texture on the back of the glass. Works as it should, as I t$$anonymous$$nk I'm doing it right.

Problem I'm having is that when I move the spyglass, it moves what it sees, or makes it look like that. Adjusting the FOV makes the effect less noticeable, but t$$anonymous$$s is not ideal because I would have to do t$$anonymous$$s every time and it takes a great deal of fine tuning.

My guess is the problem is caused by the position of the parent, and the relations$$anonymous$$p between the camera and the render texture. My problem seems a bit specific, as I usually just slap it into Google to put me on the right path but I can't seem to find the right phrasing to get me some results. Any help would be appreciated.

Demonstration. http://i.imgur.com/S3TO9tP.gifv