perspective distortion ray-casting

Hi,

im trying to create a ingame raycast through a portal.

A ray from the controller hits the yellow plane on the left. On this Plane, the cameraview is rendered to texture. The corresponding camera is at the exact position as the blue plane. The blue plane got the same dimensions as the left yellow plane.

Currently i'm casting a ray from the controller and transforming the hit.point from the left Plane to the right plane (blue) and cast another orthogonal ray from the same space, which hits the big yellow plane.

As you can see, the red dot indicator doesnt fit the view of the rendered cameraview on the left yellow plane. I'm sure i have to use trigonometry somhow but im not sure where to start. Every help very much appreciated!