Framerate drop after 1min playback with Cardboard and Easy Movie Texture

Has anyone ran into this?

I'm trying to do a 360 pano player for VR. I am currently using the latest stable Unity 5, the newest cardboard integration (GoogleVR now) and Easy Movie Texture. I've also tried with AVPro and seem to get the same result:

Basically, I create a simple scene with a sphere and some video in equirect format playing in the sphere. The video is 1080p @ 29.97 fps, H264.

When I start the app in my device (tested on iPhone6, Nexus 5 and Galaxy S6), the scene plays back pretty smooth. But after about 2min of playback, the framerate drops and the experience becomes no longer viable for VR, since the judder becomes extremely obvious.

I checked the logcat for the Galaxy S6 and I noticed that every few seconds, the GPU is being throttled down. I get the following messages:

 05-21 23:02:40.821  3521  6859 D SSRM:n  : SIOP:: AP = 470, PST = 440 (W:10), CP = 259, CUR = -361, LCD = 10
 05-21 23:02:40.821  3521  6859 D N       : limitGPUFreq:: freq = 600

the freq number (600 in my example above, drops with each subsequent debug message. Eventually I end up with about 187 or something).