How to Access specific MipMap Level from RenderTexture directly?

I need to access the top level pixel of Texture's MipMap. This is what i am doing now:

         m_ShellNormalTarget = new RenderTexture(m_WidthBuffer, m_HeightBuffer, 32, 
        RenderTextureFormat.ARGBFloat);
         m_ShellNormalTarget.wrapMode = TextureWrapMode.Clamp;
         m_ShellNormalTarget.filterMode = FilterMode.Point;
         m_ShellNormalTarget.useMipMap = true;
         m_ShellNormalTarget.autoGenerateMips = true;
         m_ShellNormalTarget.Create();
         m_GBufferTex = new Texture2D(m_WidthBuffer, m_HeightBuffer, TextureFormat.RGBAFloat, true);

Later in the programI access mipmaps using this code:

     RenderTexture.active = m_ShellNormalTarget;
     m_GBufferTex.ReadPixels(m_BufferDimensions, 0, 0, false);
     m_GBufferTex.Apply();
     Color[] gbufValues = m_GBufferTex.GetPixels(m_GBufferTex.mipmapCount - 1);
     m_Normal = (new Vector3(gbufValues[0].r, gbufValues[0].g, gbufValues[0].b) * m_WidthBuffer * 
     m_HeightBuffer).normalized;

 

However, this has proven to be very slow, ReadPixels() has to copy ALL the pixels of RenderTexture in order for me to read only one pixel values.

Is there a way to achieve what I need without expensive ReadPixels() operation?