Viewing desktop in-scene

Is there a way to view my desktop or other apps “behind” the game in-game? I know there are browsers and even potentially remote connections to view a separate machine, but I’d like to do it on the same one. I’m thinking something like viewing a terminal in-scene (probably inside a Render Texture) that your character can walk up to, but be able to open Paint or start up a game from your Steam library and leave it running on that in-scene console. Or sit on a golden throne on top of a mountain while updating the source code for your newest game on a magical floating portal that can view your desktop.

//Note: Inspired by and uses some code found here: http://forum.unity3d.com/threads/windows-api-calls.127719/

using UnityEngine;
using System.Collections;
using System;
using System.Runtime.InteropServices; // Pro and Free!!!

//WARNING!! Running this code inside Unity when not in a build will make the entire development environment transparent
//Restarting Unity would however resolve

public class TransparentWindow : MonoBehaviour
{
    [DllImport("user32.dll", EntryPoint = "SetWindowLongA")]
    static extern int SetWindowLong(int hwnd, int nIndex, long dwNewLong);
    [DllImport("user32.dll")]
    static extern bool ShowWindowAsync(int hWnd, int nCmdShow);
    [DllImport("user32.dll", EntryPoint = "SetLayeredWindowAttributes")]
    static extern int SetLayeredWindowAttributes(int hwnd, int crKey, byte bAlpha, int dwFlags);
    [DllImport("user32.dll", EntryPoint = "GetActiveWindow")]
    private static extern int GetActiveWindow();
    [DllImport("user32.dll", EntryPoint = "GetWindowLong")]
    private static extern long GetWindowLong(int hwnd, int nIndex);
    [DllImport("user32.dll", EntryPoint = "SetWindowPos")]
    private static extern int SetWindowPos(int hwnd, int hwndInsertAfter, int x, int y, int cx, int cy, int uFlags);

    void Start()
    {
        int handle = GetActiveWindow();
        int fWidth = Screen.width;
        int fHeight = Screen.height;

        //Remove title bar
        long lCurStyle = GetWindowLong(handle, -16);     // GWL_STYLE=-16
        int a = 12582912;   //WS_CAPTION = 0x00C00000L
        int b = 1048576;    //WS_HSCROLL = 0x00100000L
        int c = 2097152;    //WS_VSCROLL = 0x00200000L
        int d = 524288;     //WS_SYSMENU = 0x00080000L
        int e = 16777216;   //WS_MAXIMIZE = 0x01000000L

        lCurStyle &= ~(a | b | c | d);
        lCurStyle &= e;
        SetWindowLong(handle, -16, lCurStyle);// GWL_STYLE=-16

        // Transparent windows with click through
        SetWindowLong(handle, -20, 524288 | 32);//GWL_EXSTYLE=-20; WS_EX_LAYERED=524288=&h80000, WS_EX_TRANSPARENT=32=0x00000020L
        SetLayeredWindowAttributes(handle, 0, 51, 2);// Transparency=51=20%, LWA_ALPHA=2

        SetWindowPos(handle, 0, 0, 0, fWidth, fHeight, 32 | 64); //SWP_FRAMECHANGED = 0x0020 (32); //SWP_SHOWWINDOW = 0x0040 (64)
        ShowWindowAsync(handle, 3); //Forces window to show in case of unresponsive app    // SW_SHOWMAXIMIZED(3)
    }
}
  1. Add the above code to your project

  2. Go to your camera and select it. In the Inspector, change the colors and the alpha to 0.

  3. Build the (Windows) app - change the settings to windowed, then run

You now have a full screen window (without title bar) that allows the user to select things behind it, while at the same time still displaying anything you’ve rendered on top :slight_smile: Note: this is a windows only solution.

Edit 6: Updated and improved for Aero compatibility.

Thanks so much for letting me know! I got it working with click-through and always on top

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class TransparentWindow : MonoBehaviour
{
[SerializeField]
private Material m_Material;

private struct MARGINS
{
    public int cxLeftWidth;
    public int cxRightWidth;
    public int cyTopHeight;
    public int cyBottomHeight;
}

[DllImport("user32.dll")]
private static extern IntPtr GetActiveWindow();

[DllImport("user32.dll")]
private static extern int SetWindowLong(IntPtr hWnd, int nIndex, uint dwNewLong);

[DllImport("user32.dll")]
static extern bool ShowWindowAsync(IntPtr hWnd, int nCmdShow);

[DllImport("user32.dll", EntryPoint = "SetLayeredWindowAttributes")]
static extern int SetLayeredWindowAttributes(IntPtr hwnd, int crKey, byte bAlpha, int dwFlags);

[DllImport("user32.dll", EntryPoint = "SetWindowPos")]
private static extern int SetWindowPos(IntPtr hwnd, int hwndInsertAfter, int x, int y, int cx, int cy, int uFlags);

[DllImport("Dwmapi.dll")]
private static extern uint DwmExtendFrameIntoClientArea(IntPtr hWnd, ref MARGINS margins);

const int GWL_STYLE = -16;
const uint WS_POPUP = 0x80000000;
const uint WS_VISIBLE = 0x10000000;
const int HWND_TOPMOST = -1;

void Start()
{
    #if !UNITY_EDITOR   // You really don't want to enable this in the editor..
    
    int fWidth = Screen.width;
    int fHeight = Screen.height;
    var margins = new MARGINS() { cxLeftWidth = -1 };
    var hwnd = GetActiveWindow();

    SetWindowLong(hwnd, GWL_STYLE, WS_POPUP | WS_VISIBLE);

    // Transparent windows with click through
    SetWindowLong(hwnd, -20, 524288 | 32);//GWL_EXSTYLE=-20; WS_EX_LAYERED=524288=&h80000, WS_EX_TRANSPARENT=32=0x00000020L
    SetLayeredWindowAttributes(hwnd, 0, 255, 2);// Transparency=51=20%, LWA_ALPHA=2
    SetWindowPos(hwnd, HWND_TOPMOST, 0, 0, fWidth, fHeight, 32 | 64); //SWP_FRAMECHANGED = 0x0020 (32); //SWP_SHOWWINDOW = 0x0040 (64)
    DwmExtendFrameIntoClientArea(hwnd, ref margins);

#endif
}

void OnRenderImage(RenderTexture from, RenderTexture to)
{
    Graphics.Blit(from, to, m_Material);
}

}

I know it’s been a couple of months since you folks had this discussion, but with the help of a terrific Unity admin, I have the answer to 100% window transparency with 100% visible game objects! :slight_smile: LWA_COLORKEY is a red herring… For all the details, and an example, have a look at this thread on the Unity forums: http://forum.unity3d.com/threads/solved-windows-transparent-window-with-opaque-contents-lwa_colorkey.323057/

@StephanM I’d like to see a working example of this. Could you post one? Thanks!

Hey all!

Any idea how to this on Mac OS?

I typed a big answer about how to do what you’re trying to do, you want your PC and/or apps to run and be rendered in a running Unity build. Unfortunately, this ‘help site’ decided my post was an attack and erased what I wrote. So, I’m not going to type it all again, but it is easy to do (if you know what you’re doing).
The answers on this page seem to have nothing to do with your question, they are mostly for altering windows. You had it right when you mentioned Remote Desktop, basically. That is the way Remote Desktop works, it takes the video-memory stream, translates it to a mobile-friendly version, then feeds it over the internet to your device/PC. That’s the simple part, ANY data going through your computer can be read, copied or altered. Graphics, video, sound, text, data from programs, input from the keyboard, etc. Unity would be even faster than Remote Desktop because it wouldn’t have to send the data online, assuming Unity was running on the device you wanted to ‘stream’.
Also, instead of streaming to Unity or emulating an app inside Unity, sometimes it might just be easier to write a basic C#/java web-browser that runs inside Unity, or a ‘fake Desktop’ that interfaces with Windows/Android.

I know this chat is about 3 years old. But can someone just post the full updated code, and tell me where to and what to put the actual code on to?,

@TheWebExpert and @NeverSayDiegming , I know its been almost a year, but I did it, and I’m posting it for other people to see if they search this up.

  1. Make a script called TransparentWindow.cs

  2. Copy this code that @StephanM provided:
    (For some reason I can’t format the code correctly here, so look at the comment for this reply to get the code.)

  3. Attach this script to your camera

  4. Go to your camera in your inspector, and set Clear Flags to Solid Color. Click Background, and set all the values (R, G, B, and A) to zero.

  5. Now BUILD the game (Ctrl + b or File > Build and Run), and DO NOT click play. If you do, your editor will be transparent. It’s fine if you do though, you just gotta restart it again.

Graphics.Blit(from, to, m_Material);

why does not it work? black screen but not transparent and no desktop capture ?!
почему не работает? черный экран но не прозрачный и нет захвата рабочего стола?!